System design interview questions

Queues and messaging

Answer

• Queues decouple producers from consumers and absorb traffic spikes by buffering work.
• At-least-once delivery is common. Consumers must be idempotent.
• Dead-letter queues hold poison messages after retries for inspection and replay.
• Ordering is limited unless you use a partition/session with a single consumer at a time.
• Competing consumers scale throughput. Each message goes to one worker in the group.
• Pub/sub fans one event out to many independent subscribers. Competing queue consumers share work on one subscription.
• Visibility timeouts / ack deadlines define when an unacked message becomes available again.
• Backpressure: if consumers lag, queue depth grows. Alert on depth and age, not only on error rate.
• Poison messages that always throw must not block the partition forever. Isolate them to DLQ.
• Choose the broker for your ops skill (Azure Service Bus, RabbitMQ, SQS, Kafka for log-shaped streams).

C#
await channel.Writer.WriteAsync(new EmailJob(to, templateId));

// Consumer loop
await foreach (var job in channel.Reader.ReadAllAsync(ct))
{
    try { await email.SendAsync(job, ct); }
    catch { await deadLetters.AddAsync(job, ct); }
}

CAP Theorem and Distributed Trade-offs

Answer

• The CAP Theorem (Brewer's Theorem) states that in any distributed data store, across a network partition, a system can provide at most two of three guarantees: Consistency, Availability, and Partition Tolerance.
• Network Partitions are Inevitable: In real-world networks, dropped packets, switch failures, and cloud disconnects will happen. Because Partition Tolerance (P) is non-negotiable for distributed systems, the actual design choice is always between Consistency (CP) and Availability (AP) during a partition.
• CP (Consistency + Partition Tolerance): If network communication between replicas fails, nodes refuse writes or reads to prevent stale or conflicting data. Used in payment ledgers, inventory allocation, and distributed consensus (Raft, Paxos, Etcd).
• AP (Availability + Partition Tolerance): All nodes remain responsive to reads and writes during a partition, but data between nodes may diverge temporarily. Used in product reviews, user feeds, and shopping carts where high availability outweighs brief inconsistency.

C# (Partition Policy Simulation)
public enum DistributedPolicy { CP_StrongConsistency, AP_HighAvailability }

public class InventoryService
{
    // Ledgers must be CP: Reject write rather than sell duplicate item
    public async Task<bool> ReserveStockAsync(int itemId, int quantity, bool isPartitionDetected)
    {
        if (isPartitionDetected)
            throw new DistributedPartitionException("Consensus quorum unavailable. Write rejected to protect ledger integrity.");
            
        return await ExecuteQuorumWriteAsync(itemId, quantity);
    }
}

Beyond CAP: The PACELC Theorem

• CAP only describes behavior during rare network partitions. PACELC describes normal operations as well.
• PACELC Definition: If there is a Partition (P), how does the system trade Availability (A) versus Consistency (C)? Else (E), when the network is healthy, how does the system trade Latency (L) versus Consistency (C)?
• Example: MongoDB and DynamoDB allow tuning consistency levels per request (strong consistency with higher read latency, or eventual consistency with minimal read latency).

Load balancing

Answer

• A load balancer distributes requests across healthy instances of a service.
• Layer 4 balancers route by IP and port. Layer 7 balancers can route by path, host, and headers.
• Algorithms include round robin, least connections, and weighted targets.
• Sticky sessions pin a client to one node for in-memory session state, but they hurt even distribution and failover.
• Prefer external session stores or tokens so any node can serve any user.
• Health checks should verify the instance lightly, including critical dependency readiness when that matches your policy.
• Connection draining / deregistration delays let in-flight requests finish during deploy.
• TLS can terminate at the balancer or pass through. Know where certificates live.
• Uneven payload sizes make least-connections more fair than naive round robin for long requests.
• Client-side load balancing (in-process) still needs discovery and health awareness.

C#
// Balancer probes this; keep it cheap and accurate
app.MapGet("/health", () => Results.Ok(new { status = "healthy" }));

API versioning

Answer

• API versioning lets you change contracts without breaking existing clients on day one.
• URL (/v1/), header, and query string versioning each work. URL is obvious; headers keep paths clean.
• Breaking changes need a new version. Additive optional fields can often stay on the same version.
• Sunset old versions with metrics on traffic and a published retirement date.
• Document versions in OpenAPI. Generate separate docs per major version when they diverge.
• ASP.NET Core has versioning libraries that select controllers by version metadata.
• Avoid infinite parallel versions. Two majors in flight is a common operational limit.
• Event and message contracts need versioning too, not only HTTP.
• Strangler migrations can route /v1 to the old stack and /v2 to the new stack behind one hostname.
• Do not use versioning to hide poor additive design. Prefer backward-compatible evolution first.

C#
var v1 = app.NewVersionedApi();
v1.MapGet("/orders", () => "v1"); // example shape

// Or explicit routes:
app.MapGet("/v1/orders", () => Results.Ok());
app.MapGet("/v2/orders", () => Results.Ok());

Correlation ID

Answer

• A correlation id ties one user action across logs, traces, and outbound calls.
• Create or accept it at the edge (gateway or first middleware), then propagate a header such as X-Correlation-ID.
• Include it in every log scope so you can search one request across services.
• OpenTelemetry trace and span ids often play the same role. Keep a business-facing correlation id if ops already search that way.
• When accepting inbound ids, validate format and length so clients cannot inject huge or hostile values into logs.
• Message consumers should read the id from message headers and continue the chain.
• Return the correlation id on error responses so support can ask the user for it.
• Do not invent a new header name per service. Standardize in the platform.
• Middleware that pushes the id into AsyncLocal / logging scopes prevents forgotten manual passes.
• Missing correlation across a hop is a defect in instrumentation, not "noise".

C#
app.Use(async (ctx, next) =>
{
    var id = ctx.Request.Headers["X-Correlation-ID"].FirstOrDefault()
             ?? Guid.NewGuid().ToString("N");
    ctx.Response.Headers["X-Correlation-ID"] = id;
    using (LogContext.PushProperty("CorrelationId", id))
        await next();
});

Idempotency and Idempotency Keys

Answer

• An operation is idempotent if applying it multiple times produces the exact same side-effects and resource state on the server as applying it once. In HTTP, GET, PUT, and DELETE are defined as idempotent, while POST is not by default.
• Distributed systems face network drops, gateway timeouts, and retry storms. When a client sends a payment POST request and times out, it cannot distinguish whether the network failed before the server received the payload, during database persistence, or right after the server wrote the 200 response.
• An Idempotency Key is a unique token (typically a client-generated UUIDv4 sent via the Idempotency-Key header) that allows the server to identify repeated attempts of the same business action and avoid duplicate processing (such as double charging a credit card).
• Naive implementations that check a cache and then set a value have a critical race condition. If two identical requests hit two instances simultaneously, both find a cache miss and both execute the operation. Safe idempotency requires an atomic reservation lock and an explicit state machine.

C# (ASP.NET Core + EF Core)
public async Task<IResult> CreateOrderAsync(
    string idempotencyKey,
    CreateOrderRequest request,
    AppDbContext db,
    IPaymentService payment,
    CancellationToken ct)
{
    var requestHash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(request))));

    var record = new IdempotencyRecord
    {
        Key = idempotencyKey,
        RequestHash = requestHash,
        Status = IdempotencyStatus.Pending,
        CreatedAtUtc = DateTime.UtcNow
    };

    try
    {
        // Atomic reservation using SQL unique constraint
        db.IdempotencyRecords.Add(record);
        await db.SaveChangesAsync(ct);
    }
    catch (DbUpdateException) // Unique key collision
    {
        var existing = await db.IdempotencyRecords.AsNoTracking().FirstAsync(r => r.Key == idempotencyKey, ct);

        if (existing.RequestHash != requestHash)
            return Results.Problem("Idempotency key reused with different request payload.", statusCode: 422);

        if (existing.Status == IdempotencyStatus.Pending)
            return Results.StatusCode(StatusCodes.Status409Conflict);

        return Results.Content(existing.ResponsePayload!, "application/json", statusCode: existing.ResponseStatusCode);
    }

    // Execute side-effect and update state in one logical unit
    var order = await payment.ChargeAndCreateOrderAsync(request, ct);

    record.Status = IdempotencyStatus.Completed;
    record.ResponseStatusCode = StatusCodes.Status201Created;
    record.ResponsePayload = JsonSerializer.Serialize(order);
    await db.SaveChangesAsync(ct);

    return Results.Created($"/orders/{order.Id}", order);
}

The Idempotency State Machine: Pending, Completed, Failed

• State 1 - Pending: When the request arrives, the server atomically reserves the key in the database or Redis (e.g. via an INSERT or SETNX). If the key already exists and is marked Pending, a concurrent request is currently executing; the server returns HTTP 409 Conflict with a Retry-After header.
• State 2 - Completed: Once the business transaction succeeds, the server updates the key status to Completed and saves the response payload and HTTP status code. Subsequent requests with the same key immediately return the saved response without re-executing logic.
• State 3 - Failed: If an unhandled exception or business rule validation aborts the operation, the key is either cleared or set to Failed with an error message so the client can retry cleanly.
• Payload Integrity Verification: The server should compute and store a SHA-256 hash of the request body alongside the key. If an incoming request reuses an existing key with a modified payload, the server rejects it with HTTP 422 Unprocessable Entity to prevent payload tampering.

Storage Architectures: Relational Database vs Redis

• Relational Database: Creating an IdempotencyRecords table inside the primary database allows the business logic and the idempotency record update to commit inside the same atomic ACID transaction. A unique index on (ClientId, IdempotencyKey) guarantees that duplicate concurrent requests fail immediately at the database engine level.
• Distributed Cache (Redis): Useful for high-throughput APIs where database writes are costly. Uses SET idempotency_key {status: "pending"} NX EX 120. If Redis returns nil, the key is locked. This requires handling two-phase cleanup if the downstream service fails.
• Retention and Cleanup: Idempotency keys must be preserved longer than the caller's maximum retry duration (typically 24 to 72 hours). Old keys are expired using Redis TTLs or scheduled database partition drops.

Caching

Answer

• Caching trades freshness for lower latency and less load on the origin database or service.
• Cache-aside: read cache; on miss, load the store, then fill the cache. Application code owns the policy.
• Set a TTL and an invalidation rule. Too long means stale reads. Too short means weak hit rates.
• Redis is common for a shared cache across instances. IMemoryCache stays on one node and does not sync.
• Stampede risk: many misses on one key hit the DB together. Use single-flight, soft TTL, or staggered refresh.
• Invalidate or update cache on writes when readers must see changes sooner than TTL allows.
• Cache serialized DTOs, not EF entities with proxies. Keep keys explicit (tenant, version, id).
• Negative caching (caching "not found") can protect against repeated misses for missing keys, with a short TTL.
• Measure hit rate and origin QPS. A cache that is rarely hit is complexity without benefit.
• Security: do not cache personalized or sensitive responses in a shared key without including user identity in the key.

C#
public sealed class ProductCache
{
    private readonly IMemoryCache _cache;
    private readonly AppDbContext _db;
    public ProductCache(IMemoryCache cache, AppDbContext db)
    {
        _cache = cache;
        _db = db;
    }

    public async Task<Product?> GetAsync(int id, CancellationToken ct)
    {
        var key = $"product:{id}";
        if (_cache.TryGetValue(key, out Product? cached)) return cached;

        var product = await _db.Products.AsNoTracking()
            .FirstOrDefaultAsync(p => p.Id == id, ct);
        if (product is not null)
            _cache.Set(key, product, TimeSpan.FromMinutes(5));
        return product;
    }
}

Rate limiting

Answer

• Rate limiting stops abuse and accidental overload on public or expensive endpoints.
• Common algorithms: fixed window, sliding window, token bucket, and concurrency limits.
• When limited, return 429 Too Many Requests and a Retry-After header when you can.
• ASP.NET Core has built-in rate limiting middleware. Reverse proxies and API gateways can also enforce limits.
• Key by IP, user id, API key, or tenant. The wrong key punishes shared NATs or under-limits attackers with many IPs.
• Different endpoints need different budgets (login stricter than read-only catalog).
• Distributed rate limiting needs a shared store (Redis) if you run many nodes. In-memory limits are per node.
• Combine with authentication so anonymous and authenticated clients get different quotas.
• Load tests should include 429 behavior so clients back off instead of hammering.
• Rate limits are not a substitute for authZ or input validation. They are a capacity shield.

C#
builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter("fixed", opt =>
    {
        opt.Window = TimeSpan.FromMinutes(1);
        opt.PermitLimit = 60;
    });
});

var app = builder.Build();
app.UseRateLimiter();
app.MapGet("/search", () => "ok").RequireRateLimiting("fixed");

Observability

Answer

• Logs, metrics, and traces answer what failed, how often, and where time went across services.
• Structured logging (Serilog and similar) emits key/value fields that search systems can filter.
• Metrics are aggregated numbers (QPS, latency histograms, error rates) for dashboards and alerts.
• Distributed traces link spans across services for one request. OpenTelemetry is the common instrumentation standard.
• Alert on SLO burn rates and user impact, not on every noisy error spike.
• Cardinality explosion (high-unique labels) breaks metric systems. Keep label sets bounded.
• Correlation / trace ids must flow across HTTP and message headers or traces break at the boundary.
• Sample traces in high-volume systems, but keep errors and slow requests oversampled.
• Health checks are not full observability. They only answer "should this instance take traffic".
• Own dashboards per service and one journey-level view for critical user paths.

C#
Log.Information("Order {OrderId} placed for {CustomerId}", orderId, customerId);

// With OpenTelemetry, attach the same TraceId / SpanId to logs
// so one request is searchable across services.

Advanced: thundering herd and stampede

Answer

• A cache stampede / thundering herd happens when many requests miss the same key at once and all hit the origin.
• Mitigations: single-flight (one filler per key), soft TTL with early refresh, stale-while-revalidate, or lock around fill.
• A deploy or full cache flush can trigger the same pile-on. Warm hot keys or shed load at the edge during cold start.
• Probabilistic early expiration spreads refreshes so not every key expires on the same second.
• Rate limits and bulkheads protect the origin when misses still surge.
• Request coalescing in a gateway or BFF can collapse identical in-flight reads.
• For databases, connection pools and query timeouts limit how hard a stampede can punch.
• News-driven traffic spikes look similar. Precompute or cache the hot object before the event when you know it is coming.
• Metrics: origin QPS spikes aligned with cache miss rate point to stampede.
• Do not "fix" only with a longer TTL if staleness is unacceptable. Fix the fill coordination.

C#
private readonly ConcurrentDictionary<string, Task<Product>> _inflight = new();

public Task<Product> GetAsync(string id)
{
    if (_cache.TryGetValue(id, out Product hit))
        return Task.FromResult(hit);

    return _inflight.GetOrAdd(id, async key =>
    {
        try
        {
            var product = await _db.Products.FindAsync(key)
                ?? throw new KeyNotFoundException(key);
            _cache.Set(key, product, TimeSpan.FromMinutes(5));
            return product;
        }
        finally
        {
            _inflight.TryRemove(key, out _);
        }
    });
}

Advanced: exactly-once delivery myth

Answer

• Most message brokers give at-least-once delivery. The same message can arrive again after retry or failover.
• End-to-end exactly-once needs idempotent handlers (or a transactional inbox) and careful side effects. A broker flag alone is not enough.
• At-most-once means you may drop messages. Teams rarely choose that for business events.
• People say "effectively once" when duplicates are detected and ignored with an idempotency key or inbox row.
• Kafka transactions and similar features reduce duplication in specific topologies. Downstream side effects still need care.
• Exactly-once between three systems (DB A, broker, DB B) is a distributed systems problem, not a checkbox.
• Retries without idempotency cause double charges, double emails, and duplicate rows.
• Design the handler as if every message will arrive twice. If that is safe, you are close to effectively once.
• Outbox plus inbox is a common pair for publish and consume sides.
• Interview tip: explain the myth, then describe your dedupe strategy with concrete storage.

C#
public async Task Handle(OrderPlaced msg)
{
    if (await inbox.ExistsAsync(msg.MessageId)) return;
    await inbox.AddAsync(msg.MessageId);
    await billing.ChargeAsync(msg.OrderId);
}