System Design

How to do a rep properly. Stand up. Set a timer for 45 minutes. Say every word out loud, including the boring ones. Draw on real paper or a real whiteboard, not in your head. When the timer stops, stop mid-sentence and write down where you were — running out of clock at the data model means your requirements phase is too slow, running out at the deep dive means your architecture phase is too broad. Then score yourself against the checklist. The self-scoring is not optional; it is where the improvement comes from.

What is actually graded

The prompt will be one underspecified sentence — “design a system that lets users share photos,” “design something that runs scheduled data pipelines.” That vagueness is deliberate. The interviewer is not withholding a spec; there isn't one, and watching you construct it is the first graded moment.

Interviewers are typically scoring four axes, roughly in this weight order:

1. Scoping and requirements. Did you convert an ambiguous prompt into a bounded problem with numbers attached, and did you say what you were not building?
2. Trade-off reasoning. Not “I'd use a queue” but “I'd use a queue, and what I'm giving up is end-to-end latency and global ordering.” This is the axis that most separates senior from mid.
3. Technical depth. One component taken to the bottom — its data structures, its failure modes, its bottleneck — beats fifteen boxes each explained for ten seconds.
4. Collaboration. They were told to act as your design partner. Candidates who check in, take hints, and revise score higher than candidates who present a finished answer.

Note what is not on that list: knowing product names, producing a complete system, being certain. In 45 minutes you produce a credible skeleton plus one component in real depth, and you are allowed — encouraged — to change your mind when the numbers say to.

What it is not

• Not a memory test on cloud products. You never need to say a vendor's queue product name. You do need to say “a durable queue” and explain why.
• Not a coding round. Don't write code unless asked. Draw and talk.
• Not a place to be certain. “Let me revise that — I said one leader, but at this write volume I need to partition first” is a positive signal.
• Not a monologue. If you have talked for six minutes without the interviewer speaking, you are losing points regardless of content quality.

The clock

Memorize this. It is your safety net when the prompt is vague and your mind is blank.

If you remember one thing: minutes 0–10 are requirements and numbers, and skipping them is the most common way to fail a round you were qualified for.

That sentence does three things at once: it buys you permission to spend six minutes not drawing, it tells the interviewer you have a structure, and it invites them to steer. Say it every single time.

The scoping questions

Ask in roughly this order. Write every answer where you can both see it — you will point back at this list four times during the round.

1. Who are the users and what is the core use case? Internal engineers or external customers? One tenant or many?
2. What are the top two or three things it must do? Then immediately: “anything I should treat as out of scope?”
3. How big? Users, requests per second, data volume, growth rate.
4. Read-heavy or write-heavy, and what ratio? This single answer determines most of the architecture.
5. Latency target? Interactive (p99 under 200 ms) or batch (minutes are fine)?
6. Consistency? Must a read see the last write immediately, or is a few seconds of staleness acceptable?
7. Availability and durability? What actually happens if it's down for five minutes? Is losing data ever acceptable?

Then say the answers back as one sentence: “So: about 50,000 writes per second, reads can be a few seconds stale, availability matters more than strict consistency, and multi-region is out of scope for v1. I'll design against that.” This summary is worth several minutes of design work, because everything after it can be justified by pointing at it.

The non-functional checklist — free points

Name these explicitly in the requirements phase. Most candidates forget most of them.

The failure modes, in order of frequency

Six shapes, two arrows

When every component is the same rectangle, the picture carries no information the labels don't already carry, and the interviewer has to read instead of see. Commit to six shapes.

Two payoffs. The shapes do work your mouth would otherwise have to do — a stadium instead of a rectangle already says “losing this costs latency, not data.” And the moment you draw a cylinder you have committed to answering “what's the partition key?”, which is a question you want to be asked.

The invisible grid

Two laws and three reserved zones, set up in the first ten seconds. Left to right is the request: anything a user waits for flows left→right, no exceptions. Top to bottom is depth: higher is closer to the user, lower is closer to the disk.

Use about the left two-thirds for the main picture. On a shared virtual canvas, never zoom out to fit — the interviewer sees your viewport, and a diagram they have to squint at reads as one you lost control of.

Draw order

The single biggest improvement available: stop drawing the finished picture. Draw four pictures, each a small delta on the last, narrating the delta. The interviewer watches you reason, and you never face a blank canvas with a whole system in your head.

1. The spine. Three boxes — who calls it, the thing, where state lives — and two arrows. Forty seconds. Always correct.
2. Label every arrow, then trace one request. Verb and payload on each edge, then walk one write and one read end to end, touching each box. Errors surface here, cheaply.
3. Widen only where a number forced it. Add the balancer because you computed the QPS; the cache because you computed the read ratio. Say the number as you draw the box.
4. Annex the deep dive. Ring one component, draw its interior large in the space you reserved. Do not erase the main picture.

Labeling

An unlabeled arrow means “these two things are related somehow,” which is worth almost nothing. Labels are where the design lives.

• Every arrow gets a verb — writes, invalidates, polls every 30s, publishes. If you can't pick a verb, you don't know what the edge does.
• Every store gets its partition key, written on the cylinder the moment you draw it.
• Every stateless box gets a count — ×12. It proves horizontal scalability and that you did the arithmetic.
• Every hot edge gets a number — QPS, payload size, or latency budget, whichever the design turns on.
• Every important edge gets a failure note — timeout 50ms → fail open. This is the one that reads as operational experience.

Marker and mouse

Numbers and arithmetic

Two minutes of arithmetic buys more credibility than ten minutes of boxes. Round aggressively — nobody wants precision, they want to see you can do it at all.

Three ratios explain most design decisions: memory is ~1000× faster than SSD; SSD is ~100× faster than spinning-disk random I/O; one cross-continent round trip costs more than thousands of local disk reads. Almost every architecture argument is one of these three ratios in disguise.

Capacity arithmetic

• Seconds in a day ≈ 10⁵ (86,400). A month ≈ 2.6M. A year ≈ 31.5M.
• QPS from DAU: DAU × actions_per_day / 105. 10M DAU × 10 actions = 1,000 QPS average.
• Peak is 2–5× average. Say which multiplier you're assuming and why.
• Storage: writes/day × bytes × retention_days × replication_factor. Nearly everyone forgets the replication factor.
• Bandwidth: QPS × payload. 1,000 × 10 KB = 10 MB/s (trivial). 1,000 × 10 MB = 10 GB/s (a completely different system).
• Powers of two: 2¹⁰ = 1 K, 2²⁰ = 1 M, 2³⁰ = 1 B. So a 32-bit ID holds 4B values; a 64-bit ID holds 1.8×10¹⁹.

Anchors worth memorizing. One commodity machine: ~10K QPS of simple requests, a few TB of SSD, ~64–256 GB RAM. A single relational instance: comfortable to ~10–50K reads/sec behind a cache, and hurting on writes well before that. 1 KB × 1M rows = 1 GB; 1 KB × 1B rows = 1 TB. A single Kafka-class partition: ~10 MB/s sustained.

Availability arithmetic

Two things to say. Dependencies multiply: a service calling five dependencies each at 99.9% has a ceiling of 99.5% unless it can degrade without them — which is the actual argument for circuit breakers and graceful degradation. And most candidates over-promise: proposing five nines for a v1 internal tool is a scoping failure, not ambition.

The network

Almost every latency budget is dominated by network, and almost every candidate skips straight to the load balancer. Being able to start one hop earlier is cheap credibility.

What a first request actually costs

DNS resolution (0–50 ms, then cached for the TTL — it is not paid per request). TCP handshake, one round trip. TLS handshake, one further round trip on TLS 1.3, two on 1.2 — and zero on resumption. Then the request itself. This is why connection reuse, keep-alive, and connection pools matter so much, and why a “simple” extra service hop is never free.

Protocol choices

Push versus pull is the underlying question and it comes up constantly. Pull (polling) is simple, stateless, and wastes requests when nothing changed. Push is efficient and forces you to hold connection state and solve delivery for offline clients. The usual real answer is push with a pull-based reconciliation on reconnect — say that.

HTTP versions, in one line each

HTTP/1.1: one request at a time per connection (head-of-line blocking), so browsers open six. HTTP/2: multiplexes many streams over one TCP connection, but a lost packet stalls all of them. HTTP/3 (QUIC over UDP): removes transport-level head-of-line blocking and cuts handshake round trips. You rarely need more than this in a design round.

Consistency models

“Strong or eventual?” is a false binary and treating it as one is a common tell. There is a ladder, and different operations in the same system usually sit on different rungs.

CAP and PACELC

CAP: during a network partition, you must choose availability or consistency. That is a narrower claim than it is usually given credit for — it says nothing about normal operation. Do not lecture on it.

PACELC is the more useful sibling: if there's a Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The second half is what you actually trade on every request of every day. A design that says “I'm choosing latency over consistency here, and this is what the user might see” is doing real work.

Replication

Replication is copies of the same data. It buys read throughput and survival; it does not buy write throughput. Confusing it with partitioning is one of the most common vocabulary errors in these rounds.

The lag consequence to volunteer: with async replication a user posts a comment, the read goes to a lagging follower, and their own comment is missing. Fixes: pin that user's reads to the leader for a few seconds after a write; or pass a version token the replica must have caught up to; or read from the leader for anything the user just touched.

Conflict resolution, when writes can happen in two places

• Last-write-wins. Simple, and it silently discards concurrent writes. Only acceptable when losing one is genuinely fine.
• Version vectors / vector clocks. Detect that two writes were concurrent rather than ordered, then hand both to the application (or the user) to merge.
• CRDTs. Data types whose merge is mathematically guaranteed to converge — counters, sets, sequences. The right answer for collaborative editing and offline-first clients.
• Application-level merge. Domain rules decide. Most real systems.

Partitioning (sharding)

Partitioning is different data on different machines. It buys write throughput and capacity. You shard when one machine can no longer hold the data or absorb the writes — and not before, because sharding costs you cross-shard queries, cross-shard transactions, and rebalancing forever.

The choice of partition key is the single most consequential decision in most designs. Say that out loud, then think about it out loud.

Secondary indexes across shards

An under-discussed topic that reliably impresses. Local (document-partitioned) index: each shard indexes only its own data — writes are cheap and local, but every query by that index must scatter-gather across all shards. Global (term-partitioned) index: the index itself is partitioned by the indexed term — reads hit one shard, but writes now touch two partitions and you need to keep them consistent, usually asynchronously. Name which one you're picking and why.

Rebalancing

Never use hash(key) % N where N is the node count — adding a node remaps almost every key. Use a fixed large number of logical partitions (say 1,024) assigned to physical nodes, so growing the cluster moves whole partitions and nothing gets rehashed. This is the standard answer and it is worth a sentence.

Transactions and isolation

ACID: Atomicity (all or nothing), Consistency (invariants hold — really an application property), Isolation (concurrent transactions don't corrupt each other), Durability (a committed write survives a crash, which in practice means it hit a write-ahead log on stable storage).

Isolation is the one worth knowing properly, because “we'll use a transaction” is not an answer until you say which level.

Write skew, the classic example: two doctors are on call; each checks “is at least one other doctor on call?”, sees yes, and both go off call. Snapshot isolation permits this because neither wrote the row the other read. The fixes are serializable isolation, an explicit lock on the thing you're reasoning about (SELECT ... FOR UPDATE), or materializing the conflict into a row you do write.

How isolation is implemented

• Two-phase locking (2PL). Acquire locks, hold to commit. Correct, and readers block writers — throughput suffers under contention, and deadlocks must be detected.
• MVCC (multi-version). Writes create new versions; readers see a consistent snapshot. Readers never block writers and writers never block readers — this is why nearly every modern database uses it, and it's what makes time-travel queries possible.
• Serializable snapshot isolation (OCC). Run optimistically, track read/write sets, abort at commit if a serialization conflict occurred. Great under low contention, wasteful under high.

Distributed transactions and delivery semantics

The moment a business operation spans two services or two shards, atomicity stops being free. There are exactly four workable answers and you should be able to name all four.

1. Don't. Redesign so the atomic unit lives in one partition. Choosing a partition key that keeps related entities together is the cheapest possible fix and the one to propose first.
2. Two-phase commit. A coordinator asks everyone to prepare, then to commit. Correct, but it blocks: if the coordinator dies after prepare, participants hold locks indefinitely. Mention that you'd rather avoid it.
3. Saga. A sequence of local transactions, each with a compensating action to undo it. No global locks, and the system is temporarily inconsistent by design. The realistic answer for cross-service workflows — but compensations are business logic, not a rollback, and “un-charge the card” is not always possible.
4. Outbox + idempotency. The workhorse. See below.

The dual-write problem and the outbox

You cannot atomically write to your database and publish to a message log — one of them will succeed and the other won't, and then your state and your events disagree. The fix: write the event as a row in an outbox table in the same transaction as the state change, then have a separate relay process read the outbox and publish. Atomicity is restored because there's only one write. This is the single most useful pattern to be able to draw from memory.

Delivery semantics

How to make something idempotent, concretely: a client-generated request id stored with a uniqueness constraint; an upsert keyed on a natural id instead of an insert; a conditional write with a version number (compare-and-set); or a state machine that ignores transitions it has already made.

Consensus, leases and time

Consensus is how a set of machines agree on one value despite failures. In an interview you need the shape, not the proof.

Raft in six sentences

Nodes are follower, candidate, or leader. A follower that hears no heartbeat becomes a candidate and requests votes for a new term. A candidate with a majority becomes leader. All writes go to the leader, which appends to its log and replicates; an entry is committed once a majority has it. A majority is required for both election and commit, which is why an odd number of nodes (3, 5) is standard — and why the cluster stops accepting writes rather than splitting when it loses quorum. Reads either go through the leader or require a lease to be safe.

Where to use it: leader election, cluster membership, configuration, distributed locks, small critical metadata. Where not to: the data hot path. Consensus is a coordination bottleneck; you use it to decide who is in charge, not to serve every request.

Leases and fencing tokens — the detail that separates candidates

A lock with a timeout is a lease. The trap: a slow process is indistinguishable from a dead one. A worker takes a lease, pauses for a long garbage collection, its lease expires, the system hands the work to someone else — and then the first worker wakes up and writes, believing it still holds the lock. Now two writers.

The fix is a fencing token: the lock service issues a monotonically increasing number with each lease, the worker includes it in every write, and the downstream store rejects any write carrying a token lower than the highest it has seen. Being able to explain that failure and that fix is a genuinely strong signal.

Clocks

• Wall-clock time is not trustworthy across machines. NTP-synced clocks routinely drift by milliseconds and occasionally by much more; they can jump backwards. Never order events across machines by timestamp and never use one to decide a lock expiry.
• Lamport clocks give a total order consistent with causality, but cannot tell you whether two events were concurrent.
• Vector clocks can: they detect concurrency, at the cost of size proportional to the number of nodes.
• Hybrid logical clocks combine physical time with a logical counter, so timestamps are both causally correct and close to wall time. Increasingly the practical answer.
• Bounded-uncertainty clocks (atomic clocks plus GPS, exposing an interval rather than a point) let a system wait out the uncertainty and get globally consistent timestamps. Powerful, and it costs specialized hardware.

Failure detection

Heartbeats with a timeout are the baseline — the timeout is a guess, and a wrong guess either declares healthy nodes dead or notices real deaths too slowly. Gossip spreads membership information without a central coordinator and scales to large clusters. Phi-accrual detectors output a suspicion level from the observed distribution of heartbeat intervals rather than a binary alive/dead, which adapts to a network that is merely slow. The point to make: you cannot distinguish a dead node from a slow network, so every design must be correct under both interpretations.

The front door

Load balancing algorithms, briefly

Round robin ignores that requests differ in cost. Least connections is better when request durations vary widely. Least outstanding requests is the practical default at scale. Consistent hashing when you need the same key to land on the same backend (cache affinity, sticky sessions) — and it is worth saying that stickiness is a form of state you are choosing to accept. Power of two random choices — sample two backends, pick the less loaded — gets most of the benefit of global load awareness with none of the coordination, and mentioning it is a nice touch.

Health checks deserve a sentence: shallow checks (is the process up) miss a broken dependency; deep checks (can I reach my database) can take your whole fleet out of rotation simultaneously when that dependency blips. The usual answer is a shallow check for the balancer plus a separate deep readiness signal.

Rate limiting — know all four algorithms

Caching

The highest-value block to know deeply, because caching questions have crisp right answers and most candidates only know one strategy.

The four problems to raise before you're asked

• Stampede / thundering herd. A hot key expires and ten thousand requests hit the store at once. Fix: a single-flight lock so only one request refills, probabilistic early expiry, or serving stale while refreshing in the background. Also add jitter to TTLs so keys don't expire in lockstep.
• Hot key. One key takes 40% of traffic and melts one node. Fix: replicate the key across nodes with a suffix, or put a tiny in-process cache in front of the shared one.
• Invalidation. TTL is simple and stale; explicit invalidation is fresh and easy to get wrong (especially with multiple writers). A third option that avoids the problem: version the key so a data change produces a new cache key and the old one just ages out.
• Cold start. If the cache absorbs 97% of reads, losing it means 30× load on the store instantly. Say what protects you: request coalescing, load shedding, and a store provisioned for some multiple of steady state.

Eviction policies: LRU is the default; LFU is better for stable popularity distributions; TTL-only is simplest and wastes memory; W-TinyLFU (admission control plus LFU) is what modern caches use, and mentioning that admission matters — caching an item that is never read twice is pure cost — is a good detail.

Storage engines and datastores

Pick the shape that matches the access pattern

The honest answer to “SQL or NoSQL?” is “it depends on the access pattern and the consistency requirement, not on the scale” — then pick one and defend it. Blanket “NoSQL scales better” is a weak answer; a sharded relational database scales fine, and you keep transactions.

B-tree versus LSM tree — worth one confident sentence

“This is write-heavy, so I want an LSM-based store — I'm accepting compaction overhead and read amplification in exchange for sequential writes” is a genuinely senior sentence.

Other storage vocabulary you should have

Write-ahead log: append the intent before mutating, so a crash is recoverable — the basis of durability everywhere. Bloom filter: “is this key definitely absent?” No false negatives, some false positives; used to avoid disk reads in LSM trees. Index: a second data structure that speeds reads and slows every write — say that trade when you add one. Covering index: contains all columns a query needs, so the base table is never touched. Denormalization: trading write complexity and storage for read speed; the standard move when reads dominate by 100:1.

Queues, logs and streams

Reach for asynchrony when you want to decouple producers from consumers, absorb bursts, retry failures without blocking a user, or fan one event out to many consumers. The distinction that matters:

Things to say about any queue

• At-least-once is the default, so consumers must be idempotent. Dedupe on an event id.
• Dead-letter queue after N attempts, so one poison message can't wedge the pipeline — and alarm on the DLQ, or it becomes a silent black hole.
• Ordering is per-partition. If you need per-user ordering, partition by user id. Global ordering means one partition, which means no parallelism.
• Consumer lag is the metric you page on — not queue depth alone, because a growing backlog with healthy consumers means you are simply under-provisioned.
• Backpressure: when consumers can't keep up, either buffer (bounded!), shed load at the edge, or slow producers. Say which. Unbounded buffering is how systems die.

Stream processing vocabulary

Windowing: tumbling (fixed, non-overlapping), sliding (overlapping), session (gap-defined). Event time versus processing time: events arrive late and out of order, so aggregating by arrival time gives wrong answers. Watermarks: a declaration that you believe all events up to time T have arrived, which is what lets a window close. Late data: either drop it, or emit a correction. Checkpointing: periodic state snapshots so a failed operator resumes rather than restarts.

Lambda versus kappa architecture: lambda runs a batch path and a streaming path in parallel and reconciles (accurate, but two implementations of every computation); kappa runs only the stream and replays the log when it needs to recompute. Kappa is usually the better answer now, precisely because the log is replayable.

Search and analytics

Inverted index: term → the list of document ids containing it. Queries intersect posting lists. Add positions for phrase queries. Ranking is typically BM25 (term frequency, inverse document frequency, length normalization) plus business signals, and in modern systems a vector/embedding retrieval stage combined with keyword retrieval.

Typeahead / autocomplete: a trie of prefixes with the top-K completions precomputed and cached at each node, rebuilt offline from query logs. Serve from memory at the edge. The interesting parts are cache invalidation cadence, personalization, and typo tolerance (edit distance or fuzzy matching on the trie).

Columnar storage for analytics: store values column by column, so a query reads only the columns it touches and compresses far better because a column is homogeneous. Partition pruning: keep min/max metadata per column per chunk so a query can prove a chunk is irrelevant and skip reading it — if you independently propose this, say out loud that you have just re-derived how analytical stores work. Separation of storage and compute: data sits in object storage; independent compute clusters read it, scale to zero, and cannot contend with each other.

Probabilistic structures — cheap credibility

Reach for these the moment someone says “now there are a billion elements and you have one gigabyte of RAM.”

“You can't average percentiles” is a small, extremely reliable credibility win in any monitoring or metrics discussion.

End to end: where a request actually goes

Most candidates start their diagram at the load balancer. Starting one hop earlier and narrating every stop to the disk and back is where the latency-budget conversation comes from.

Two follow-ups to be ready for. “What if the client is in Europe?” The 25 ms becomes 90 ms each way, which is when read replicas or a second region stop being optional. “Where would you put retries?” At the client with jitter and at the gateway — never at every layer, because layered retries multiply into a retry storm the moment the origin gets slow.

Where it actually runs

“How would you deploy this?” is asked in most senior rounds and answered badly in most senior rounds. It is not a question about vendor product names — it is a question about failure domains: what is the largest thing that can die without taking the service with it. Draw it as nested dashed boxes; the nesting is the answer.

• Pod dies: the orchestrator restarts it; the balancer's health check removes it in seconds. Nobody notices — which is why the service box is stateless.
• Zone dies: half the capacity vanishes, so each zone must be provisioned to carry the whole load. That is the real cost of multi-AZ and naming it is a strong signal. The database fails over to the synchronous standby with no data loss.
• Region dies: now you're choosing. Active-active is expensive and forces you to solve write conflicts; warm standby is cheap and costs an RPO of seconds and an RTO of minutes. Say which you picked and why — there is no correct answer, only a stated one.
• The trap in this diagram is the cross-zone write. Multi-AZ is not latency-free, and a chatty service crossing zones five times per request has spent 2.5 ms on nothing.

Vocabulary to have straight: RPO (recovery point objective) is how much data you can lose; RTO (recovery time objective) is how long you can be down. Warm standby optimizes cost; active-active optimizes RTO; pilot light sits between them.

The read path

Reads are a funnel. Every layer is cheaper and staler than the one behind it, and the design question is always how much staleness buys how much cost.

The percentages are the argument. A layer that sheds 2% of traffic is a layer you should delete rather than operate — and saying you deleted a layer is as strong a move as adding one.

The write path and the outbox

The organizing idea is a line across the diagram: above it, work the user waits for; below it, work someone else does later. Getting as much as possible below that line is most of what “make it fast” means in practice.

Shard and replicate

Shards go across, replicas go down. Drawing the grid makes the distinction impossible to blur.

Fan-out: how to draw a trade-off

When choosing between two designs, do not describe them in sequence. Draw both side by side and point at the one edge that differs, so the interviewer can see what they are choosing between.

The real answer is almost always hybrid: fan out on write for normal accounts, merge at read time for the few thousand accounts with enormous followings. The celebrity case is not an edge case; it is the design.

Service boundaries and multi-tenancy

Monolith or microservices

The honest senior answer: start with a well-structured monolith and extract services when a specific pressure demands it — independent scaling, independent deploy cadence, team ownership, or a genuinely different resource profile. Splitting for its own sake converts local function calls into network calls that can fail, and turns one transaction into a saga.

Costs to name if you propose services: distributed tracing becomes mandatory; every call needs a timeout, a retry policy and a circuit breaker; you inherit versioning and backwards compatibility; and cross-service data consistency is now your problem.

CQRS and event sourcing

CQRS separates the write model from one or more read models, each shaped for its query and updated asynchronously from the write side. Reach for it when read and write patterns diverge sharply; the cost is eventual consistency between them and more moving parts.

Event sourcing stores the sequence of changes as the source of truth and derives current state by replaying it. You get a perfect audit log, time travel and rebuildable projections; you pay with schema evolution of old events, snapshotting for performance, and the fact that “just query the current state” is no longer simple. Propose it only when auditability or replay is an actual requirement.

Multi-tenancy

Three isolation levels, in ascending cost: shared everything with a tenant id column (cheapest, and one bug leaks data across tenants); shared infrastructure, separate schemas or databases; dedicated infrastructure per tenant (simplest isolation story, worst economics, usually reserved for the largest customers). Noisy neighbours are the recurring failure: per-tenant rate limits, per-tenant concurrency caps, and weighted fair queueing so one tenant's burst cannot starve everyone else. Say this unprompted in any B2B design.

Reliability patterns

Blast radius: answering “what happens when X dies?”

Point at your existing diagram and annotate — a cross on the box, an arrow showing where traffic goes instead, one line on what the user sees. Do not redraw.

Observability and SLOs

Volunteering this material separates people who have operated systems from people who have only designed them.

• Three signals. Metrics — cheap, aggregated, what you alert on. Logs — expensive, high-cardinality, what you debug with. Traces — per-request causality across services, what you use to find which hop got slow. Add events/audit for compliance.
• RED for request-driven services: Rate, Errors, Duration. USE for resources: Utilization, Saturation, Errors. Quote one of these and you sound like you've been on call.
• Alert on symptoms, not causes. Page on “p99 latency above target” or “error rate above budget,” not on “CPU above 80%.” High CPU with a healthy service is not an emergency.
• SLI, SLO, error budget. The SLI is the measurement (fraction of requests under 200 ms); the SLO is the target (99.9%); the error budget is what's left over, and it is what makes the deploy-versus-stability argument quantitative rather than political.
• Cardinality is the killer in any metrics system. Every distinct label combination is a separate series; adding a user_id label turns a thousand series into ten million and takes the system down. Raise this unprompted in any monitoring design.

Security, in the amount a design round wants

• AuthN versus authZ. Authentication is who you are; authorization is what you may do. Keep them separate in the diagram — authN at the gateway, authZ in the service that owns the resource, because only it knows the rules.
• Tokens. OAuth 2.0 is the delegation framework; OIDC adds identity on top; a JWT is a signed, self-contained claim. The trade to name: self-contained tokens can't be revoked before expiry, so use short lifetimes plus refresh tokens, or accept a revocation-list lookup and lose the statelessness.
• Service-to-service: mTLS, or signed short-lived tokens. Never a shared static secret.
• Encryption in transit (TLS everywhere, including inside the VPC) and at rest (managed keys, rotation). Secrets in a secret manager with rotation, never in config or images.
• Tenant isolation. Every query is scoped by tenant, enforced in one place rather than in each handler. Say where that enforcement lives.
• Abuse and defense. Rate limits per key and per IP, quotas, request size caps, query cost limits, input validation, and audit logging of who did what. For anything user-supplied and public: content scanning and a takedown path.
• PII. Minimize what you store, encrypt it, restrict who can read it, and have a deletion story — which is genuinely hard once data is replicated into caches, logs, backups and analytics. Naming that difficulty is a strong signal.

Deployment and schema change

Rolling replaces instances gradually — cheapest, and two versions run at once, so your API and schema must be compatible with both. Blue-green keeps a full second environment and flips traffic — instant rollback, double the cost. Canary sends a small percentage to the new version and watches error rates before proceeding — the usual answer at scale. Feature flags separate deploying code from releasing behaviour, which is what makes fast rollback possible without a redeploy.

Schema migration on a live system — expand, migrate, contract: add the new column or table (compatible with old code); deploy code that writes both old and new; backfill existing rows in batches; switch reads to the new; then remove the old. Four deploys, no downtime, every step reversible. Volunteering this sequence is unusual and reads as real production experience.

1 · Distributed rate limiter

Why start here: small enough to finish in 45 minutes, and it contains most of the reusable ideas.

Scoping. Per user, per IP, or per endpoint? Hard limit or soft? Distributed across many servers — assume yes, that is the whole problem. And the question that earns the first good signal: fail open or fail closed if the limiter itself is down? Almost always fail open — do not let your limiter take down your API.

Scale. 1M API keys, 10K QPS, the decision must add under 5 ms.

Interface. allow(key, cost=1) → (bool, retry_after).

Algorithm. Know all four (Part 4). Token bucket if bursts should be permitted; sliding window counter if they shouldn't.

The graded moment is the race condition: two servers both read 99 and both write 100, letting 101 through. Fix with an atomic read-modify-write — a Lua script or equivalent server-side operation. Naming that race unprompted is most of the round.

Deep dive. The counter store is now on the hot path of every request, so: tight timeout, fail open, and optionally a local approximate fallback. At 10K QPS a network hop per request is expensive — the alternative is each server enforcing N/num_servers locally with zero coordination, which is approximate and breaks under uneven load. Name the trade and pick. Hot tenant: split their counter into K sub-counters and sum, accepting slight over-admission. Response: 429 with Retry-After, and clients back off with jitter.

2 · URL shortener

Scoping. Custom aliases? Expiry? Analytics? Who can delete? Massively read-heavy — say that early, it drives everything.

Scale. 100M new URLs/month → ~40 writes/sec. Reads at 100:1 → ~4,000 reads/sec, peak ~20K. 100M × 500 bytes × 5 years ≈ 3 TB. Small data, high read rate — this is a caching problem, not a storage problem.

Key generation is the actual design question. Three options: hash the URL (deterministic, dedupes, needs collision handling); a global counter base-62 encoded (no collisions, but a coordination bottleneck and the ids are enumerable); or — the answer that shows you think about coordination cost — each app server leases a block of 10,000 ids from a central allocator and hands them out locally: one coordination call per 10,000 URLs instead of per URL. Seven base-62 characters is 62⁷ ≈ 3.5 trillion keys.

Storage. Key-value: short_code → (long_url, owner, created_at, expires_at), partitioned by hash of short_code. No joins needed, so a KV store beats relational here.

Read path. CDN → shared cache (hit rate is enormous; the popular 1% of links are most of the traffic) → store. Return 301 if you don't need analytics and 302 if you do — a 301 is cached by the browser and you stop seeing the clicks. That detail lands well.

Deep dive. Analytics as fire-and-forget events into a log, aggregated asynchronously — never block the redirect on counting. Expiry by TTL plus a background sweeper. Abuse: rate-limit creation, scan destinations against a malware list, and support takedown.

3 · News feed / timeline

Scoping. How fresh must the timeline be? How large can a following get? Chronological or ranked? Can we drop posts, or must delivery be complete?

Scale. 300M DAU, 2 posts/day → 600M writes/day ≈ 7K writes/sec. Reads: 20 feed loads/day → 6B reads/day ≈ 70K reads/sec, peak ~300K. Read-dominated by ~50:1.

The core decision is fan-out on write versus fan-out on read (Part 5). Fan-out on write precomputes one timeline per user: cheap reads, expensive writes, and it explodes for accounts with 40M followers. Fan-out on read stores each post once and merges at read time: cheap writes, and p99 becomes the slowest of N queries.

The answer is hybrid, and saying so with the threshold named is the point: precompute for normal accounts; for the few thousand accounts above some follower count, don't fan out — merge their recent posts into the timeline at read time. Two paths, one merge step.

Storage. posts partitioned by post_id; timeline[user_id] as a capped list of the most recent few hundred post ids in a cache, with a durable backing store; follows as an adjacency list partitioned by follower.

Deep dive. The timeline cache holds ids, not post bodies — hydrate bodies from a separate cache so an edited or deleted post doesn't need rewriting into millions of timelines. Ranking as an offline scoring pipeline whose output is a feature store the serving path reads. Backfill for a new follow. And the failure mode to name: fan-out lag under a traffic spike means some users see a stale feed — which is acceptable, and saying so explicitly is the trade-off answer.

4 · Chat / messaging

Scoping. One-to-one only or group? Group size cap? Delivery receipts and read receipts? Message history retention? End-to-end encryption? Online presence?

Scale. 50M DAU, 40 messages/day → 2B/day ≈ 25K writes/sec, peak 100K. Connections are the other capacity dimension: 50M concurrent WebSockets at ~10K per box is 5,000 gateway machines — do that arithmetic out loud, it is unusual and impressive.

Architecture. Clients hold WebSocket connections to a stateless-ish gateway tier. A session registry maps user_id → gateway_id so a message can be routed to the right connection. The chat service persists the message first, then publishes to the recipient's gateway; if the recipient is offline it goes to a push-notification path and waits in their inbox.

Data model. messages(channel_id, seq, sender, body, ts) partitioned by channel_id and clustered by seq — every read is “the last N messages in this channel,” so this key makes the common query a single sequential scan. A per-channel monotonically increasing sequence number gives ordering without trusting clocks.

Deep dive — the two hard parts. Ordering: assign the sequence number server-side at the channel's partition, so all participants agree; client timestamps are unusable. Exactly-once display: the client generates a message id, the server dedupes on it, and the client reconciles on reconnect by asking for everything after its last known sequence number — at-least-once delivery plus idempotent display. Group fan-out is the feed problem again: fan out on write for small groups, read-time merge for very large ones. Presence is best-effort with a TTL heartbeat, and it is worth saying that presence at this scale often costs more than messaging.

5 · Notification service

Scoping. Which channels — push, email, SMS, in-app? Transactional or marketing (different latency and compliance requirements)? Are user preferences and quiet hours in scope? What are the delivery guarantees?

Scale. 100M notifications/day ≈ 1.2K/sec average, but campaigns arrive as bursts of millions in a minute — the burst, not the average, is what you design for.

Architecture. Producers publish an intent to a log. A preference and eligibility service filters (unsubscribed, quiet hours, rate caps per user). A templating step renders per channel. Then per-channel worker pools with their own queues and their own rate limits, because each third-party provider has different throughput and failure behaviour. A delivery status store closes the loop with provider webhooks.

Deep dive. Isolation: separate queues per channel and per priority, so a marketing blast cannot delay a password reset — this is the single most important structural decision. Idempotency: dedupe on a notification id so a retry doesn't send twice; users notice duplicates immediately. Third-party failure: circuit breaker per provider with failover to a secondary, and a DLQ. Backpressure: a campaign of 10M is chunked and admitted at a controlled rate rather than dumped into the queue. Fan-out storms: cap per-user notification rate and coalesce ("3 people liked your post") — both a product feature and a load-shedding mechanism.

6 · Distributed job scheduler with dependencies

The richest prompt on this list and the one that rewards having operated something.

Scoping. Users submit DAGs of tasks; run each task after its dependencies succeed; retry on failure; survive a worker dying mid-task. Scheduled (cron) and event-triggered runs. Out of scope for v1: cross-DAG dependencies and backfills.

Scale. 100K DAGs, 10M task executions/day → ~120/sec average, ~1,000/sec peak. Tasks run from seconds to hours — that range is why leases exist.

Data model. dag_definitions(dag_id, spec, schedule); runs(run_id, dag_id, state, started_at); tasks(run_id, task_id, state, attempt, worker_id, lease_expires_at) partitioned by run_id; task_deps(run_id, task_id, depends_on).

The dependency logic is Kahn's algorithm: keep an in-degree per task, decrement as each parent succeeds, enqueue at zero. Saying that turns an abstract design into something you have obviously implemented. Cycle detection happens at submit time — run Kahn's and if you don't drain every node, there's a cycle.

Deep dive — where the round is won. Exactly-once execution is impossible; at-least-once plus idempotency is the real answer, because a worker can finish and die before reporting. Worker death: leases with expiry, reclaimed when heartbeats stop — and the risk you must name is that a merely slow worker is still running while you reschedule, so fence with a monotonic token. Scheduler HA: single leader via a consensus service; the same fencing idea prevents split-brain double-scheduling. Fairness: per-tenant queues with concurrency caps so one customer's 100K tasks can't starve everyone. Retries: backoff with jitter, a max-attempts cap, then a terminal FAILED that propagates downstream as UPSTREAM_FAILED rather than leaving children pending forever. The bottleneck: the metadata database, since every state transition is a write — batch updates, partition by run_id, keep the hot ready-set in memory with the database as durable backing.

7 · Metrics, monitoring and alerting

Scoping. Ingest metrics from 100K hosts; query recent data interactively and historical data for dashboards; users define threshold alerts and get notified. Recent means seconds of freshness; old means downsampled.

Scale. 100K hosts × 1,000 metrics each, sampled every 10 s = 10M data points/sec. That number alone dictates the design — say it early.

Write path. Append-only, and it must never do a random read. Agents push to an ingest tier → a durable log for buffering → an in-memory write buffer → periodic flush of immutable, compressed, columnar blocks to object storage. Time-based partitioning, because every query is a time-range query; accept that the current partition is a hot shard — it is designed for.

Retention. Raw for 7 days, one-minute rollups for 90 days, one-hour rollups for 2 years. Without downsampling, storage is unbounded.

Query path. An inverted index from label → series ids, then fetch time ranges for those series and aggregate.

Alerting. A rule evaluator that periodically runs each user's query over the recent window; matches produce an alert state transition, which goes to a notification path (design 5). The details that matter: deduplication and grouping so one bad deploy doesn't send 500 pages; hysteresis ("firing for 5 minutes") so a single spike doesn't page anyone; silences during maintenance; and evaluation sharded by rule so it scales horizontally.

Deep dive. Cardinality is the killer — every distinct label combination is a separate series, and adding a user_id label turns 1,000 series into 10 million and takes the system down. Raise it unprompted; it is the failure mode every practitioner has lived through. Also: you cannot compute a true p99 by averaging p99s across hosts, so store histograms or t-digests and merge those instead.

8 · Distributed key-value store

The “prove you know the fundamentals” prompt. It is really a checklist of Part 3.

• Partitioning: consistent hashing with virtual nodes, so adding a node moves ~1/N of keys and the distribution is actually even.
• Replication: N=3 replicas on the next three distinct nodes around the ring, deliberately spread across failure domains (racks or zones).
• Consistency: quorum reads and writes, tunable per request. W + R > N gives strong-ish reads; W=1 gives fast writes and weak reads.
• Conflicts: version vectors to detect concurrency, or last-write-wins — and say what LWW loses, namely that concurrent writes silently disappear.
• Durability: write-ahead log, then memtable, then flush to sorted immutable files — an LSM tree, with bloom filters to avoid touching files that can't contain the key.
• Membership and failure detection: gossip.
• Anti-entropy: Merkle trees to find divergent ranges between replicas cheaply, then repair only those.
• Hinted handoff: a replica is down, so a neighbour accepts the write and forwards it when the node returns.

Deep dive options: the read path with bloom filters and compaction strategy (size-tiered favours writes, levelled favours reads); how a range scan works when keys are hashed (it doesn't — you need an ordered partitioner, and that reintroduces hot shards); and how you add a node without a latency spike (throttled streaming of partitions, serve from the old owner until handoff completes).

9 · Typeahead / autocomplete

Scoping. Prefix matching only, or fuzzy? Personalized? How fresh must new terms be — minutes or days? Multi-language?

Scale. Every keystroke is a request: 10M searches/day with 20 keystrokes each is 200M requests/day ≈ 2.3K/sec average, and it must return in under 100 ms or the feature feels broken. Client-side debouncing (fire after ~50 ms of no typing) is a legitimate part of the design — mention it, because it cuts load by more than half.

Architecture. Two decoupled halves. Offline: aggregate query logs into term frequencies over a rolling window, build a trie with the top-K completions precomputed and stored at every node, and ship that structure to the serving tier. Online: a read-only in-memory lookup — walk the prefix, return the stored list. No computation at request time at all.

Deep dive. The trie is large, so shard it by prefix (all queries beginning “ca” live together) and serve from memory at the edge. Rebuild cadence is the freshness/cost trade — hourly is usually fine, with a small hot-terms overlay for breaking events. Typo tolerance via edit-distance expansion on short prefixes, which is expensive, so cap it. Filtering for offensive or unsafe suggestions belongs in the offline build, not the serving path. Personalization is a second, much smaller per-user list merged with the global one at request time.

10 · Video upload and streaming

Scoping. Upload plus playback, or playback only? Live or on-demand? What resolutions? DRM? Global audience?

Scale. The number that matters is egress, not requests: 1M concurrent viewers at 5 Mbps is 5 Tbps. No origin serves that — this is a CDN problem, and saying so in the first two minutes is the correct framing.

Upload path. Client requests a pre-signed URL and uploads directly to object storage, bypassing your servers entirely — say this explicitly, because routing multi-gigabyte uploads through an application tier is a common and expensive mistake. Chunked and resumable, because mobile connections drop. Completion publishes an event.

Transcode pipeline. A worker pool consumes upload events, splits the video into segments, transcodes segments in parallel into a ladder of bitrates, then packages into HLS/DASH with a manifest. Parallel-by-segment is what makes a two-hour film transcode in minutes. Output back to object storage; CDN pulls from there.

Playback. The player fetches the manifest, then requests segments, choosing bitrate adaptively from measured throughput and buffer level. Segments are immutable and cacheable forever, which is what makes CDN hit rates near-perfect.

Deep dive. Transcode is expensive and bursty — use a priority queue (a creator's first upload matters more than a re-encode), spot/preemptible capacity with checkpointing, and idempotent per-segment jobs so a lost worker costs one segment, not the whole video. Thumbnail and metadata extraction as separate consumers of the same event. For live, the same pipeline with much smaller segments and a latency/robustness trade you should name explicitly.

11 · Proximity / ride-hailing dispatch

Scoping. Find nearby drivers, or nearby static places? How fresh must locations be? What radius? Is matching in scope, or only search?

Scale. 1M active drivers reporting location every 4 s = 250K writes/sec of location updates — a write-heavy problem disguised as a search problem, and noticing that is the first good signal. Rider queries are far fewer, maybe 10K/sec.

The core technique: turn two-dimensional proximity into a one-dimensional key. Geohash encodes lat/long into a string where a shared prefix means spatial proximity, so “nearby” becomes a prefix range query in any ordinary key-value store. Quadtree subdivides adaptively and handles dense cities better. H3/S2 use hexagonal or spherical cells and avoid geohash's edge artifacts. Pick one and name the weakness: with geohash, two points either side of a cell boundary have completely different prefixes, so you must always query the cell and its eight neighbours.

Architecture. Location updates go to an in-memory geospatial index sharded by cell, with a durable log behind it for recovery — driver positions are cheap to lose and expensive to store, so treat them as ephemeral. Rider queries hit the index for candidate cells, then a matching service ranks by ETA (not straight-line distance) and dispatches.

Deep dive. Dense cells are hot shards by construction — a downtown cell has 100× the drivers of a suburb, so subdivide adaptively rather than using a uniform grid. Matching must avoid double-booking a driver: a short lease on the driver record while an offer is outstanding, released on decline or timeout. Location updates are the write bottleneck — batch them, and reduce reporting frequency for stationary drivers.

12 · Payments / ledger

Why do this one: it is the clearest test of whether you actually understand idempotency and consistency, and the correct answers are the opposite of most of this document's defaults.

Scoping. Internal ledger, or integration with external processors? Which currencies? Are refunds, chargebacks and holds in scope? What are the audit requirements?

The non-negotiables to state up front. Money requires strong consistency — this is the design where you choose C over A, and say so. No lost writes, ever. Full auditability. And every operation must be idempotent, because a network timeout on a payment must never mean “maybe we charged them twice.”

Data model — double-entry. Never store a mutable balance. Store immutable entries(entry_id, account_id, amount, direction, txn_id, ts) where every transaction writes at least two entries that sum to zero, and derive balances by summing (with periodic snapshots so you don't sum from the beginning of time). This is append-only, auditable, and correct under concurrency — and proposing it unprompted is a very strong signal.

Idempotency. The client supplies an idempotency key with every payment intent. Store it with a uniqueness constraint alongside the resulting transaction id; a replay returns the original result rather than performing the operation again. Keys expire after a bounded window.

Deep dive. External processors are unreliable and slow: so the flow is a state machine — pending → authorized → captured → settled — persisted at every transition, with a reconciliation job that compares your ledger to the processor's daily report and flags divergence. Never treat an HTTP timeout as a failure; treat it as unknown, and resolve it by querying the processor with your idempotency key. Cross-account transfers in one partition if possible; if not, a saga with explicit compensating entries (a reversal entry, never a deletion). Sharding by account id keeps a single account's entries together and makes its balance a local computation, at the cost of cross-account transfers spanning shards — name that trade.

One-page cheat sheet

Rapid-fire drills

Cover the answers. If you can't produce one in ten seconds, that's a section to reread.

Glossary

Anti-entropy

Background process that finds and repairs divergence between replicas, usually with Merkle trees.

Backpressure

Signalling upstream to slow down when a consumer can't keep up, instead of buffering without limit.

Bulkhead

Isolated resource pools so one failing dependency can't consume the resources others need.

CDC (change data capture)

Streaming a database's change log as events, so other systems can react without polling.

Compaction

Merging LSM files to reclaim space and reduce read amplification. Consumes I/O and causes latency spikes.

CRDT

A data type whose merge is guaranteed to converge regardless of order — used for offline-first and collaborative editing.

Dead-letter queue

Where messages go after repeated failures, so one poison message can't block the pipeline.

Fencing token

A monotonically increasing number issued with a lease; downstream rejects stale tokens, preventing two writers.

Hinted handoff

A healthy node temporarily accepts writes for a down node and forwards them on recovery.

Hot key / hot shard

One key or partition receiving a disproportionate share of traffic; the usual reason even hashing isn't enough.

Idempotent

Applying the operation twice has the same effect as applying it once.

Lease

A lock with an expiry, so a crashed holder doesn't block forever. Needs fencing to be safe.

MVCC

Multi-version concurrency control: writes create new versions, so readers never block writers.

Outbox pattern

Writing an event row in the same transaction as the state change, relayed separately — solves the dual-write problem.

Quorum

A majority (or a configured W/R) required for an operation to be considered durable or current.

Read amplification

Reading more data than requested — e.g. an LSM read touching several files.

RPO / RTO

How much data you can lose / how long you can be down after a disaster.

Saga

A sequence of local transactions with compensating actions, replacing a distributed transaction.

Scatter-gather

Fanning a query to every shard and merging results; p99 becomes the slowest shard.

Single-flight

Collapsing concurrent identical requests into one, so only the first does the work.

Split brain

Two nodes both believing they are leader. Prevented by quorum plus fencing.

Thundering herd

Many clients hitting a resource simultaneously after an expiry or restart.

Watermark

A declaration that all events up to time T are believed to have arrived, allowing a window to close.

Write amplification

Writing more bytes than the logical change requires — page splits, compaction, replication.

Write skew

Two transactions read overlapping data and each writes something only safe if the other didn't.

Lines that score

• “Before I draw anything, let me pin down scope — and say what I'm treating as out of scope.”
• “Let me do quick math so we're designing against a real number.”
• “I'll use X here. What I'm giving up is Y, and I'd revisit it if Z changed.”
• “The partition key is the most consequential decision here, so let me think about it out loud.”
• “This is at-least-once, so the consumer has to be idempotent. I'd dedupe on the event id.”
• “That's even in expectation, but it's vulnerable to skew if one tenant is a large fraction of traffic.”
• “I'd start with one lock for correctness, then shard it if it becomes the bottleneck.”
• “For v1 I'd do the simple thing — a single instance — because at these numbers it works, and I'd revisit at roughly 10× the write rate.”
• “Let me revise what I said earlier — at this write volume that doesn't hold.”
• “Is that the direction you'd want, or is there a part you'd rather I go deeper on?”
• “I want to be honest that I haven't operated X at this scale. Here's how I'd reason about it and what I'd validate first.”

That last one is safe to say. Manufactured certainty is a far worse signal than calibrated honesty, and interviewers can tell the difference immediately.

Self-critique checklist

Score every practice rep against this. The self-scoring is where the improvement comes from.

• Spent five-plus minutes on requirements before drawing anything
• Explicitly said what was out of scope
• Did capacity math out loud and landed on a concrete number
• Stated the read/write ratio and the consistency requirement per operation
• Named and justified the partition key
• Drew left-to-right, with the requirements strip and the deep-dive annex reserved
• Put a verb on every arrow and a count on every stateless box
• Traced one write and one read end to end through the picture
• Went deep on one component without being asked
• Said “because ___” and “the cost is ___” for every technology named
• Named at least three failure modes and what the user sees in each
• Said something about how it is deployed and how it is monitored
• Said what breaks first at 10×
• Checked in with the interviewer at least twice
• Revised something out loud when the numbers said to
• Stayed audible the entire time

The last one is not filler. In post-interview debriefs, “went quiet and I couldn't tell what they were thinking” sinks more otherwise-strong candidates than any technical gap. An ambiguous prompt is not a trick — it is an invitation. Treat the interviewer like a colleague you're whiteboarding with on a Tuesday afternoon, and the round mostly runs itself.