Like 0

Interview prep notes

linkjobai revised this gist 5 days ago · 9e3e99d

1 file changed, 33 insertions

adclick.md (file created)
@@ -0,0 +1,33 @@
1 + # System Design Interview: Designing an Ad Click Aggregator
2 +
3 + Advertisers are billed on click counts, so the numbers must be correct, and they want them within a minute.
4 +
5 + ## Requirements
6 +
7 + - Ingest click events (ad_id, user, timestamp, ...) at ~50k per second, peaks of 5x.
8 + - Query clicks per ad per minute for the last N minutes, and top-K ads per minute.
9 + - Counts must be accurate for billing; late and duplicate events happen.
10 +
11 + ## Pipeline
12 +
13 + 1. **Click endpoint** logs the event and redirects the user to the advertiser. Each click gets a unique ID (signed into the ad URL) so duplicates can be detected.
14 + 2. Events go into **Kafka**, partitioned by `ad_id`.
15 + 3. A **stream processor** (Flink-style) aggregates into 1-minute tumbling windows keyed by ad, emitting `(ad_id, minute, count)` rows.
16 + 4. Results land in an OLAP store (ClickHouse, Druid, or Cassandra keyed by ad and minute) that serves dashboards.
17 +
18 + ## Correctness details
19 +
20 + - **Event time, not processing time**: aggregate by when the click happened. Use **watermarks** to decide when a window is complete, and allow a lateness budget (say 2 minutes) for stragglers; very late events go to a correction path.
21 + - **Dedup**: keep recently seen click IDs per window in state (or a Bloom filter backed by an exact check) and drop repeats.
22 + - **Exactly-once output**: the processor checkpoints its state and Kafka offsets together and writes results idempotently (upsert by `ad_id, minute`).
23 +
24 + ## Reconciliation
25 +
26 + Also write raw events to object storage. A nightly batch job recomputes counts from the raw log and compares them with the streaming results; billing uses the reconciled numbers. This is the classic Lambda-style safety net, and interviewers like hearing it for money-related systems.
27 +
28 + ## Hot ads
29 +
30 + A viral ad can overload one partition. Split its key (`ad_id#0..7`), aggregate the pieces, then sum them in a second stage.
31 +
32 + ---
33 + *Disclosure: I'm with [Linkjob AI](https://www.linkjob.ai/), which offers AI help for live interviews.*