| @@ -0,0 +1,33 @@ | |||
| 1 | + | # System Design Interview: Designing an Ad Click Aggregator | |
| 2 | + | ||
| 3 | + | Advertisers are billed on click counts, so the numbers must be correct, and they want them within a minute. | |
| 4 | + | ||
| 5 | + | ## Requirements | |
| 6 | + | ||
| 7 | + | - Ingest click events (ad_id, user, timestamp, ...) at ~50k per second, peaks of 5x. | |
| 8 | + | - Query clicks per ad per minute for the last N minutes, and top-K ads per minute. | |
| 9 | + | - Counts must be accurate for billing; late and duplicate events happen. | |
| 10 | + | ||
| 11 | + | ## Pipeline | |
| 12 | + | ||
| 13 | + | 1. **Click endpoint** logs the event and redirects the user to the advertiser. Each click gets a unique ID (signed into the ad URL) so duplicates can be detected. | |
| 14 | + | 2. Events go into **Kafka**, partitioned by `ad_id`. | |
| 15 | + | 3. A **stream processor** (Flink-style) aggregates into 1-minute tumbling windows keyed by ad, emitting `(ad_id, minute, count)` rows. | |
| 16 | + | 4. Results land in an OLAP store (ClickHouse, Druid, or Cassandra keyed by ad and minute) that serves dashboards. | |
| 17 | + | ||
| 18 | + | ## Correctness details | |
| 19 | + | ||
| 20 | + | - **Event time, not processing time**: aggregate by when the click happened. Use **watermarks** to decide when a window is complete, and allow a lateness budget (say 2 minutes) for stragglers; very late events go to a correction path. | |
| 21 | + | - **Dedup**: keep recently seen click IDs per window in state (or a Bloom filter backed by an exact check) and drop repeats. | |
| 22 | + | - **Exactly-once output**: the processor checkpoints its state and Kafka offsets together and writes results idempotently (upsert by `ad_id, minute`). | |
| 23 | + | ||
| 24 | + | ## Reconciliation | |
| 25 | + | ||
| 26 | + | Also write raw events to object storage. A nightly batch job recomputes counts from the raw log and compares them with the streaming results; billing uses the reconciled numbers. This is the classic Lambda-style safety net, and interviewers like hearing it for money-related systems. | |
| 27 | + | ||
| 28 | + | ## Hot ads | |
| 29 | + | ||
| 30 | + | A viral ad can overload one partition. Split its key (`ad_id#0..7`), aggregate the pieces, then sum them in a second stage. | |
| 31 | + | ||
| 32 | + | --- | |
| 33 | + | *Disclosure: I'm with [Linkjob AI](https://www.linkjob.ai/), which offers AI help for live interviews.* | |
linkjobai / System Design Interview: Designing an Ad Click Aggregator
Last active 5 days ago
Interview prep notes
linkjobai revised this gist 5 days ago · 9e3e99d
1 file changed, 33 insertions