# System Design Interview: Designing an Ad Click Aggregator Advertisers are billed on click counts, so the numbers must be correct, and they want them within a minute. ## Requirements - Ingest click events (ad_id, user, timestamp, ...) at ~50k per second, peaks of 5x. - Query clicks per ad per minute for the last N minutes, and top-K ads per minute. - Counts must be accurate for billing; late and duplicate events happen. ## Pipeline 1. **Click endpoint** logs the event and redirects the user to the advertiser. Each click gets a unique ID (signed into the ad URL) so duplicates can be detected. 2. Events go into **Kafka**, partitioned by `ad_id`. 3. A **stream processor** (Flink-style) aggregates into 1-minute tumbling windows keyed by ad, emitting `(ad_id, minute, count)` rows. 4. Results land in an OLAP store (ClickHouse, Druid, or Cassandra keyed by ad and minute) that serves dashboards. ## Correctness details - **Event time, not processing time**: aggregate by when the click happened. Use **watermarks** to decide when a window is complete, and allow a lateness budget (say 2 minutes) for stragglers; very late events go to a correction path. - **Dedup**: keep recently seen click IDs per window in state (or a Bloom filter backed by an exact check) and drop repeats. - **Exactly-once output**: the processor checkpoints its state and Kafka offsets together and writes results idempotently (upsert by `ad_id, minute`). ## Reconciliation Also write raw events to object storage. A nightly batch job recomputes counts from the raw log and compares them with the streaming results; billing uses the reconciled numbers. This is the classic Lambda-style safety net, and interviewers like hearing it for money-related systems. ## Hot ads A viral ad can overload one partition. Split its key (`ad_id#0..7`), aggregate the pieces, then sum them in a second stage. --- *Disclosure: I'm with [Linkjob AI](https://www.linkjob.ai/), which offers AI help for live interviews.*