System Design Interview: Designing an Ad Click Aggregator
Advertisers are billed on click counts, so the numbers must be correct, and they want them within a minute.
Requirements
- Ingest click events (ad_id, user, timestamp, ...) at ~50k per second, peaks of 5x.
- Query clicks per ad per minute for the last N minutes, and top-K ads per minute.
- Counts must be accurate for billing; late and duplicate events happen.
Pipeline
- Click endpoint logs the event and redirects the user to the advertiser. Each click gets a unique ID (signed into the ad URL) so duplicates can be detected.
- Events go into Kafka, partitioned by
ad_id. - A stream processor (Flink-style) aggregates into 1-minute tumbling windows keyed by ad, emitting
(ad_id, minute, count)rows. - Results land in an OLAP store (ClickHouse, Druid, or Cassandra keyed by ad and minute) that serves dashboards.
Correctness details
- Event time, not processing time: aggregate by when the click happened. Use watermarks to decide when a window is complete, and allow a lateness budget (say 2 minutes) for stragglers; very late events go to a correction path.
- Dedup: keep recently seen click IDs per window in state (or a Bloom filter backed by an exact check) and drop repeats.
- Exactly-once output: the processor checkpoints its state and Kafka offsets together and writes results idempotently (upsert by
ad_id, minute).
Reconciliation
Also write raw events to object storage. A nightly batch job recomputes counts from the raw log and compares them with the streaming results; billing uses the reconciled numbers. This is the classic Lambda-style safety net, and interviewers like hearing it for money-related systems.
Hot ads
A viral ad can overload one partition. Split its key (ad_id#0..7), aggregate the pieces, then sum them in a second stage.
Disclosure: I'm with Linkjob AI, which offers AI help for live interviews.