“Exactly-once delivery” is one of the most requested guarantees in distributed systems, and one of the most misunderstood. Uber’s ad-billing pipeline reveals the uncomfortable truth: it doesn’t exist — and you build it anyway, out of two things that do.
Uber bills advertisers for impressions and clicks. Count an event twice and you over-charge a customer; miss one and you lose revenue. The pipeline processes these at enormous throughput, and the requirement is absolute: each billable event counts exactly once. The trouble is that the network underneath offers no such promise.
When one system sends a message to another over a network, it gets one of three guarantees. At-most-once: send and pray — might be lost. At-least-once: keep retrying until acknowledged — never lost, but might arrive twice. Exactly-once: the holy grail — never lost, never duplicated.
Here's the catch they don't put on the box: true exactly-once delivery is impossible over an unreliable network, because the sender can never be sure its message (or the acknowledgment) wasn't lost, so it must sometimes retry — risking a duplicate. What you can actually build is effectively-once: deliver at-least-once, then make duplicates harmless by detecting and discarding them.
Two honest pieces, one useful lie
Uber's pipeline gets at-least-once delivery the normal way (retries), then guarantees correctness by making the destination idempotent: every event carries a unique ID (a UUID), and the sink — Apache Pinot — uses upserts so a re-delivered event with a known ID overwrites rather than double-counts. At-least-once delivery plus idempotent dedup equals effectively-once billing. The “exactly-once” guarantee is assembled, not delivered.
The supporting cast matters. Flink transactional producers with Kafka read_committed consumers and ~2-minute checkpoints give consistent processing boundaries. The Chaperone audit system independently counts records at four pipeline tiers and reconciles them — a separate watchdog that proves the money math rather than assuming it. And uForwarder, a consumer proxy, decouples Kafka's partition count from processing concurrency, so Uber can scale workers without the painful re-partitioning that normally ties consumer parallelism to topic layout.
Worth knowing
The Chaperone idea is the underrated lesson: for anything involving money, don't just build correctness — continuously verify it with an independent system that counts the same thing a different way. The pipeline guarantees effectively-once; Chaperone catches the day the guarantee quietly breaks. Trust, but reconcile.
The gap it reveals
Half the engineers in any room will confidently say “Kafka gives exactly-once.” The senior truth is subtler: exactly-once delivery is unachievable, and what systems actually provide is at-least-once plus idempotent dedup (“effectively-once”). Knowing that distinction — and that you need a UUID + upserting sink, not a magic flag — is the difference between repeating a marketing phrase and designing a billing pipeline.
In the interview room
If your design counts money or events, expect: “how do you guarantee no double-counting?” Don't say “exactly-once delivery.” Say: “at-least-once delivery with idempotent dedup at the sink via a per-event ID, plus an independent audit to reconcile counts.” That phrasing tells the interviewer you understand what is and isn't possible over a network.
The reframe
The most dangerous guarantees are the ones people believe in literally. “Exactly-once” is a useful outcome built from an honest delivery model and a deduplicating destination — never a property of the wire itself. Uber's pipeline is a masterclass in not wishing away the network's limits, but designing correctness on top of them — and then auditing to be sure.
You can't deliver exactly once. You can make the second delivery not matter.
Primary source →
uber.com — Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot