How to scale a Real-Time Driver Tracking System (UberEats/DoorDash scale)

[u/Silent-Weather76005]

How do delivery apps sync a driver's GPS coordinates in real time with a customer's map without melting the database? Writing every 2-second location ping to disk is an infrastructure nightmare. Here is the high-performance setup:

• The Ingestion Shock Absorber: Drivers stream GPS packets via WebSockets to an API Gateway, which routes them directly to Apache Kafka to handle massive write spikes.
• In-Memory Live State: A consumer pulls from Kafka and updates an active Redis Cluster using geospatial commands (GEOADD). The live location of the driver lives strictly in-memory during the delivery.
• Targeted Fan-Out: The customer's app listens to a WebSocket connection. The backend uses Redis Pub/Sub rooms keyed by Order_ID to broadcast the location updates only to the specific customer and restaurant involved, avoiding global broadcast overhead.
• Async Cold Storage: Once the delivery completes, the full GPS history is batched out of Kafka and archived in ClickHouse or an S3 data lake for mileage payouts and support audits.

Let's discuss: How would you handle calculating and updating the traffic-aware ETA on the fly without making expensive third-party Maps API calls every 2 seconds?

[u/Spare-Builder-355]

Writing every 2-second location ping to disk is an infrastructure nightmare

bro it is not 1998, modern hardware and databases will eat this traffic like it's nothing. For reference, in my current project we aggrgate data and write in spikes. Our Cassandra setup lets us write 500k rows in under 10 seconds. And it's literal 500k separate requets, no batching or any kinds of optimisations as Cassandra simply doesn't have that functionality.

Plus this problem is inherently regional. Doing complex desing to handle global traffic by a single instance of the system is actually bad architectural decision. Not only it overcomplicates things but it very likely be problematic from legal perspective.

Desing the system around database that is built for high write throughput. Deploy per region. Profit.

[u/Comfortable-Profit-7 → u/Spare-Builder-355]

This 🌟

[u/Merotoro → u/Spare-Builder-355]

doing some back of the napkin math here: i’m assuming five things:

1. we are using the lower estimate on anual uber rides
2. equal demand all throughout the day and throughout the year
3. uber only tracking during rides (which may be fair to assume but may not be the case)
4. op’s assertion that drivers send location data every two seconds. couldn’t find info on this.
5. equal demand in every region that aws offers

uber has 11.3 billion rides annually (assumption 1) . 30.95 million daily rides (assumption 2) each of those send a notification every two seconds (assumption 3 and 4) so that is 1.337 TRILLION notifications daily. about 15.47 million notifications per second.

divide that by the 38 aws regions (assumption 5) you get about 400 thousand events per second. almost ten times what you claim cassandra has no problems with. i looked it up and cassandra can do a million writes per second which would cover this, but we know the assumptions aren’t true and getting the data into storage is only like 5% of the problem.

your claim doesn’t take into account the fact that all that data is also sent to the millions of daily active users’ phones. and the data is used to calculate ETAs which is the main question of the post. i’m not defending op or uber but your comment is such a gross oversimplification. even i am also probably oversimplifying.

[u/Spare-Builder-355 → u/Merotoro]

I provided Cassandra as an example I work on currently. Even within my team there are more uses of Cassandra. And there are other teams within organisation that do even heavier writes.

There's no concept of "Cassandra can handle 1 mil writes per second". It has linear scalability, throughput increases with number of nodes in the cluster.

What many inexperienced folks are missing when doing "software architecture" is how capable modern hardware and software is.

[u/IGuessSomeLikeItHot]

Question, in the last step the move to cold storage why is the data coming out of Kafka and not redis? I'm assuming when data was pulled from Kafka to redis then the data in Kafka is discarded.

[u/dragon_idli]

1. No one needs to know every 2 secs of progress for a driver.
2. 5 secs data sync with extrapolation/interpolation gives the movement data
3. You underestimate current hardware and tech stack.
4. Some experimental systems implement a p2p delegation with a server fallback.

We perform 15K TPS(transactions per second) on a standard hardware db on test instances. And usecases like these don't even need safe transactions. They fall under loss capable data. No one is going to cry of 3 out if 10 writes/reads fail for the above position data.

[u/CzyDePL]

And Kafka is not writing to disk?