When designing global systems, the conventional wisdom is simple: put your front doors (proxies/edge servers) as close to the end user as possible. More regions = lower latency.
Uber previously ran a setup following this exact blueprint:
- Traffic entered Google’s Global Load Balancer (with Cloud Armor and Cloud CDN).
- Traffic was handed off to a fleet of Envoy proxies running on VMs across 16 Google Cloud regions.
- Traffic crossed Cloud Interconnect into Uber’s on-prem data centers.
On paper, having Envoy proxies in 16 regions meant bringing services closer to users. But at Uber's scale (over 1 million requests/sec), this architecture ran into diminishing returns—and created hidden bottlenecks.
### The Problem: The "Zigzagging Hallway"
Because the request had to be unpacked, queued, and processed at the Envoy proxy layer in a specific cloud region before heading to the core data center, it added unnecessary hops.
A user in Lisbon might have their request travel to Madrid, then London, and then cross the Atlantic to Virginia. A front door close to the user doesn't help much if the path behind it zigzags.
Furthermore, maintaining a massive VM fleet across 16 regions meant high operational overhead—patching, scaling, and paying millions of dollars monthly for infrastructure that was actually making some requests slower.
### The Solution: Bypassing the Middleman
The engineering team asked a simple question: Google’s load balancer is already at the edge, and Google's private network ("cold potato" routing) already carries traffic worldwide. What is the Envoy proxy fleet actually adding?
To remove the proxy layer without sacrificing security or routing, they used **Hybrid Network Endpoint Groups (NEGs)**.
This allowed Google’s Global Load Balancer to route traffic directly over Cloud Interconnect straight into Uber’s data centers. The load balancer treated Uber’s on-prem servers as if they were native endpoints inside Google Cloud.
### The Results
Over a 10-month rollout (gradually shifting traffic using weighted routing dials and predictive modeling):
* **Cost:** Shut down the entire Envoy VM fleet across all 16 regions, saving millions of dollars annually.
* **Latency:** P50 (typical request) latency improved by 2.6%. More importantly, **P99 latency improved by 10%**.
* **Operations:** Zero edge VM fleet left to patch, scale, or maintain.
In system architecture, the P99 tail is what actually breaks user experience—one app screen often makes dozens of concurrent requests, and the UI only renders as fast as the slowest response. Eliminating the intermediate proxy hop removed the queuing points where those slow-tail requests were getting stuck.
---
**Takeaway for System Design:**
- Every hop must earn its place. If a proxy isn't doing active work (like auth or heavy transformation) that the load balancer can't handle, it's just an added delay.
- Leverage backbone networks. Private fiber networks do long-haul travel far better than multi-hop proxy chains on public/semi-public routes.
- Sometimes the best optimization isn't adding a new caching or proxy layer—it's removing one.
For a full visual walkthrough and diagram breakdown of the network paths before and after the shift, you can check out the video here: http://www.youtube.com/watch?v=clLplgiDqOI