How Uber Conquered Database Overload: The Journey from Static Rate-Limiting to Intelligent Load Management

Key Actionable Insights

1

Place overload protection as close to the storage layer as possible rather than at the routing or gateway layer. Uber's initial approach of rate limiting in the stateless query engine failed because it couldn't accurately track real-time health across thousands of partitions, requiring expensive Redis calls and introducing a new failure point.

In stateful systems, the storage layer has the most accurate picture of actual resource usage, making it the ideal location for admission control decisions. This eliminates the need to propagate health information across stateless nodes.

2

Use concurrency (in-flight operations) rather than QPS as your primary overload signal for database systems. Following Little's Law (Concurrency = Throughput × Latency), concurrency directly reflects actual system load and maps closely to resource usage in stateful systems, while QPS-based limiting fails to account for workload variability.

A full table scan query that returns one row and a simple point lookup may have the same QPS cost but vastly different resource consumption. Concurrency captures this difference naturally.

3

Implement priority-aware load shedding that drops low-priority traffic (batch jobs, pipelines, garbage collection) before user-facing requests. At Uber, many overloads stemmed from asynchronous jobs that shouldn't have the same survivability as ride requests or real-time pricing queries.

Use a tiering model (e.g., t0 for critical infrastructure, t1 for user-facing online traffic, t5 for least important) and assign default priorities based on calling service when no explicit priority is present.

4

Replace static rate limiting configurations with PID-based control loops that dynamically adjust queue timeouts and inflight limits. CoDel's fixed, static wait times led to thundering herd problems where simultaneously rejected requests all retried at once, triggering repeated cycles of overload and rejection.

PID regulation acts like a dimmer switch rather than a hammer, incorporating system history and directional trends for smooth, sustained overload control. Uber saw significantly reduced premature shedding after adopting this approach.

5

Combine system-wide overload protection with per-tenant fairness enforcement operating in parallel. During system-wide stress, shed by priority; when a single noisy actor hogs resources without triggering global overload, use per-tenant concurrency limits (like Uber's Scorecard) that work independently of system load.

This dual approach ensures that both systemic overload and localized noisy neighbor issues are handled with precision, reducing blast radius during incidents.

6

Design your load management system as a pluggable platform with a Bring Your Own Signal (BYOS) architecture rather than building it for a single overload signal. Uber's initial design was excellent for concurrency-based shedding but wasn't extensible to new signals like follower commit index lag or memory pressure.

As systems grow, new overload patterns will inevitably emerge. A pluggable framework lets teams embed new overload signals and route them to the right control path without redesigning the core engine.