How Uber Scaled Data Replication to Move Petabytes Every Day

Key Actionable Insights

1

Move resource-intensive preparation tasks from the client to distributed worker nodes to eliminate shared-resource contention. When multiple threads compete for a single HDFS client's RPC lock, performance degrades dramatically at scale. By offloading Copy Listing and Input Splitting to individual Application Master containers, each job gets its own isolated HDFS client.

This approach reduced Distcp job submission latency by 90% at Uber. The principle applies broadly to any system where centralized clients become bottlenecks as parallelism increases.

2

Identify which operations can be safely parallelized by analyzing their ordering constraints. Copy Listing files can be processed in parallel because files are independent, but chunks within each file must remain ordered. Understanding these constraints allows targeted parallelization without breaking correctness.

Uber used a producer-consumer pattern with a blocking queue: multiple listing threads create file splits in parallel, while a single writer thread maintains ordering when writing to the sequence file. This achieved a 60% reduction in p99 listing latency with 6 threads.

3

Leverage in-process execution (Uber/Uber jobs pattern) for small tasks that don't justify container allocation overhead. When 52% of your distributed jobs need only a single mapper for less than 512 MB, the JVM startup and container allocation time dominates actual work time. Running these tasks within the Application Master's existing JVM eliminates this overhead entirely.

This optimization eliminated 268,000 container launches daily at Uber. Configure with mapreduce.job.ubertask.enable=true, maxmaps=1, and maxbytes=512MB to automatically route qualifying jobs.

4

Implement circuit breakers when reducing latency in distributed systems, because faster submission rates can overwhelm downstream resources. Lowering Distcp submission latency caused YARN Queue Full errors because more jobs were submitted per unit time. A circuit breaker that temporarily pauses new submissions until retries succeed prevents cascading failures.

This is a common pattern when optimizing one part of a pipeline—faster throughput upstream can create new bottlenecks downstream. Pair circuit breakers with real-time metrics to monitor and dynamically adjust queue capacity.

5

When moving long-running tasks between components in a distributed system, carefully consider heartbeat and liveness check mechanisms. Moving Copy Listing to the AM's setup phase caused timeout failures because the RM's heartbeat sender hadn't started yet, and Copy Listing could exceed the 10-minute timeout.

The fix was to move Copy Listing to the output committer's setup phase, which runs after the heartbeat sender has started. Always map task placement against the lifecycle of health-check mechanisms.

6

Add comprehensive observability metrics before and during performance optimization rollouts. Uber introduced metrics for job submission times, submission rates, Copy Listing and Copy Committer task performance, heap memory usage, and per-job copy rates. These metrics were crucial for both tuning optimizations and diagnosing production incidents.

Without detailed metrics, it would be impossible to identify that Copy Listing was responsible for 90% of job submission latency or to detect OOM issues in the Application Master. Invest in observability early to guide optimization decisions.