Apache Hudi™ at Uber: Engineering for Trillion-Record-Scale Data Lake Operations

Key Actionable Insights

1

Implement a metadata layer to eliminate filesystem-level bottlenecks for large-scale data lake operations. Storing file listings, column statistics, and bloom filters in an indexed key-value store within the table format itself allows O(1) lookups instead of expensive filesystem list operations. This is especially critical when managing thousands of partitions with millions of files.

Uber deployed their Metadata Table to over 90% of production datasets, fundamentally changing the economics of operating their data lake by removing dependence on HDFS NameNode for file listings.

2

Build native record-level indexing into your data lake storage layer rather than relying on external index services like HBase. An embedded index that maps record keys to file groups enables O(1) lookups without external operational dependencies, reducing both latency and infrastructure complexity for upsert-heavy workloads.

Uber's Record Index achieves 1-2 millisecond lookup latency per record key and supports sharding into up to 10,000 HFiles for tables exceeding 300 billion rows, enabling parallelized lookups across thousands of executors.

3

Tackle the hardest technical challenges in your data platform first rather than deferring them. Building scalable indexing, schema evolution, and mutable data support early prevents architectural dead-ends that become exponentially harder to fix as the platform grows and more dependencies accumulate.

Uber learned that while small deployments can use bloom filters and file scanning for upserts, these approaches become completely intractable at trillion-row scale. Without planning for scalable indexing from the start, no amount of compute would make such workloads viable.

4

Centralize configuration management for your data lake tables to prevent configuration drift from becoming a scaling bottleneck. What starts as a few parameter overrides per dataset quickly turns into tens of thousands of subtly different configurations, making it impossible to roll out improvements safely.

Uber's Hudi Config Store provides a controlled, globally consistent configuration layer that enables new features, compaction logic, and file-sizing improvements to be rolled out across thousands of datasets safely and predictably.

5

Treat schema evolution as a first-class concern from day one by implementing strong backward-compatibility rules, automated schema validation, and pre-change validation. At scale, a seemingly harmless field rename can break thousands of downstream jobs, and the operational cost of schema issues grows exponentially with dataset count.

Uber's schemas change constantly as product teams iterate and launch new features. Without automated guardrails, schema changes became one of the most persistent sources of operational incidents.

6

Invest in automated observability and validation tooling that continuously checks data lake consistency, rather than relying on reactive incident response. Silent issues like corrupted files, configuration errors, and open source bugs are the most dangerous because they can cascade into large-scale outages before detection.

Uber's Hudi Validation tool detects inconsistencies early, and their M3-based metrics system tracks commit latency, compaction throughput, file growth rates, and table health across all 19,500 datasets.