How Uber Uses Apache Spark

This article details how Uber optimized Apache Hadoop's Distcp (Distributed Copy) tool to scale their data replication infrastructure from handling 250 TB to petabytes of daily data movement.

This article details how Uber built and scaled Apache Hudi to power one of the world's largest data lakes, managing 19,500 datasets with trillions of records across a multi-hundred-petabyte reposit...

The article discusses Uber's transition from traditional keyword-based search using Apache Lucene to implementing semantic vector search with Amazon OpenSearch.

This article discusses how Uber utilizes a pull-based ingestion model in OpenSearch™ to effectively index streaming data.

This article discusses Uber's transition from batch to streaming data ingestion using Apache Flink, which significantly enhances data freshness and operational efficiency.

The article discusses Uber's implementation of I/O observability for its massive petabyte-scale data lake, focusing on the challenges and solutions in monitoring data access patterns across its hyb...

Uber's migration from Spark 2. 4 to Spark 3. 3 involved upgrading over 2 million Spark applications, utilizing innovative automation tools like Iron Dome.

This article discusses the architecture and implementation of Uber's HiveSync, a critical service for data replication across its massive data lake.

This article discusses the development and implementation of forecasting models aimed at improving driver availability at airports, which are critical to Uber's ridesharing ecosystem.

The article discusses the evolution of Uber's Search Platform, highlighting its transition from Elasticsearch to an in-house solution called Sia, and ultimately to the adoption of OpenSearch.

This article details Uber's migration from Apache Hive to Apache Spark SQL for ETL workloads, highlighting the motivations behind the transition, the architecture involved, and the challenges faced...

The article discusses Uber's implementation of a configuration-driven archival and retrieval framework designed to manage vast amounts of regulatory data efficiently.

The article discusses Uber's migration of large-scale interactive compute workloads from Peloton to Kubernetes, focusing on minimizing disruption while enhancing resource management and cloud readi...

This article discusses Uber's implementation of elastic resource management on Kubernetes, focusing on enhancements made to support Ray-based job management.

Uber's blog post discusses their migration of machine learning workloads to Kubernetes using Ray, detailing the challenges faced with their previous setup and the improvements achieved with the new...

The article discusses how Uber utilizes Ray®, a general compute engine for Python®, to enhance the efficiency of its rides business through improved machine learning model performance and optimizat...

The article discusses Uber's advanced settlement accounting system, which is crucial for managing financial transactions involving payment service providers (PSPs).

The article discusses how Uber optimizes the training of Large Language Models (LLMs) using both open-source and in-house models.

The article discusses Genie, Uber's generative AI on-call copilot designed to enhance communication and efficiency in on-call operations.

The article discusses Uber's migration of its batch data platform to the cloud, focusing on the implementation of DataMesh principles.

The article discusses Uber's upgrade of its search platform from Lucene version 7. 5. 0 to 9. 4.

The article discusses how Uber utilizes Apache Pinot for low-latency offline table analytics, highlighting its capabilities in handling various use cases, including real-time and offline data inges...

The article discusses the Sparkle framework developed by Uber to standardize modular ETL processes, enhancing developer productivity and data quality.

This article discusses Uber's migration of its Apache Hadoop-based data lake to Google Cloud Storage (GCS) and the security measures implemented during this transition.

This article discusses the modernization of Uber's logging infrastructure using CLP, focusing on the development of an end-to-end system for managing unstructured logs.

Uber is modernizing its batch data infrastructure by migrating to Google Cloud Platform (GCP) to enhance data analytics and machine learning capabilities.

This article discusses how Uber counts job participation at scale, detailing the integration of Apache Pinot™ to address challenges in data processing and analysis.

The article discusses Uber's DataK9 platform, which automates the categorization of vast amounts of data at the field level using AI/ML techniques.

The article discusses Uber's evolution in machine learning (ML) through its centralized platform, Michelangelo, highlighting its transition from predictive to generative AI.

This article details Uber's migration of over a trillion entries of ledger data from DynamoDB to LedgerStore, focusing on the challenges, strategies, and outcomes of the process.

The article discusses Uber's journey in scaling its AI/ML infrastructure, highlighting the transition from on-premise to cloud solutions, the implementation of new technologies, and the optimizatio...

DataCentral is Uber's proprietary platform designed for Big Data observability, chargeback, and governance.

uVitals is an anomaly detection and alerting system developed by Uber to enhance the reliability of its services by quickly identifying and addressing issues in multi-dimensional time series data.

The article discusses how Uber utilizes Apache Pinot for real-time analytics of mobile app crashes, enhancing their ability to detect and resolve issues quickly.

The article discusses the implementation of a Unified Session for analytical events at Uber, aimed at enhancing data consistency and analytics across various applications.

The article discusses the challenges Uber faces with increasing data storage costs and presents a solution through selective column reduction in Apache Parquet™ files.

The article discusses CheckEnv, a tool developed by Uber for fast detection of remote procedure calls (RPCs) between different environments using graph technology.

The article discusses the implementation of dynamic executor core resizing in Apache Spark to address out-of-memory (OOM) exceptions.

The article discusses the implementation of Two-Tower Embeddings (TTE) at Uber, highlighting its role in enhancing the efficiency and scalability of recommendation systems.

The article discusses Spark Analysers, a system developed by Uber to identify anti-patterns in Spark applications.

The article discusses how Uber implemented an incremental ETL process using Apache Hudi to manage its transactional data lake.

The article discusses Uber's journey in scaling the adoption of Kerberos authentication across its extensive data analytics platform.

The article discusses Uber's Machine Learning Education Program, which leverages engineering principles to scale ML education for its employees.

Uber's article discusses the implementation of a highly scalable and distributed Remote Shuffle Service (RSS) designed to enhance the efficiency of data processing in Apache Spark.

The article discusses Uber's approach to automating offline inferences using machine learning and natural language processing on support interaction data.

This article discusses the implementation of finer-grained encryption in Apache Parquet™, focusing on how it addresses data access restrictions, retention, and encryption at rest.

The article discusses DeepETA, Uber's advanced model for predicting arrival times using deep learning techniques.

The article discusses Project RADAR, an intelligent fraud detection system developed by Uber that integrates machine learning and human expertise to identify and mitigate fraudulent activities in r...

This article details Uber's migration of financial data from DynamoDB to Docstore, highlighting the challenges faced and the architectural decisions made to ensure data integrity and operational ef...

The article discusses Uber's initiatives to enhance the efficiency of its Big Data platform, focusing on cost reduction through optimizations in file formats, HDFS erasure coding, YARN scheduling i...