Overview
The article discusses how Uber utilizes Apache Pinot for low-latency offline table analytics, highlighting its capabilities in handling various use cases, including real-time and offline data ingestion. It details the ingestion process, the self-serve platform for users, and the integration of Pinot with Apache Spark for data exports.
What You'll Learn
1
How to implement offline table ingestion in Apache Pinot
2
Why Apache Pinot is suitable for low-latency analytics
3
How to use Spark for data exports from Pinot to Hive
Prerequisites & Requirements
- Understanding of Apache Pinot and its table structures
- Familiarity with Apache Spark and its ecosystem(optional)
Key Questions Answered
How does Uber utilize Apache Pinot for offline analytics?
Uber employs Apache Pinot to support over 100 low-latency analytics use cases across various business lines, leveraging its ability to handle both real-time and offline data efficiently. This includes ingesting data from sources like Apache Kafka and Apache Hive, enabling quick analytics with sub-second latencies.
What are the types of tables supported by Apache Pinot?
Apache Pinot supports three types of tables: real-time, offline, and hybrid. Real-time tables ingest data from streams like Kafka, offline tables allow for pre-built segments to be uploaded, and hybrid tables combine both real-time and offline data ingestion capabilities.
What is the process for ingesting offline tables in Pinot?
Offline tables in Pinot are ingested by uploading pre-built immutable segments through the Pinot Controller's API. Users can create segments outside Pinot using specific interfaces and then upload them to a deep-store for Pinot servers to access and query.
Key Statistics & Figures
Number of offline tables running in production at Uber
500+
This statistic highlights the scale at which Uber utilizes Apache Pinot for various analytics use cases.
Number of low-latency analytics use cases supported
100+
This showcases the versatility of Pinot in handling diverse analytics requirements across Uber's business lines.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Apache Pinot
Used for low-latency analytics and data ingestion from various sources.
Data Processing
Apache Spark
Utilized for data exports and processing tasks in conjunction with Pinot.
Streaming
Apache Kafka
Source of real-time data ingestion for Pinot.
Data Warehousing
Apache Hive
Used as a source for offline data ingestion into Pinot.
Key Actionable Insights
1Implement a self-serve platform for users to manage their offline tables in Pinot, which can streamline data ingestion processes.This approach allows users to configure their ingestion jobs without needing deep technical knowledge, thus enhancing productivity and reducing dependency on engineering teams.
2Utilize column partitioning in append tables to optimize data storage and query performance.Column partitioning can significantly reduce data volume and improve query efficiency by colocating similar data, making it easier for Pinot to perform segment pruning.
3Leverage Spark for batch data exports from Pinot to other systems like Hive to handle large datasets effectively.Using Spark's capabilities allows for efficient data handling and reduces the memory load on Pinot servers, especially during large data transfers.
Common Pitfalls
1
Overloading Pinot servers with batch read workflows can degrade performance.
It's crucial to monitor and limit read parallelism when implementing batch reads to prevent resource contention and ensure stable performance.
Related Concepts
Apache Kafka
Apache Spark
Data Ingestion Strategies
Real-time Analytics