Overview
This article discusses the challenges and solutions for estimating the cardinality of set intersections at scale using Apache Pinot and Theta Sketches. It highlights the improvements made in data processing speed and accuracy, ultimately enhancing audience reach estimation for LinkedIn's advertising solutions.
What You'll Learn
1
How to use Theta Sketches for set intersection cardinality estimation
2
Why reducing data size can improve data freshness in real-time applications
3
How to enhance query execution in Apache Pinot for complex aggregations
Prerequisites & Requirements
- Understanding of set operations and cardinality estimation
- Familiarity with Apache Pinot and Theta Sketches(optional)
Key Questions Answered
How does Apache Pinot improve audience reach estimation?
Apache Pinot enhances audience reach estimation by utilizing Theta Sketches for efficient cardinality approximations, allowing for faster data processing and more accurate insights. This approach reduced data size by approximately 80%, enabling more frequent updates and improved data freshness for advertisers.
What are the challenges of using count-distinct aggregation in large datasets?
The main challenges include limited optimization opportunities due to highly dimensional, multi-valued columns, and the need for daily data regeneration, which increases latency. These factors can lead to data freshness issues and inefficiencies in processing large volumes of data.
What is the role of Theta Sketches in set intersection operations?
Theta Sketches are used to approximate the cardinality of set operations, including intersections. They provide a more efficient method for estimating unique counts compared to traditional methods, particularly in scenarios with large datasets and complex conditions.
How did the schema modification impact data size and performance?
The schema modification reduced data size by nearly 88%, from close to 1 TB down to 120 GB. This change allowed for more frequent data refreshes and improved performance, as the new schema scales better with the number of distinct dimension values.
Key Statistics & Figures
Data size reduction
88%
The new schema reduced the data size from close to 1 TB to 120 GB, enabling more frequent updates.
Latency improvement
95th percentile read latency of 800ms
This latency was achieved after optimizations within Pinot's implementation, significantly improving performance.
Technologies & Tools
Database
Apache Pinot
Used for real-time distributed OLAP data storage and processing.
Data Structure
Theta Sketch
Utilized for approximating cardinality of set operations, including intersections.
Key Actionable Insights
1Implementing a new schema that utilizes serialized sketches can drastically reduce data size and improve processing speed.This approach allows for more efficient storage and retrieval of data, which is crucial for applications that require real-time insights, such as advertising platforms.
2Utilize Theta Sketches for approximating cardinality in set operations to enhance accuracy in data analytics.Theta Sketches support various set operations, making them ideal for scenarios where traditional methods fall short, particularly in high-dimensional datasets.
3Enhance query execution capabilities in Apache Pinot to support complex aggregations and set operations.By modifying the query execution engine, users can perform more sophisticated analyses, leading to better insights and decision-making in data-driven applications.
Common Pitfalls
1
Relying solely on count-distinct aggregation for large datasets can lead to performance bottlenecks and data freshness issues.
This occurs because count-distinct operations are computationally expensive, especially as data volume increases. It's essential to explore approximation techniques like Theta Sketches to maintain performance.
Related Concepts
Data Processing At Scale
Cardinality Estimation Techniques
Real-time Data Analytics