•Anshal Shukla, Vineeth Tatipathri, Nipun Vats, Dinesh Jagannathan, Kousik Nath•19 min read•advanced•
--
•View OriginalOverview
The article discusses D3, an automated system developed by Uber to detect data drifts in datasets, which is crucial for maintaining data quality and ensuring the performance of machine learning models. It highlights the challenges of manual data monitoring and the significant improvements achieved through the implementation of D3.
What You'll Learn
1
How to implement automated monitoring for data quality in datasets
2
Why detecting data drift is critical for machine learning model performance
3
How to utilize D3 for faster time to detection of data issues
Prerequisites & Requirements
- Understanding of data quality concepts
- Familiarity with Spark and Hive(optional)
Key Questions Answered
What is the purpose of the D3 framework?
The D3 framework is designed to automate the monitoring of data quality by detecting data drifts in datasets, which helps improve the performance of machine learning models and ensures accurate data for decision-making.
How does D3 improve the time to detect data issues?
D3 has reduced the average time to detect data issues from 45 days to 2 days, achieving a more than 20X improvement in time to detection, allowing for quicker responses to data quality problems.
What types of monitors does D3 support?
D3 supports various monitors including Null percentage, False percentage, and Percentile checks, which can be configured based on the specific needs of the dataset being monitored.
What challenges does D3 address in data monitoring?
D3 addresses the challenges of manual data monitoring, which is often slow and prone to errors, by providing an automated solution that detects data quality issues proactively and efficiently.
Key Statistics & Figures
Time to detect data issues
2 days
Reduced from an average of 45 days after implementing D3.
Accuracy of detected issues
95.23%
Achieved on fact tables using D3.
Cost of computing per dataset
$0.01
Reduced from $1.5 due to query optimization in the D3 framework.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Data Monitoring Framework
D3
Used for automated detection of data drifts and monitoring data quality.
Data Processing
Spark
Used for executing compute jobs against datasets in the D3 framework.
Data Storage
Hive
Used for persisting computed stats from D3.
Key Actionable Insights
1Implement D3 to automate data quality monitoring in your datasets.Using D3 can significantly reduce the time to detect data issues, allowing teams to respond quickly and maintain the integrity of machine learning models.
2Utilize dimension-based monitoring to catch data issues specific to certain segments.By monitoring data quality across dimensions such as city or app version, you can identify localized issues that may not be apparent in overall data quality metrics.
3Leverage the automated onboarding feature of D3 for efficient setup.This feature minimizes manual input and accelerates the process of getting datasets monitored, saving valuable time for data teams.
Common Pitfalls
1
Over-reliance on manual monitoring processes can lead to delayed detection of data issues.
Manual processes are often slow and prone to human error, which can result in significant revenue loss if data issues go unnoticed for extended periods.
Related Concepts
Data Quality Monitoring
Machine Learning Model Performance
Automated Data Processing Frameworks