Overview
The article discusses the importance of data governance in large organizations like LinkedIn, emphasizing the need for effective schema annotations and automation in managing vast datasets. It explores challenges in current data governance practices and proposes a shift-left approach to embed annotations directly into data schemas.
What You'll Learn
1
How to implement schema annotations directly in data definitions
2
Why embedding governance into the data creation process enhances data quality
3
How to leverage DataHub for effective data governance
Prerequisites & Requirements
- Understanding of data governance concepts and practices
- Familiarity with DataHub and schema languages like Avro and PDL(optional)
Key Questions Answered
What are the challenges of traditional data governance methods?
Traditional data governance methods often involve attaching business metadata after datasets are indexed, leading to stale and inaccurate annotations. This can result in incorrect business metrics and erodes trust in data catalogs, causing organizations to abandon these tools despite significant investments.
How does LinkedIn approach data governance with DataHub?
LinkedIn uses DataHub to manage millions of datasets across various platforms, ensuring compliance with data privacy regulations like GDPR and CCPA. The platform automates metadata extraction and allows data stewards to attach business metadata, enhancing data governance efforts.
What is the proposed shift-left approach in data governance?
The shift-left approach suggests embedding annotations directly into the schemas of datasets during their creation and updates, similar to how documentation is integrated with code. This ensures that metadata remains accurate and up-to-date, improving data quality and governance.
Key Statistics & Figures
Number of datasets managed by LinkedIn
Millions
LinkedIn manages over two orders of magnitude more datasets than there are engineers to manage them.
Technologies & Tools
Metadata Management Platform
Datahub
Used for managing and governing datasets across various platforms.
Schema Language
Pdl
Used for service APIs at LinkedIn.
Schema Language
Avro
Used for data streaming and storage APIs at LinkedIn.
Key Actionable Insights
1Embed metadata annotations directly into data schemas to maintain accuracy.By integrating annotations at the point of data creation, organizations can ensure that metadata evolves alongside the data, reducing the risk of stale information and enhancing data governance.
2Utilize DataHub for a centralized view of metadata across datasets.DataHub's stream-based architecture allows for real-time updates to metadata, ensuring that all stakeholders have access to the most current information, which is crucial for compliance and decision-making.
3Foster a culture of data ownership among engineering teams.By merging data stewardship with data production, organizations can improve accountability and data quality, leading to more reliable data governance practices.
Common Pitfalls
1
Relying on post-hoc metadata attachment can lead to inaccuracies.
As datasets evolve rapidly, waiting for data stewards to annotate after discovery can result in outdated information, risking decision-making based on incorrect data.
Related Concepts
Data Governance
Schema Annotations
Data Privacy Regulations
Metadata Management