How Airbnb evolved our data catalog into a platform for managing and governing our data warehouse at scale.
Overview
The article discusses Metis, Airbnb's next-generation data management platform designed to empower the company to manage its complex data ecosystem at scale. It outlines the evolution of data management at Airbnb, the architecture of Metis, and the specific components that enable effective metadata management and data lineage.
What You'll Learn
1
How to implement a centralized metadata management system using Metis
2
Why metadata governance is crucial for data quality and compliance
3
How to leverage Apache Atlas for data lineage tracking
4
When to utilize Elasticsearch for search and discovery in data management
Prerequisites & Requirements
- Understanding of metadata management concepts
- Familiarity with Apache Atlas and Elasticsearch(optional)
Key Questions Answered
What is Metis and how does it improve data management at Airbnb?
Metis is Airbnb's next-generation data management platform that centralizes metadata management and data lineage tracking. It allows the company to manage millions of data assets effectively, ensuring compliance and improving data quality through structured governance and metadata capture.
How does Dataportal facilitate data discovery at Airbnb?
Dataportal serves as the user interface for Airbnb's data catalog, enabling users to search, discover, and manage data assets. It employs a GraphQL API to ensure efficient data retrieval and presents relevant metadata directly in search results to enhance user experience.
What role does the Unified Metadata Service (UMS) play in Metis?
The Unified Metadata Service (UMS) is the backend core of Metis, providing a centralized schema and GraphQL API for accessing metadata. It connects siloed metadata, enabling compliance and governance without requiring individual integrations for each metadata provider.
How does Airbnb ensure data quality and compliance using Metis?
Airbnb ensures data quality and compliance through structured governance within Metis, which includes features like audit history, validation, and approval workflows for sensitive operations. This enables effective management of critical business metadata.
Key Statistics & Figures
Number of data assets managed by Metis
millions
Metis is designed to manage a vast number of data assets across various domains at Airbnb.
Size of lineage graph in Apache Atlas
over 100 million nodes and 300 million edges
This extensive lineage graph helps track data transformations and relationships within Airbnb's data ecosystem.
Weekly data users benefiting from Metis
1000+
Metis serves critical roles across data compliance, data reliability, and data quality initiatives, impacting a large number of users weekly.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Atlas
Used for data lineage tracking within Metis.
Backend
Elasticsearch
Serves search and discovery use cases for data assets.
Database
Mysql
Stores critical metadata that needs to be centrally managed.
API
Graphql
Facilitates communication between the Dataportal and other services.
Frontend
React
Framework used to build the Dataportal UI.
Frontend
Typescript
Programming language used for developing the Dataportal.
Key Actionable Insights
1Implement a centralized metadata management system to streamline data governance processes.By centralizing metadata management, organizations can reduce redundancy and improve compliance across various data assets, making it easier to track data lineage and maintain data quality.
2Leverage Apache Atlas for effective data lineage tracking to enhance data reliability.Using Apache Atlas allows organizations to visualize and manage the relationships between data assets, which is crucial for debugging and ensuring compliance with data regulations.
3Utilize Elasticsearch for efficient search and discovery of data assets.Elasticsearch can significantly improve the speed and accuracy of data retrieval, which is essential for data consumers looking to access high-quality datasets quickly.
Common Pitfalls
1
Failing to establish clear governance for metadata management can lead to data quality issues.
Without proper governance, data assets may become unreliable, making it difficult to ensure compliance and maintain data integrity.
2
Neglecting the importance of data lineage can result in challenges during debugging and compliance audits.
Data lineage is crucial for understanding the flow of data and its transformations, and failing to track it can lead to significant operational risks.
Related Concepts
Data Governance
Data Quality Management
Metadata Management
Data Lineage Tracking