Overview
DataHub is a generalized metadata search and discovery tool developed by LinkedIn to enhance the productivity of data teams. The article discusses the evolution from WhereHows to DataHub, highlighting its architecture, features, and the lessons learned during its development.
What You'll Learn
1
How to implement a push-based approach for metadata collection
2
Why a generalized architecture is more scalable than a specific one
3
How to utilize Pegasus for metadata modeling
4
When to use Kafka for metadata ingestion
Prerequisites & Requirements
- Understanding of metadata concepts and data architecture
- Familiarity with REST APIs and Kafka(optional)
Key Questions Answered
What are the main features of DataHub?
DataHub features a modular UI, a generalized metadata architecture, and supports metadata ingestion through APIs and Kafka streams. It allows users to search, browse, and view/edit metadata, enhancing the discovery of data assets across LinkedIn.
How does DataHub improve metadata collection compared to WhereHows?
DataHub adopts a push-based approach for metadata collection, allowing individual metadata providers to push updates via APIs. This contrasts with WhereHows, which relied on a pull-based method, leading to scalability issues.
What technologies are used in the DataHub architecture?
DataHub utilizes Ember Framework for the frontend, Play for the backend, and Pegasus for metadata modeling. It also employs Kafka for streaming metadata ingestion and Espresso for document-oriented queries.
What challenges does DataHub address in metadata management?
DataHub addresses challenges such as modeling diverse metadata types, ingesting large volumes of metadata, serving complex queries, and indexing metadata efficiently. These improvements enhance the overall productivity of data teams.
Key Statistics & Figures
Number of metadata records stored
tens of millions
DataHub currently indexes a vast amount of metadata across various entities, enhancing data discoverability.
Number of different entities supported
19
These entities include datasets, metrics, jobs, charts, and AI features, showcasing the breadth of DataHub's capabilities.
Number of LinkedIn employees using DataHub weekly
1,500
This indicates the tool's adoption and importance within LinkedIn's data ecosystem.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Frontend
Ember Framework
Used for building the DataHub web application.
Backend
Play
Serves as the middle tier for DataHub.
Data Modeling
Pegasus
Utilized for general-purpose metadata modeling.
Stream Processing
Kafka
Facilitates metadata ingestion through streaming.
Database
Espresso
Used for document-oriented storage and queries.
Key Actionable Insights
1Implement a push-based metadata collection strategy to ensure timely updates and scalability.This approach allows metadata providers to send updates directly, reducing the complexity of maintaining crawlers and improving the accuracy of the metadata reflected in the system.
2Utilize Pegasus for metadata modeling to leverage its robust schema capabilities.Pegasus allows for flexible modeling of metadata while maintaining a unified view, which is essential for managing complex relationships in a distributed environment.
3Adopt a modular UI design to enhance user interaction with metadata.A modular approach allows for easy updates and customization of the UI, making it adaptable to the evolving needs of users interacting with the metadata.
Common Pitfalls
1
Relying solely on a pull-based approach for metadata collection can lead to scalability issues.
This method requires constant maintenance of crawlers, which can become cumbersome as the data ecosystem grows.
2
Not modeling relationships between metadata can hinder analytical capabilities.
Failing to represent relationships as first-class entities limits the ability to perform impact analysis and understand data lineage.
Related Concepts
Metadata Management
Data Architecture
Data Discovery Tools
Open-source Software