Overview
The article discusses Brooklin, a data ingestion service developed by LinkedIn to facilitate streaming data from various sources to multiple destinations. It highlights the challenges of building sustainable data pipelines and how Brooklin addresses these issues through Change Data Capture (CDC) and seamless integration across different environments.
What You'll Learn
1
How to use Brooklin for Change Data Capture in your applications
2
Why using a dedicated service like Brooklin simplifies data streaming across environments
3
How to provision and manage datastreams dynamically with Brooklin
Prerequisites & Requirements
- Understanding of data streaming concepts and architectures
- Familiarity with Kafka and data ingestion tools(optional)
Key Questions Answered
What is Brooklin and how does it facilitate data streaming?
Brooklin is a data ingestion service that streams data from various sources like Kafka and databases to multiple destinations. It simplifies the process of building data pipelines by allowing developers to focus on processing events rather than managing the infrastructure needed to move data.
How does Brooklin implement Change Data Capture (CDC)?
Brooklin uses Change Data Capture to stream changes from databases without impacting online queries. It creates a change stream that applications can consume, allowing them to access database changes efficiently and in real-time.
What are the key components of Brooklin's architecture?
Brooklin's architecture consists of multiple server instances coordinated by Zookeeper, which manages task assignments and metadata. Key components include datastream management APIs, connectors for data ingestion, and producers that publish data to destination streams.
What capabilities does Brooklin offer for data streaming?
Brooklin supports pluggable sources and destinations, dynamic provisioning of datastreams, multitenancy, and guarantees for at-least-once delivery. It also integrates with Apache Samza for processing and supports various serialization formats.
Key Statistics & Figures
Messages processed per day
12.5 billion
Brooklin serves this volume of messages as it streams events from over 150 sources.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Messaging System
Kafka
Used as the default platform for message delivery in Brooklin.
Coordination Service
Zookeeper
Manages coordination between Brooklin server instances and stores metadata.
Stream Processing
Apache Samza
Integrates with Brooklin to allow existing Samza consumers and producers to be used as connectors.
Key Actionable Insights
1Implementing Brooklin can significantly reduce the complexity of data streaming across different environments.By using Brooklin, developers can abstract away the complexities of data movement, allowing them to focus on business logic rather than infrastructure management.
2Utilizing Change Data Capture with Brooklin can enhance application performance by minimizing resource contention.By consuming from a change stream instead of directly from databases, applications can avoid impacting online query performance, which is crucial for real-time applications.
3Dynamic provisioning of datastreams in Brooklin allows for rapid adjustments to data pipelines.This capability is essential in environments where data requirements change frequently, enabling developers to adapt quickly without significant overhead.
Common Pitfalls
1
Failing to account for the complexities of data serialization and access mechanisms across different environments.
This can lead to increased development time and difficulties in maintaining data integrity. It's crucial to plan for these differences when designing data pipelines.
Related Concepts
Data Streaming
Change Data Capture
Distributed Systems
Apache Kafka
Stream Processing