Overview
The article discusses Spotify's approach to reliably exporting Cloud Pub/Sub streams to Cloud Storage, detailing the architecture and processes involved in handling over 100 billion events generated daily. It emphasizes the importance of deduplication, timely delivery, and the use of various services to ensure data consistency and operational reliability.
What You'll Learn
1
How to implement a reliable event delivery system using Cloud Pub/Sub
2
Why immutability of data buckets is crucial for data consistency
3
How to design a deduplication process for event data
Prerequisites & Requirements
- Understanding of event-driven architectures and data processing concepts
- Familiarity with Cloud Pub/Sub and Cloud Storage(optional)
Key Questions Answered
How does Spotify ensure reliable delivery of events from Cloud Pub/Sub to Cloud Storage?
Spotify uses a dedicated Extract Transform Load (ETL) process that consumes data from Cloud Pub/Sub streams, groups events into hourly buckets, deduplicates them, and delivers them to Cloud Storage. This process involves multiple services like Consumer, Completionist, and Deduper to manage data consistency and operational reliability.
What are skewed events and how are they handled?
Skewed events are late-arriving data that cannot be included in closed hourly buckets. They are written to pending hourly buckets until they can be processed, ensuring that all data is accounted for and maintaining the integrity of the event delivery system.
What role does the Completionist service play in the data delivery process?
The Completionist service tracks all intermediate files written by the Consumer service to determine when an hourly bucket can be safely closed. It uses heuristics based on the timing of file writes and the status of unacknowledged messages in Cloud Pub/Sub to make this determination.
How does Spotify handle deduplication of event data?
Deduplication is managed by the Deduper batch job, which runs for each hourly bucket to process intermediate files. It queries the Completionist service to identify which files belong to the bucket and ensures that only unique events are retained in the final output.
Key Statistics & Figures
Daily events generated by Spotify users
100 billion
This volume underscores the scale at which Spotify operates and the need for a robust event delivery system.
Number of different event types generated
300
This diversity in event types necessitates a flexible and scalable architecture for data processing.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Cloud Pub/Sub
Used for publishing and consuming events generated by Spotify users.
Storage
Cloud Storage
Primary storage solution for delivered event data.
Data Processing
Dataflow
Initially considered for ETL processes but later replaced with custom microservices.
Backend
Apollo
Framework used to build the Consumer and Completionist services.
Deployment
Helios
Used for deploying Apollo services.
Data Processing
Dataproc
Used for executing Deduper jobs.
Scheduling
Styx
Used for scheduling batch jobs in the data processing pipeline.
Database
Cloud SQL
Used to persist the state of the Completionist service.
Key Actionable Insights
1Implementing an ETL process tailored for each event type can enhance scalability and maintainability.By isolating event types in their own ETL processes, Spotify can easily scale as new event types are introduced without complicating the overall system.
2Utilizing stateless services for event consumption allows for efficient autoscaling.Spotify's Consumer service is designed to be stateless, enabling the use of managed instance groups that can scale based on CPU usage, optimizing resource utilization.
3Monitoring and operational reliability are critical in data processing systems.Spotify faced challenges with Dataflow's monitoring capabilities, leading them to build custom microservices that integrate better with their existing operational tools.
Common Pitfalls
1
Relying solely on a single data processing tool can lead to operational challenges.
Spotify initially faced issues with Dataflow's monitoring and deployment capabilities, prompting them to develop a more tailored solution using microservices.
2
Neglecting the importance of deduplication can result in data inconsistencies.
Without a robust deduplication process, duplicate events could skew analytics and reporting, leading to inaccurate insights.
Related Concepts
Event-driven Architecture
Data Processing Pipelines
Microservices Architecture
Scalability In Cloud Environments