Capturing Every Change From Shopify’s Sharded Monolith

Shopify is moving from existing query-based CDC (Change-Data Capture) tooling (Longboat) to an immutable, append-only, log-based mechanism. This has required a complete re-platforming, involving a shift to new, open source technology such as Kafka Connect and Debezium.

Shopify logo
Shopify
•
John Martin
•25 min read•intermediate•
--
•View Original
ApacheApache KafkaApache SparkGolangGoogle CloudGoogle Cloud StorageKubernetesMonolithMySQL

Overview

The article discusses Shopify's evolution from a traditional data warehouse to a more dynamic data platform using Change Data Capture (CDC) and event streaming technologies. It highlights the challenges faced with their existing data extraction methods and the implementation of a new CDC platform using Debezium and Apache Kafka to improve data freshness and reliability.

What You'll Learn

1

How to implement a log-based Change Data Capture system using Debezium and Kafka

2

Why event streaming is essential for real-time data processing

3

When to use soft deletes versus hard deletes in database design

Prerequisites & Requirements

  • Understanding of data warehousing concepts and event-driven architecture
  • Familiarity with Apache Kafka and Debezium(optional)

Key Questions Answered

What are the main challenges with Shopify's previous data extraction methods?
Shopify faced issues with slow query times, limited support for structured datasets, and the inability to capture hard deletes, which hindered their analytics capabilities. These challenges necessitated the development of a new Merchant Analytics Platform and the eventual shift to a log-based Change Data Capture system.
How does Debezium improve data extraction for Shopify?
Debezium enhances data extraction by capturing changes directly from the database logs and writing them to Kafka, allowing for near real-time data availability. This approach resolves issues related to data freshness and accuracy, enabling better analytics and reporting.
What is the role of Apache Kafka in Shopify's data platform?
Apache Kafka serves as the backbone for Shopify's new data platform, facilitating the transmission and storage of Change Data Capture records. It allows for a standardized API for consumption and supports the integration of various data sources into a unified system.
What are the performance metrics achieved with the new CDC platform?
The new CDC platform has achieved a p99 latency of less than 10 seconds from MySQL insertion to data availability in Kafka, significantly improving the speed at which data can be processed and utilized for analytics.

Key Statistics & Figures

Data processing rate
65,000 records/second
During the Black Friday Cyber Monday weekend in 2020, Shopify processed this average rate, with spikes up to 100,000 records/second.
CDC data storage
400TB+
Shopify currently stores over 400TB of Change Data Capture data in their Kafka cluster.
p99 latency
less than 10 seconds
This is the latency from MySQL insertion to data availability in Kafka, indicating the efficiency of the new CDC platform.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
Apache Kafka
Used for transmitting and storing Change Data Capture records.
Backend
Debezium
Serves as the Change Data Capture tool to capture database changes and write them to Kafka.
Storage
Google Cloud Storage
Used for storing large records that exceed Kafka's size limits.

Key Actionable Insights

1
Implement a log-based Change Data Capture system to enhance data freshness and reliability.
Transitioning to a CDC system allows organizations to capture real-time changes in their databases, which is crucial for analytics and decision-making processes.
2
Utilize Apache Kafka for event streaming to unify data extraction mechanisms.
By leveraging Kafka, businesses can streamline their data processing workflows, enabling faster and more efficient access to real-time data.
3
Consider the implications of soft deletes versus hard deletes in your database design.
Understanding the differences between these deletion methods can help prevent data integrity issues and improve the performance of your database.

Common Pitfalls

1
Relying solely on updated_at fields for data extraction can lead to missed updates.
If updates do not modify the updated_at field, those changes will not be captured, resulting in incomplete data.
2
Using soft deletes can lead to performance issues in large tables.
Soft deletes can cause tables to accumulate many 'deleted' rows, which can degrade query performance and complicate data integrity.

Related Concepts

Change Data Capture
Event Streaming
Data Warehousing
Real-time Analytics