A deep dive into how Shopify Data revamped the data infrastructure powering our BFCM live map using Apache Flink.
Overview
The article discusses the redesign of Shopify's BFCM live map using Apache Flink to enhance scalability and insights. It highlights the challenges faced with the previous system and the successful implementation of a new architecture that processed a higher volume of data with improved uptime.
What You'll Learn
1
How to scale a live data visualization system using Apache Flink
2
Why using stateful stream processing can improve data handling in high-volume scenarios
3
How to implement deduplication in streaming data processing
Prerequisites & Requirements
- Understanding of streaming data processing concepts
- Familiarity with Apache Flink and Kafka(optional)
- Experience with real-time data systems(optional)
Key Questions Answered
How did Shopify scale its BFCM live map for 2021?
Shopify scaled its BFCM live map by redesigning the backend architecture using Apache Flink, which allowed for processing one percent of the event volume while filtering irrelevant checkout events. This approach enabled the system to handle peak loads of 50,000 messages per second with 100 percent uptime.
What were the main challenges with the previous BFCM live map model?
The previous BFCM live map model faced challenges such as being unable to handle the increased volume from over 1.7 million merchants, leading to concerns about backend architecture overload and inefficiencies in message queuing using Redis.
What new metrics were added to the 2021 BFCM live map?
In 2021, new metrics included product trends calculated from the top 500 product categories with the greatest change in sales volume and unique shoppers aggregated over time. This enhanced the insights provided by the live map.
How did the implementation of Flink impact the live map's performance?
The implementation of Flink significantly improved the live map's performance, allowing it to run with 100 percent uptime during BFCM without backpressure and requiring no manual intervention, thus enhancing the overall user experience.
Key Statistics & Figures
Number of merchants
1.7 million
The increase in merchants from the previous year necessitated a redesign of the live map's backend architecture.
Peak messages per second
50,000
During the 2021 BFCM weekend, the system processed approximately 50,000 messages per second.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Flink
Used for stateful stream processing to scale the live map and handle high data volumes.
Backend
Kafka
Used for message brokering to facilitate real-time data streaming.
Backend
Redis
Initially used for queuing messages and broadcasting metrics to the frontend.
Backend
Go
The programming language used to build the original Cricket service.
Frontend
React
Used for building the frontend of the live map.
Key Actionable Insights
1Utilize Apache Flink for real-time data processing to enhance scalability in applications.Flink's ability to handle high volumes of data with minimal manual intervention can significantly improve system performance, especially during peak usage times like BFCM.
2Implement deduplication strategies in streaming applications to optimize data handling.By managing duplicate data efficiently, systems can reduce unnecessary load and improve processing speed, which is critical in high-traffic scenarios.
3Consider the trade-offs between real-time data processing and system complexity.While real-time processing offers immediate insights, it can introduce complexity that requires careful management to avoid bottlenecks and performance issues.
Common Pitfalls
1
Overloading the message queuing system can lead to performance degradation.
In the previous model, Redis became a bottleneck due to the increased number of messages and connections, causing delays in data delivery to users.
2
Neglecting to account for data loss in real-time applications can lead to inaccurate insights.
The live map's metrics could tolerate some data loss, but failing to manage this could result in misleading visualizations.
Related Concepts
Real-time Data Processing
Stateful Stream Processing
Data Visualization Techniques