How Shopify scaled our Server Pixels service to increase its event processing performance by 170%.
Overview
This article discusses how Shopify scaled its Server Pixels service to enhance event processing performance by 170% using Go worker pools. It details the challenges faced with event consumption lag and how implementing a fixed number of workers improved resource management and processing efficiency.
What You'll Learn
1
How to implement Go worker pools for efficient task processing
2
Why controlling concurrency is crucial for resource management in high-load systems
3
When to use channels for communication between goroutines in Go
Prerequisites & Requirements
- Understanding of Go programming language and concurrency concepts
- Experience with event-driven architectures and Kafka(optional)
Key Questions Answered
How did Shopify increase event processing performance using Go?
Shopify increased its event processing performance by 170% by implementing Go worker pools, which allowed for controlled concurrency and reduced resource strain. This change enabled the service to handle 21 thousand events per second per pod, compared to the previous 7.75 thousand.
What challenges did Shopify face with its original Server Pixels design?
The original design faced issues with consumption lag in Kafka due to the spawning of an unlimited number of goroutines when processing events. This led to resource exhaustion and delays in event processing, especially during peak loads.
What is the role of channels in Go worker pools?
In Go worker pools, channels serve as communication pipes between goroutines, allowing workers to receive tasks from a job queue. This design facilitates efficient task management and ensures that workers only process jobs when they are available.
What improvements were observed after implementing the worker pool design?
After deploying the worker pool design, Shopify observed a significant performance improvement, processing up to 21 thousand events per second per pod, a 170% increase from the previous implementation. This allowed better handling of peak traffic during events like Black Friday and Cyber Monday.
Key Statistics & Figures
Event processing performance increase
170%
Increased from 7.75 thousand events per second per pod to 21 thousand events per second per pod.
Maximum events processed during BFCM 2021
46 thousand events per second
This peak performance was achieved using the new worker pool design.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Go
Used to implement the Server Pixels service and the worker pool pattern.
Backend
Kafka
Used for managing the consumption and production of events.
Key Actionable Insights
1Implementing a worker pool pattern can significantly enhance the performance of your event-driven applications.By controlling the number of concurrent tasks, you can prevent resource exhaustion and ensure timely processing of events, especially during high-load scenarios.
2Utilizing Go channels for inter-goroutine communication simplifies task management and enhances code readability.Channels allow for a clean and efficient way to pass data between goroutines, making it easier to maintain and scale your application.
3Conduct thorough load testing after implementing new designs to validate performance improvements.Load testing helps identify bottlenecks and ensures that your application can handle expected traffic, especially during peak periods.
Common Pitfalls
1
Spawning an unlimited number of goroutines can lead to resource exhaustion and processing delays.
This often occurs when the system receives a sudden spike in events, overwhelming the available resources and causing lag in event processing.
Related Concepts
Concurrency In Go
Event-driven Architecture
Performance Optimization Techniques