Surviving Flashes of High-Write Traffic Using Scriptable Load Balancers (Part II)

7 minute read In the first post of this series, I outlined Shopify’s history with flash sales, our move to Nginx and Lua to help manage traffic, and the initial attempt we made to throttle traffic that didn’t account sufficiently for customer experience. We had underestimated the impact of not giving preference to customers who’d entered the queue at the beginning of the sale, and now we needed to find another way to protect the platform without ruining the customer experience.

Shopify logo
Shopify
•
Emil Stolarsky
•7 min read•beginner•
--
•View Original
LuaMATLABRedis

Overview

This article discusses how Shopify improved its handling of high-write traffic during flash sales by implementing a stateless fair queueing system using scriptable load balancers. It details the transition from a database-dependent approach to a more efficient method using timestamps and PID controllers to prioritize user requests based on their entry time into the queue.

What You'll Learn

1

How to implement a stateless fair queueing system for handling high-write traffic

2

Why using PID controllers can optimize request handling in distributed systems

3

How to avoid single points of failure in load balancing architectures

Prerequisites & Requirements

  • Understanding of load balancing and traffic management concepts
  • Familiarity with PID controllers and control theory(optional)

Key Questions Answered

How does Shopify ensure fairness in user queue times during flash sales?
Shopify implemented a system that assigns priority to users based on their checkout attempt timestamps. By calculating a threshold dynamically on each load balancer, users who entered the queue earlier are prioritized, ensuring a fairer experience without relying on a centralized data store.
What role do PID controllers play in managing traffic at Shopify?
PID controllers are used to dynamically adjust the threshold for user requests based on the current traffic load. By calculating the error between the desired capacity and actual requests, the system can optimize which requests are prioritized, improving overall performance during high traffic periods.
What problems arise from using a data store for managing user timestamps?
Using a data store introduces a potential single point of failure, especially under high load conditions. Additionally, cross-data center replication can be slow, which complicates the management of user timestamps in a distributed system like Shopify's.
How did the implementation of the new queueing system affect customer experience?
The new stateless fair queueing system significantly reduced variance in queue times, leading to a more equitable experience for customers. After implementation, there were no reports of long wait times or unfairness during checkouts, indicating improved performance during flash sales.

Key Statistics & Figures

Queue time variance
Significantly decreased after implementing the new system
This improvement was observed in the p95, average, and median queue times, leading to a more consistent customer experience.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
Nginx
Used as part of the load balancing solution to manage incoming traffic.
Backend
Lua
Utilized for scripting within Nginx to implement the load balancing logic.

Key Actionable Insights

1
Implement a timestamp-based priority system for user requests to enhance fairness during high-traffic events.
This approach allows for prioritizing users who entered the queue earlier, ensuring a better customer experience during critical sales.
2
Utilize PID controllers to dynamically adjust thresholds in response to real-time traffic conditions.
This technique helps maintain optimal request handling without manual intervention, adapting to varying load conditions effectively.
3
Avoid relying on centralized data stores for critical user data during peak traffic.
Instead, consider stateless solutions that can operate independently across multiple data centers to enhance reliability and performance.

Common Pitfalls

1
Relying on a centralized data store can lead to performance bottlenecks and increased latency during high traffic.
This happens because a single point of failure can disrupt the entire system, especially when handling bursts of traffic. Instead, consider decentralized approaches to enhance resilience.

Related Concepts

Load Balancing Techniques
Traffic Management Strategies
Distributed Systems Architecture