Let’s unpack our approach to BFCM Scale Testing to explore some of what it takes to ensure that our ecommerce platform can handle the busiest weekend of the year.
Overview
The article discusses Shopify's approach to performance testing at scale, particularly during the critical Black Friday Cyber Monday (BFCM) period. It outlines the processes involved in BFCM Scale Testing, including scaling up the platform, running load tests, and analyzing results to ensure the ecommerce platform can handle peak traffic effectively.
What You'll Learn
1
How to scale up core platform components for peak traffic scenarios
2
Why full-scale testing is critical for ecommerce platforms during high traffic events
3
How to analyze performance testing results to identify system bottlenecks
Prerequisites & Requirements
- Understanding of load testing concepts and performance metrics
- Familiarity with cloud infrastructure and scaling strategies(optional)
Key Questions Answered
What is BFCM Scale Testing and why is it important?
BFCM Scale Testing is the process of preparing Shopify's platform to handle the high traffic levels during Black Friday Cyber Monday. It involves scaling up core components, running load tests, and analyzing results to ensure the platform can support peak traffic without degradation.
How does Shopify scale up its platform for load testing?
Shopify scales up its platform by increasing the number of instances of core components like the Storefront Renderer and ProxySQL tier. This is done in a coordinated effort among multiple teams to ensure that the platform can handle the expected traffic during BFCM.
What types of load tests does Shopify perform?
Shopify performs five main types of load tests: browsing and buying flow, admin flow, flash sale flow, Storefront API flow, and Hydrogen and Oxygen flow. Each test simulates different user interactions to evaluate the platform's performance under various conditions.
What challenges does Shopify face during BFCM Scale Testing?
Challenges include managing increased TCP connections, ensuring balanced load distribution among Kubernetes pods, and identifying system bottlenecks. These issues are addressed through iterative testing and system optimizations.
Key Statistics & Figures
Percentage increase in virtual machines for scaling up
200 percent
This increase is necessary to accommodate the expected load during BFCM Scale Testing.
Maximum number of endpoints per Kubernetes cluster
1000
This limit was encountered due to an issue with ingress-nginx, impacting the scaling strategy.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Orchestration
Kubernetes
Used for managing containerized applications and scaling components during load testing.
Database
Proxysql
Acts as a connection pooler and load balancer for MySQL, crucial for handling database connections during high traffic.
Web Server
Nginx
Used for request routing and handling incoming traffic to the Shopify platform.
Key Actionable Insights
1Implement a structured approach to scaling up components before high traffic events.By planning and executing a scale-up strategy in advance, teams can identify potential issues early and ensure system stability during peak traffic periods.
2Utilize multiple load testing flows to simulate real user behavior accurately.Different flows help capture a comprehensive view of system performance, allowing for targeted optimizations and better preparation for actual traffic scenarios.
3Conduct blameless retrospectives after performance tests to foster a culture of continuous improvement.This practice encourages teams to learn from each testing cycle, addressing issues without assigning blame, which can lead to more effective solutions and enhancements.
Common Pitfalls
1
Failing to scale down components in the correct order can lead to system overload.
If upstream dependencies are scaled down before their dependent components, it can cause issues like connection overload, which can degrade performance.
2
Overlooking the impact of increased TCP connections on system resources.
Higher numbers of TCP connections can lead to increased memory usage and connection limits being exceeded, resulting in performance degradation.
Related Concepts
Load Testing Methodologies
Cloud Infrastructure Management
Performance Optimization Strategies