Overview
The article discusses Spotify's load testing strategies for its annual Wrapped campaign, which faced significant traffic challenges due to millions of users engaging simultaneously. It highlights the architecture of the Wrapped backend, the use of the Moshpit tool for load testing, and the lessons learned from their testing processes.
What You'll Learn
1
How to effectively use load testing tools like Moshpit for backend services
2
Why it's crucial to notify upstream service owners before load testing
3
When to scale pods horizontally during load testing sessions
4
How to ensure diverse payloads represent real-world usage in load tests
Prerequisites & Requirements
- Understanding of load testing concepts and backend service architecture
- Familiarity with Spotify's internal tools like Moshpit and Backstage(optional)
Key Questions Answered
What is the purpose of the Moshpit tool in Spotify's load testing?
Moshpit is Spotify's internal tool designed for load testing backend services. It allows developers to send customizable payloads to any internal service using multiple network protocols, ensuring that the services can handle expected traffic loads during high-demand events like Wrapped.
How does Spotify prepare for the high traffic during Wrapped?
Spotify prepares for high traffic during Wrapped by conducting extensive load testing using Moshpit. They notify upstream service owners, adjust routing to remove employee flags, and scale their services horizontally to ensure that all components can handle the expected tens of thousands of requests per second.
What are the key lessons learned from Spotify's load testing for Wrapped?
Key lessons include the importance of testing with diverse payloads to represent real-world usage, provisioning resources for upstream services, and ensuring that tests are conducted across multiple regions to simulate global user engagement effectively.
What challenges do developers face when using Moshpit for load testing?
Developers often struggle with sourcing a diverse and representative sample set of test payloads for Moshpit. This is crucial to accurately simulate real-world loads and ensure that the backend can handle various user scenarios during peak traffic times.
Key Statistics & Figures
Unique engaged users during Wrapped
more than 150 million
This number reflects the scale of user engagement Spotify anticipates during the Wrapped campaign.
Expected requests per second during load testing
tens of thousands
This figure indicates the volume of traffic that the main service is expected to handle during peak times.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Moshpit
Internal tool for load testing backend services at Spotify.
Backend
Backstage
Platform that integrates with Moshpit and other backend services.
Data Format
Protocol Buffers
Used for encoding payloads in Moshpit.
Key Actionable Insights
1Always notify upstream service owners before conducting load tests to avoid unexpected alerts.This proactive communication allows teams to prepare for potential impacts on their services, ensuring smoother testing and minimizing disruptions.
2Utilize a diverse set of payloads that accurately reflect real-world usage scenarios in your load tests.Testing with varied user data helps identify performance issues that might arise under different conditions, leading to a more robust and reliable service.
3Scale your services horizontally during load tests to match expected production traffic.By provisioning enough resources, you can ensure that both your service and any dependent upstream services can handle the load without performance degradation.
Common Pitfalls
1
Failing to source a diverse enough set of payloads can lead to inaccurate load testing results.
Without a representative sample, the load tests may not reveal potential bottlenecks or issues that could arise during actual user engagement, leading to unexpected failures.