Overview
The article discusses Spotify's journey with Docker and the challenges faced during its rollout across critical infrastructure. It highlights the need for a controlled rollout process, leading to the development of Tsunami, a service that enables gradual infrastructure changes.
What You'll Learn
1
How to manage Docker upgrades to minimize service disruption
2
Why gradual rollouts are essential for critical infrastructure
3
How to implement a centralized service for infrastructure rollouts using Tsunami
Prerequisites & Requirements
- Understanding of Docker and container orchestration
- Familiarity with infrastructure management tools like Helios and Puppet(optional)
Key Questions Answered
What issues did Spotify encounter when upgrading Docker?
Spotify faced multiple issues during Docker upgrades, including regressions related to container hostnames and CMD/ENTRYPOINT settings. They also encountered orphaned containers that caused traffic routing problems, which highlighted the need for more controlled upgrade processes.
How does Tsunami improve infrastructure rollout at Spotify?
Tsunami allows Spotify to perform unattended, gradual rollouts of changes to configuration and infrastructure components. By controlling the rollout process, Tsunami improves reliability and enables early detection of issues while affecting only a small portion of the fleet.
Why is gradual upgrading important for Spotify's backend services?
Gradual upgrading is crucial because it limits the impact of potential bugs and allows engineers to detect and address issues in real-time. This approach is necessary as Spotify's backend services have become mission-critical, with 80% running as containers by February 2017.
What is the primary function of Tsunami in Spotify's infrastructure?
The primary function of Tsunami is to manage the rollout of infrastructure changes in a controlled manner, allowing for gradual upgrades and monitoring of service-level objectives. This centralized service helps ensure that changes are implemented smoothly across thousands of hosts.
Key Statistics & Figures
Percentage of backend services running as containers
80%
As of February 2017, this statistic highlights the critical role Docker plays in Spotify's infrastructure.
Number of Docker instances running at Spotify
several thousand
This indicates the scale of Docker's adoption within Spotify's backend services.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Containerization
Docker
Used for running backend services and managing containerized applications.
Orchestration
Helios
A Docker orchestration tool used by Spotify to manage container traffic.
Rollout Management
Tsunami
A service developed to facilitate gradual rollouts of infrastructure changes.
Configuration Management
Puppet
Used for managing configurations and automating deployment processes.
Key Actionable Insights
1Implement gradual rollouts for infrastructure changes to mitigate risks associated with upgrades.By adopting a gradual rollout strategy, teams can limit the impact of bugs and ensure that issues are identified and resolved before they affect a larger portion of the infrastructure.
2Utilize centralized services like Tsunami to manage configuration changes effectively.Centralized management allows for better tracking of changes and provides features like audit logs and monitoring, which enhance the reliability of infrastructure deployments.
3Monitor orphaned containers and processes during Docker upgrades to prevent service disruptions.By actively monitoring for orphaned containers, teams can address issues proactively, ensuring that traffic is routed correctly and services remain operational.
Common Pitfalls
1
Failing to monitor the effects of Docker upgrades can lead to significant service disruptions.
Without proper monitoring, teams may not detect issues like orphaned containers or misrouted traffic until they impact user experience, making it essential to implement monitoring strategies during upgrades.
Related Concepts
Container Orchestration
Infrastructure As Code
Continuous Integration/Continuous Deployment (ci/Cd)