Overview
This article discusses fleet-wide refactoring at Spotify, detailing the tools and strategies developed to manage code changes across thousands of Git repositories. It highlights the importance of automating dependency management and the use of the Fleetshift platform to facilitate large-scale code updates efficiently.
What You'll Learn
1
How to implement fleet-wide refactoring using the Fleetshift platform
2
Why automated dependency management is crucial for large codebases
3
How to configure automated merging of pull requests in a CI/CD pipeline
4
When to use gradual rollouts for code changes in production
Prerequisites & Requirements
- Understanding of Git and CI/CD practices
- Familiarity with Docker and Kubernetes(optional)
Key Questions Answered
What is fleet-wide refactoring and why is it important?
Fleet-wide refactoring is the process of updating code across multiple repositories simultaneously to improve consistency and reduce technical debt. It is crucial for managing dependencies efficiently in a polyrepo layout, allowing teams to adopt changes quickly and maintain code quality across thousands of components.
How does Fleetshift automate code changes across repositories?
Fleetshift automates code changes by running Docker images against each targeted repository, creating pull requests for the changes. This process leverages Spotify's Kubernetes infrastructure to handle multiple repositories simultaneously, making fleet-wide refactoring feasible and efficient.
What benefits does automated dependency management provide?
Automated dependency management simplifies the process of updating libraries across many components, ensuring compatibility and reducing the risk of vulnerabilities. For example, during the Log4j vulnerability incident, 80% of Spotify's production backend services were patched within 9 hours due to effective dependency management.
What is the role of the automerger in the refactoring process?
The automerger automatically merges pull requests if they pass all tests, reducing the need for manual reviews. This capability allows for quicker adoption of changes and encourages teams to maintain robust test suites to ensure code quality.
Key Statistics & Figures
Percentage of production backend services using Java BOM
96%
This high adoption rate allows engineers to update dependencies easily across the fleet.
Time taken for new releases of backend service framework to reach 70% adoption
less than 7 days
This is a significant improvement from about 200 days, showcasing the effectiveness of fleet-wide refactoring.
Number of pull requests created by Fleetshift in 2022
over 270,000
This reflects a 4–6x increase in pull requests compared to 2021, indicating the tool's impact on productivity.
Lines of code changed through merged pull requests
4.2 million
This substantial amount of code change highlights the scale of refactoring achieved through the Fleetshift platform.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Containerization
Docker
Used to create Docker images that automate code refactoring across multiple repositories.
Orchestration
Kubernetes
Provides the infrastructure to run code transformations and manage the execution of Fleetshift jobs.
Build Management
Apache Maven
Utilized for managing dependencies in backend services through the Java BOM.
Data Processing
Scio
Framework used for batch data pipelines, which also benefits from automated dependency management.
Key Actionable Insights
1Invest in tools like Fleetshift to automate code refactoring across multiple repositories.By using Fleetshift, teams can significantly reduce the time and effort required to implement changes across a large codebase, fostering a culture of continuous improvement and collaboration.
2Implement automated dependency management to enhance security and maintainability.Automating the management of dependencies allows teams to quickly address vulnerabilities and ensure that all components are using compatible library versions, which is critical in a fast-paced development environment.
3Utilize gradual rollouts for critical changes to minimize risk.By applying changes in cohorts, teams can monitor the impact of updates and ensure stability before wider deployment, which is particularly important for complex systems like batch data pipelines.
Common Pitfalls
1
Overlooking the importance of comprehensive testing before automerging changes.
Without sufficient tests, automated changes can introduce bugs or regressions. Teams should ensure their components meet testing standards to leverage automerging effectively.
2
Failing to monitor the health of components post-merge.
Neglecting to track the impact of automated changes can lead to undetected issues. Implementing monitoring systems like Firewatch is essential for maintaining code quality.
Related Concepts
Automated Testing Practices
Continuous Integration And Delivery (ci/Cd)
Microservices Architecture
Dependency Management Strategies