Overview
This article discusses the challenges and solutions Uber faced in controlling the rollout of large-scale changes in a monorepo environment. It highlights the introduction of cross-service deployment orchestration to minimize the risk of deploying changes that could affect numerous microservices simultaneously.
What You'll Learn
1
How to implement cross-service deployment orchestration for large-scale changes
2
Why monitoring deployment signals from multiple services is crucial for safety
3
When to apply state machines for orchestrating service deployments
Prerequisites & Requirements
- Understanding of continuous deployment practices
- Experience with microservices architecture(optional)
Key Questions Answered
How does Uber minimize the risk of large-scale changes affecting many services?
Uber minimizes risk by implementing cross-service deployment orchestration, which uses signals from multiple services to prevent problematic changes from reaching critical services. This approach allows for a more controlled rollout, ensuring that any issues can be detected and addressed before impacting the entire system.
What statistics indicate the frequency of large-scale changes in Uber's monorepo?
Analysis of 500,000 commits in Uber's Go monorepo revealed that 1.4% of commits affect more than 100 services, while 0.3% affect over 1,000 services. This indicates that large-scale changes occur regularly, amplifying the risk associated with deployment.
What is the role of state machines in the orchestration process?
State machines are used to maintain a simple state for each large-scale commit, allowing the system to track the deployment status across services. This helps in determining whether to proceed with deployments based on the success or failure of previous stages.
Key Statistics & Figures
Percentage of commits affecting more than 100 services
1.4%
This statistic highlights the frequency of large-scale changes in Uber's Go monorepo.
Percentage of commits affecting more than 1,000 services
0.3%
Indicates the scale at which changes can impact Uber's services, emphasizing the need for careful orchestration.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Key Actionable Insights
1Implement cross-service deployment orchestration to enhance deployment safety.This approach allows teams to utilize signals from various services to ensure that changes are safe before reaching critical services, thereby reducing the risk of widespread failures.
2Regularly analyze commit patterns to understand the impact of large-scale changes.By reviewing statistics on commits that affect multiple services, teams can better prepare for potential risks and adjust their deployment strategies accordingly.
3Utilize simulations to predict the outcomes of parameter changes in deployment processes.Simulations can help teams make informed decisions about deployment parameters, ensuring a balance between speed and safety in the rollout of changes.
Common Pitfalls
1
Failing to adequately monitor deployment signals can lead to undetected issues during rollout.
This oversight can result in widespread service failures if problematic changes are deployed without proper checks.
Related Concepts
Continuous Deployment Practices
Microservices Architecture
Deployment Orchestration Strategies