Overview
The article discusses Uber's approach to continuous deployment (CD) for large monorepos, detailing the challenges faced and the solutions implemented to automate deployments safely and efficiently. It highlights the significant improvements in deployment frequency and reliability achieved through the new Up CD system.
What You'll Learn
1
How to automate deployments in a microservices architecture
2
Why standardization in deployment processes is crucial for large teams
3
How to leverage commit history for better service management
Prerequisites & Requirements
- Understanding of microservices and continuous deployment concepts
- Familiarity with Uber's internal tools and platforms like Up(optional)
Key Questions Answered
What improvements did Uber see after implementing Up CD?
After implementing Up CD, Uber saw the number of services deployed automatically increase from less than 10% to almost 70% over a 12-month period. Additionally, the overall rate of production incidents decreased by more than 50% per 1,000 code changes, demonstrating the effectiveness of the new system.
How does the new CD system handle commits in a monorepo?
The new CD system uses a simple data structure linking each service to relevant commits, allowing for efficient identification of changes impacting each service. This approach prevents unnecessary builds and deployments for all services when only a subset is affected by a commit.
What challenges did Uber face with cross-cutting changes in the monorepo?
Uber faced increased risks when making changes to shared code in the monorepo, as these changes could rapidly affect multiple services. To mitigate this, they introduced a deployment strategy that staggers risky commits across service tiers, ensuring that any issues could be detected early.
Key Statistics & Figures
# of services fully automated to production
65%
This is an increase from 7% before the implementation of Up CD.
Production deployments / week
50,000
This is a significant increase from 7,000 deployments per week prior to the new system.
Decrease in reported incidents per 1,000 code changes
more than 50%
This decrease occurred during the same period when CD adoption rose.
Technologies & Tools
Platform
Up
Used as the integrated platform for the new continuous deployment system.
Messaging
Kafka Consumer Proxy
Used to consume events from Git commits for processing in the CD pipeline.
Workflow Orchestration
Cadence
Utilized to implement build and deployment workflows in the new CD system.
Key Actionable Insights
1Standardizing deployment processes across teams can significantly improve deployment safety and reliability.By implementing a unified CD system, Uber reduced the risk associated with diverse deployment practices, allowing for safer and more efficient releases.
2Utilizing commit history effectively can streamline service management in large monorepos.Linking services to specific commits helps engineers quickly identify changes affecting their services, reducing the time spent analyzing extensive Git logs.
3Adopting automation in deployment can lead to higher adoption rates and reduced incident reports.Uber's experience shows that as automation increases, teams can deploy more frequently while maintaining or even improving code quality.
Common Pitfalls
1
Failing to standardize deployment processes can lead to confusion and increased risk.
Without a unified approach, teams may implement varied practices that complicate the deployment landscape, making it harder to manage and scale.
2
Overlooking the impact of shared code changes in a monorepo can result in widespread issues.
Changes to common libraries can quickly propagate errors across multiple services, necessitating robust monitoring and rollback strategies.
Related Concepts
Continuous Deployment
Microservices Architecture
Automation In Software Development