Overview
The article discusses Uber's implementation of a system called SubmitQueue, which manages code changes in a monorepo. It focuses on the challenge of large diffs that slow down the verification process and introduces a new approach called Bypass Large Diffs (BLRD) that allows changes to be landed out of order, resulting in a 74% improvement in wait times.
What You'll Learn
1
How to implement Bypass Large Diffs (BLRD) in a CI/CD pipeline
2
Why reducing wait times for code changes is critical in large codebases
3
When to apply speculation trees for verifying conflicting changes
Prerequisites & Requirements
- Understanding of CI/CD processes and monorepo management
Key Questions Answered
How does Uber's SubmitQueue manage large diffs?
Uber's SubmitQueue uses a conflict analyzer to determine if changes are independent or conflicting. It allows for speculative execution of CI in parallel, which helps maintain a green main branch while managing thousands of commits daily. The introduction of Bypass Large Diffs (BLRD) further optimizes this process by allowing changes to be landed out of order, significantly reducing wait times.
What improvements were observed after implementing BLRD?
After enabling Bypass Large Diffs (BLRD) in late May 2023, Uber observed a 74% reduction in the P95 wait time for code changes compared to April. This improvement was achieved without significant changes to build and test times, indicating the effectiveness of the new approach in managing large diffs.
What challenges does SubmitQueue face with large codebases?
As Uber's codebase grew, the number of packages affected by large diffs increased, leading to longer verification times. This created a backlog of changes waiting to be verified, which prompted the need for new solutions like Bypass Large Diffs to improve efficiency and developer experience.
Key Statistics & Figures
P95 wait time reduction
74%
This reduction was observed in June 2023 compared to April 2023 after implementing BLRD.
Increase in root targets affected by large changes
2.5 times
This increase was noted over a span of two years as Uber's codebase expanded.
Change rate increase
more than 2 times
This increase was observed in the same two-year period as the growth of the codebase.
Technologies & Tools
Backend
Submitqueue
Used for managing and verifying code changes in Uber's monorepo.
Process
CI/CD
Continuous integration and deployment practices to ensure code quality and rapid delivery.
Key Actionable Insights
1Implementing Bypass Large Diffs (BLRD) can drastically reduce wait times for code changes.By allowing changes to be landed out of order, teams can maintain productivity and reduce bottlenecks in the CI/CD pipeline, especially in large codebases.
2Utilizing speculation trees can enhance the verification process for conflicting changes.This method allows for concurrent verification of changes, which can significantly speed up the landing process, particularly when dealing with large diffs.
3Establishing clear policies for landing large diffs can improve developer experience.By optimizing the timing and conditions under which large diffs are processed, organizations can minimize disruptions and maintain a more efficient workflow.
Common Pitfalls
1
Assuming that all changes can be processed sequentially without considering their impact on the main branch.
This can lead to significant delays, especially with large diffs that block other changes. Implementing strategies like BLRD can help mitigate these issues.
Related Concepts
Continuous Integration
Continuous Deployment
Monorepo Management
Speculative Execution