Overview
This article discusses how Pinterest's Engineering Productivity team redesigned their Continuous Integration (CI) system to achieve over 50% faster build times. Key strategies included adopting Bazel as the build tool, implementing language-based monorepos, and optimizing the CI process to only test and release changed code.
What You'll Learn
1
How to implement Bazel as a build tool for CI systems
2
Why parallelizing CI tasks can significantly reduce build times
3
How to use Bazel queries to optimize testing and release processes
Prerequisites & Requirements
- Understanding of Continuous Integration concepts
- Familiarity with Bazel(optional)
Key Questions Answered
How did Pinterest achieve a 50% reduction in CI build times?
Pinterest achieved a 50% reduction in CI build times by adopting Bazel as their build tool, implementing language-based monorepos, and optimizing their CI process to only test and release the code that was changed. This approach allowed for significant improvements in efficiency and speed.
What is the role of the build-collector CLI in the CI process?
The build-collector CLI is used to determine the minimal set of targets that need to run in CI by analyzing the changes in code commits. It utilizes Bazel queries to identify affected targets, which helps in speeding up the build process by avoiding unnecessary tests.
What are the benefits of using language-based monorepos?
Language-based monorepos allow Pinterest to manage CI pipelines more efficiently by grouping services into single repositories per language. This reduces the complexity of managing multiple CI pipelines and helps streamline the CI process.
How does Pinterest handle Pull Request (PR) builds?
Pinterest's PR pipeline triggers a CI run whenever a new pull request is created. It patches the code changes and uses a temporary commit to pass to build-collector, ensuring that the same CI setup is reused for PR builds without creating release artifacts.
Key Statistics & Figures
P50 CI build time before redesign
52 minutes
This was the average time taken for CI builds before implementing the new design.
P50 CI build time after redesign
19 minutes
After the redesign, the average CI build time dropped significantly, demonstrating the effectiveness of the changes.
P90 CI build time before redesign
69 minutes
This was the 90th percentile time for CI builds before the new optimizations were applied.
P90 CI build time after redesign
49 minutes
The 90th percentile time for CI builds improved after the redesign, indicating better performance for slower builds.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Build Tool
Bazel
Used as the primary build tool for CI workflows at Pinterest.
Programming Language
Golang
Used to develop the build-collector CLI for optimizing CI processes.
Key Actionable Insights
1Implementing Bazel as a unified build tool can streamline CI processes across multiple languages.By adopting Bazel, teams can create CI workflows applicable to any repository, reducing the overhead of managing dependencies and configurations.
2Utilizing a build-collector CLI can optimize CI by running only necessary tests.This approach minimizes build times and resource usage, especially in large codebases where only a subset of the code may change frequently.
3Parallelizing CI tasks can significantly enhance build performance.Using a Dispatcher Model allows for independent execution of tasks, ensuring that developers receive their release artifacts as soon as they are ready, rather than waiting for the entire build process to complete.
Common Pitfalls
1
Failing to optimize CI processes can lead to unnecessarily long build times.
Many teams overlook the importance of running only the necessary tests, which can significantly slow down development cycles. Implementing targeted testing strategies can mitigate this issue.
Related Concepts
Continuous Integration
Bazel
Monorepos
CI/CD