How and Why We Migrated Airbnb’s Large-Scale Web Monorepo to Bazel
Overview
Airbnb's adoption of Bazel as a universal build system for their large-scale web monorepo is detailed in this article. The post outlines the challenges faced during migration, the preparation of the codebase, and the performance improvements achieved through this transition.
What You'll Learn
1
How to prepare a large codebase for Bazel migration
2
Why Bazel is beneficial for managing CI jobs in large repositories
3
How to implement caching strategies to improve CI performance
4
When to use custom Bazel rules for TypeScript and ESLint
5
How to maintain developer experience during migration to Bazel
Prerequisites & Requirements
- Understanding of build systems and CI/CD processes
- Familiarity with Bazel and JavaScript tooling(optional)
Key Questions Answered
What challenges did Airbnb face when migrating to Bazel?
Airbnb encountered several challenges during their migration to Bazel, including the lack of publicized industry precedents for integrating Bazel with web at scale, performance issues with large codebases, and the need to break cycles in the dependency graph. They also had to ensure that the migration did not negatively impact developer productivity.
How did Bazel improve CI performance for Airbnb?
Bazel significantly improved CI performance at Airbnb, with TypeScript type checking becoming 34% faster, ESLint linting 35% faster, and Jest unit tests achieving a 42% speedup in incremental runs and a 29% overall speedup. This enhancement allowed the CI system to scale better as the codebase grew.
What strategies did Airbnb use to prepare their repository for Bazel?
To prepare their repository for Bazel, Airbnb focused on cycle breaking in the dependency graph and automated generation of BUILD.bazel files. They modeled the cycle-breaking process as finding the minimum feedback arc set to minimize disruption and used a CLI tool to automate the creation of configuration files.
What caching strategies were implemented to enhance Bazel performance?
Airbnb implemented caching strategies that included using a Docker cache and remote worker cache to speed up dependency extraction. This reduced the extraction time from 1-3 minutes to 3-7 seconds per target, resulting in a 25% speedup for Jest unit testing CI jobs.
Key Statistics & Figures
TypeScript type checking speed improvement
34%
This improvement was achieved after migrating to Bazel for CI jobs.
ESLint linting speed improvement
35%
The speedup was noted after the migration to Bazel.
Jest unit tests speed improvement
42% faster incremental runs, 29% overall
These improvements were part of the performance enhancements realized through Bazel adoption.
Dependency extraction time reduction
from 1-3 minutes to 3-7 seconds
This reduction was achieved by implementing caching strategies.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Build Tool
Bazel
Used as the universal build system across backend, web, and iOS platforms.
Programming Language
Typescript
Used for type checking in the CI jobs.
Linting Tool
Eslint
Used for code quality checks in the CI jobs.
Testing Framework
Jest
Used for unit testing in the CI jobs.
Containerization
Docker
Used for caching dependencies to speed up CI processes.
Key Actionable Insights
1Implement automated generation of Bazel configuration files to streamline migration efforts.By automating the creation of BUILD.bazel files, teams can reduce manual overhead and ensure consistency across the codebase, which is crucial during large-scale migrations.
2Utilize caching strategies to enhance CI performance when using Bazel.Implementing caching can significantly reduce CI job runtimes, as demonstrated by Airbnb's experience, where they achieved a 25% speedup in Jest tests through effective caching mechanisms.
3Focus on breaking cycles in the dependency graph to prepare for Bazel integration.Identifying and resolving cycles is critical for Bazel's operation, as it requires a directed acyclic graph (DAG) of build targets to function correctly.
4Encourage developers to maintain a high-quality experience during the transition to Bazel.Ensuring that developers can run Bazel locally and reproduce CI failures helps maintain productivity and reduces frustration during the migration process.
5Gradually migrate CI jobs to Bazel to minimize disruption.By migrating CI jobs incrementally, teams can deliver value sooner and build confidence in their approach, which is essential for maintaining developer morale.
Common Pitfalls
1
Failing to break cycles in the dependency graph can lead to migration challenges.
Bazel requires a directed acyclic graph for its operations, and not addressing cycles can complicate the migration process significantly.
2
Neglecting to automate configuration file generation can increase manual workload.
Manual updates to BUILD.bazel files can create additional overhead and hinder developer experience, making automation essential.
3
Overlooking the importance of caching can result in poor CI performance.
Without effective caching strategies, CI jobs may run significantly slower, leading to developer frustration and inefficiencies.
Related Concepts
Bazel Migration Strategies
CI/CD Best Practices
Performance Optimization In Build Systems