Recently, we’ve been working to make our CI experience better by leveraging the newly released GitHub feature, Actions larger runners, to run our CI.
Overview
This article discusses how GitHub transitioned its continuous integration (CI) system to GitHub Actions, utilizing larger runners to enhance developer experience and scalability. It highlights the benefits of using larger runners, including improved performance metrics and the introduction of features like reusable workflows and custom VM images.
What You'll Learn
1
How to leverage GitHub Actions larger runners for CI jobs
2
Why using custom VM images can reduce CI workflow times significantly
3
How to implement reusable workflows across multiple repositories
4
When to utilize the feature of reusing previous workflow outcomes
Prerequisites & Requirements
- Understanding of continuous integration concepts
- Familiarity with GitHub Actions
Key Questions Answered
What are GitHub Actions larger runners and their benefits?
GitHub Actions larger runners are managed virtual machines with enhanced resources compared to standard runners. They allow for autoscaling, enabling GitHub to run 15,000 CI jobs within an hour across 150,000 cores, significantly improving the developer experience and CI performance.
How does GitHub ensure private service access from larger runners?
GitHub developed a remote access solution that uses an OIDC token to securely access private services from larger runners. This solution allows workflows to communicate with private resources within GitHub's virtual private cloud, ensuring both security and functionality.
What impact do custom VM images have on CI workflow times?
Custom VM images can significantly reduce CI workflow times from around 50 minutes to just 12 minutes by pre-installing necessary tools and bundling project source code, which speeds up the build process during CI jobs.
What are reusable workflows in GitHub Actions?
Reusable workflows allow teams to define a workflow in one repository that can be used across multiple repositories, promoting consistency and reducing redundancy in CI configurations. This feature simplifies the integration of CI into new projects.
Key Statistics & Figures
Concurrent runners in use
4,500
GitHub utilizes 4,500 concurrent 32-core runners to manage CI jobs efficiently.
Build minutes per hour
125,000
The CI system runs 125,000 build minutes per hour, showcasing its scalability.
Jobs queued and run per hour
15,000
Approximately 15,000 jobs are queued and run within an hour, demonstrating the system's capacity.
Cores of compute allocated
150,000
The CI system allocates around 150,000 cores of compute to handle job demands.
Workflow duration reduction
from 50 minutes to 12 minutes
The introduction of custom VM images has reduced the average workflow duration significantly.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
CI/CD
Github Actions
Used for automating the CI processes at GitHub.
Security
Oidc
Used for secure access to private services from larger runners.
Key Actionable Insights
1Utilize GitHub Actions larger runners to enhance your CI processes.By leveraging larger runners, you can significantly increase the number of concurrent jobs and reduce build times, which is crucial for teams looking to scale their CI/CD pipelines efficiently.
2Implement custom VM images for your CI workflows to speed up build times.Custom VM images allow you to pre-install necessary dependencies, which can drastically reduce the time spent on setup during CI jobs, leading to faster feedback cycles for developers.
3Adopt reusable workflows to streamline CI configurations across projects.By centralizing workflow definitions, you can avoid duplication and ensure that all projects adhere to best practices, making it easier to maintain and update CI processes.
4Leverage the feature of reusing previous workflow outcomes to optimize CI runs.This feature can save significant time by skipping unnecessary CI checks when there are no changes in the codebase, allowing developers to focus on more critical tasks.
Common Pitfalls
1
Neglecting to update custom VM images can lead to outdated dependencies.
If custom images are not regularly updated, the cached project source code may become stale, resulting in longer build times and potential integration issues. It's essential to implement a strategy for keeping these images current.
Related Concepts
Continuous Integration
Github Actions Features
CI/CD Best Practices
Scaling CI Systems