Keeping Developers Happy with a Fast CI

Christian Bruckmayer

Slow CI systems are often responsible for making frustrated developers. Keeping these systems fast requires an ongoing effort. However, before jumping into implementing performance tweaks, it's important to set up a solid monitoring system.

Shopify

•

Christian Bruckmayer

•11 min read•intermediate•

--

•View Original

ActiveRecordDockerJSONMySQLRailsRubySQLYAML

Overview

The article discusses how Shopify's Test Infrastructure team significantly improved the speed of their Continuous Integration (CI) system, reducing the 95th percentile build time from 45 minutes to 18 minutes. It highlights the strategies used to identify bottlenecks, optimize processes, and enhance developer satisfaction.

What You'll Learn

1

How to implement instrumentation for CI to identify bottlenecks

2

Why reducing Docker start time can significantly improve CI performance

3

How to optimize CI by selectively running tests based on code changes

Prerequisites & Requirements

Understanding of Continuous Integration concepts
Familiarity with Buildkite(optional)

Key Questions Answered

How did Shopify reduce CI build times from 45 minutes to 18 minutes?

Shopify's Test Infrastructure team implemented several strategies including instrumentation to identify bottlenecks, optimizing Docker start times, and selectively running tests based on code changes. These changes allowed them to significantly enhance CI performance and developer satisfaction.

What role does Buildkite play in Shopify's CI system?

Buildkite serves as the CI platform for Shopify, allowing them to run CI servers on their own cloud infrastructure. This flexibility enables aggressive scaling, support for different architectures, and customization, which are crucial for optimizing CI performance.

What is the significance of the 95th percentile in CI performance?

The 95th percentile indicates that 95% of all CI builds complete in under a specified time—in this case, under 10 minutes. This metric helps set performance goals and assess the effectiveness of optimizations made to the CI system.

How does selective test execution improve CI efficiency?

By only running tests related to the specific code changes made in a pull request, Shopify reduced unnecessary test executions. This approach not only speeds up the CI process but also improves test stability, as it focuses on relevant tests.

Key Statistics & Figures

Initial p95 CI build time

45 minutes

This was the starting point before optimizations were implemented.

Final p95 CI build time

18 minutes

This is the improved build time achieved after the optimizations.

Percentage of CI time spent on preparing agents and building dependencies

68%

This highlights the significant overhead before tests were even executed.

Improvement in Docker start time

from 90 seconds to 25 seconds

This was achieved by addressing disk I/O bottlenecks.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

CI/CD

Buildkite

Used as the platform for running CI servers on Shopify's cloud infrastructure.

Containerization

Docker

Utilized for running services during the CI process.

Key Actionable Insights

1
Implement a robust monitoring system for your CI to identify bottlenecks effectively.
Having detailed insights into CI performance allows teams to spot issues early and address them before they escalate, ensuring a smoother development workflow.

2
Optimize Docker container start times by addressing disk I/O bottlenecks.
Improving the speed of Docker container initialization can significantly reduce overall CI build times, as demonstrated by Shopify's improvements in their CI system.

3
Utilize data-driven development to prioritize CI optimizations based on actual usage patterns.
By analyzing command execution times and frequencies, teams can focus their efforts on the most impactful areas, leading to more efficient CI processes.

Common Pitfalls

1

Focusing on fixing symptoms rather than identifying root causes of CI performance issues.

This can lead to temporary solutions that do not address underlying problems, resulting in recurring issues and frustration among developers.

Related Concepts

Continuous Integration

Performance Optimization

Data-driven Development

Docker Container Management

How would we build a web application only using the standard Ruby libraries? In this article, I’ll break down the key foundational concepts of how a web application works while building one from the ground up. If we can build a web application only using Ruby libraries, why would we need web server interfaces like Rack and web applications like Ruby on Rails? By the end of this article, you’ll gain a new appreciation for Rails and its magic.

PostgreSQLGolangRuby

30 min read

Includes Code

Has Summary

--

Shopify

Advanced

Horizontally scaling the Rails backend of Shop app with Vitess

Shop app horizontally scaled a Ruby on Rails app with Vitess. This blog describes Vitess and our detailed approach for introducing Vitess to a Rails app.

RubySQLRails

31 min read

Includes Code

Has Summary

--

Shopify

Intermediate

Upgrading Shopify to Rails 5

Today, Shopify runs on Rails 5.0, the latest version. It’s important to us to stay on the latest version so we can improve the performance and stability of the application without having to increase the maintenance cost of applying monkey patches. This guarantees we would always be in the version maintained by the community; and, that we would have access to new features soon. Upgrading the Shopify monolith—one of the oldest and the largest Rails applications in the industry—from Rails 4.2 to 5.0 took us nearly a year. In this post, I’ll share our upgrade story and the lessons we learned. If you're wondering how the Shopify scale looks like or you plan a major Rails upgrade, this post is for you.

DockerRubyRails

8 min read

Includes Code

Has Summary

--

These articles from Shopify and other leading engineering teams share similar topics with "Keeping Developers Happy with a Fast CI". Explore more engineering insights on PostgreSQL, Golang, Ruby.