Building a Dynamic Mobile CI System

18 minute read The mobile space has changed quickly, even within the past few years. At Shopify, the world’s largest Rails application, we have seen the growth and potential of the mobile market and set a goal of becoming a mobile-first company. Today, over 130,000 merchants are using Shopify Mobile to set up and run their stores from their smartphones. Through the inherent simplicity and flexibility of the mobile platform, many mobile-focused products have found success. This post was co-written with Arham Ahmed, and shout-outs to Sean Corcoran of MacStadium and Tim Lucas of Buildkite.

Shopify logo
Shopify
•
Sander Lijbrink
•17 min read•intermediate•
--
•View Original
AWSCachingDockerGoogle CloudJavaKubernetesPackerRailsRedisREST APIRubySwiftTravis CI

Overview

This article discusses the development of a dynamic mobile Continuous Integration (CI) system at Shopify, highlighting the transition from a static setup to a scalable, efficient architecture. It covers the challenges faced, the technologies used, and the improvements achieved in build times and deployment processes.

What You'll Learn

1

How to build a scalable mobile CI system using Buildkite

2

Why using linked clones can optimize VM deployment times

3

How to implement caching strategies for faster builds

4

When to use virtual machines for isolated build environments

Prerequisites & Requirements

  • Understanding of CI/CD concepts and practices
  • Familiarity with Buildkite and VMware technologies(optional)

Key Questions Answered

What improvements were made to Shopify's mobile CI system?
Shopify revamped its mobile CI system to achieve faster build times, with most builds now running in 10 minutes or less. The new system allows for isolated environments, reducing build flakiness and enabling horizontal scalability as demand grows.
How does Shopify handle iOS app builds in CI?
iOS applications must be built on macOS due to Apple's EULA. Shopify uses fastlane match to manage certificates and provisioning profiles, and employs a manual approach for better control over the signing process in CI environments.
What role does caching play in the new CI system?
Caching is crucial for speeding up builds in the new CI system. It allows projects to specify directories for caching, ensuring that dependencies are available for each build, thus improving build times significantly.
Why did Shopify choose Buildkite over other CI services?
Shopify opted for Buildkite due to its flexibility, ability to use their own infrastructure, and the need for better performance and scalability than third-party services like CircleCI and Travis CI could provide.

Key Statistics & Figures

Build time for mobile CI
10 minutes or less
Most mobile CI builds now complete in 10 minutes or less due to the new system.
Ubuntu deploy time
5 minutes
Deploy times for Ubuntu VMs improved from roughly 40 minutes to 5 minutes.
macOS deploy time
15 minutes
macOS deploy times decreased from 120 minutes to 15 minutes.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

CI/CD
Buildkite
Forms the backbone of Shopify's CI systems, allowing for flexible job scheduling and execution.
Automation
Fastlane
Used for managing certificates and provisioning profiles for iOS builds.
Virtualization
Vmware
Used for managing virtual machines in the CI environment.
Database
Redis
Maintains a list of cached directories and revisions for build optimization.

Key Actionable Insights

1
Invest in high-performance hardware for CI systems to improve deployment times.
Using a shared SAN for VM templates significantly reduced deployment times from hours to minutes, enabling horizontal scalability and efficient resource management.
2
Utilize linked clones in your virtual machine setup to optimize storage and speed.
Linked clones allow for faster creation of VMs and use less disk space, which is essential for maintaining performance in a CI environment.
3
Implement a caching layer to maintain build speed across disposable worker VMs.
By caching dependencies and using RAM disks, you can significantly reduce I/O load and improve the performance of your CI pipeline.

Common Pitfalls

1
Failing to manage VM images properly can lead to slow deployment times.
Without a shared SAN and efficient image management, deploying VM images can take hours, hindering CI performance.
2
Not implementing proper caching can result in longer build times.
If caching strategies are not in place, each build starts from scratch, which can significantly slow down the CI process.