E-Commerce at Scale: Inside Shopify's Tech Stack - Stackshare.io

9 minute read Before 2015, we had an Operations and Performance team. Around this time, we decided to create the Production Engineering department and merge the teams. The department is responsible for building and maintaining common infrastructure that allows the rest of product development teams to run their code. Both Production Engineering and all the product development teams share responsibility for the ongoing operation of our end user applications. This means all technical roles share monitoring and incident response, with escalation happening laterally to bring in any skill set required to restore service in case of problems.

Shopify logo
Shopify
•
Kir Shatrov
•10 min read•intermediate•
--
•View Original
ActiveRecordDockerHerokuHTMLJavaScriptjQueryKubernetesLuaMemcachedMySQLRailsReactRedisRubyTypeScript

Overview

The article provides an in-depth look at Shopify's tech stack and engineering practices, detailing how the platform scales to support over 600,000 merchants and 80,000 requests per second. It discusses the evolution of Shopify's architecture, the use of various technologies, and the challenges faced in maintaining performance and reliability.

What You'll Learn

1

How to implement sharding for database scalability

2

Why using pods can enhance application reliability

3

How to leverage Docker and Kubernetes for deployment orchestration

4

When to apply feature flags for safe deployments

Prerequisites & Requirements

  • Understanding of distributed systems and database management
  • Familiarity with Docker and Kubernetes(optional)

Key Questions Answered

How does Shopify handle high traffic during flash sales?
Shopify manages high traffic by employing a robust architecture that includes sharding and pods, allowing them to isolate merchants and handle increased loads without affecting the entire platform. This approach minimizes the risk of global outages and ensures that only specific pods are impacted during peak times.
What technologies are used in Shopify's tech stack?
Shopify utilizes a variety of technologies including Ruby on Rails as its core framework, MySQL for relational database management, Redis for queues and background jobs, and Docker and Kubernetes for deployment orchestration. This diverse stack supports the platform's scalability and reliability.
What is the role of ServicesDB in Shopify's infrastructure?
ServicesDB is an internal application that tracks all production services at Shopify, helping developers manage app ownership, uptime, logs, and security updates. It automates issue reporting and facilitates queries about the infrastructure to ensure all services are maintained properly.
How does Shopify ensure fast CI/CD processes?
Shopify employs BuildKite as its CI platform, allowing for parallel test execution across hundreds of CI workers. This setup significantly reduces build times to 15-20 minutes, enabling rapid deployment of new features while maintaining high testing standards.

Key Statistics & Figures

Number of merchants supported
600K
Shopify powers 600,000 merchants on its platform.
Peak requests per second
80K
Shopify serves 80,000 requests per second at peak times.
Number of unit tests in the monolith
100K
The Shopify monolith has around 100,000 unit tests to ensure code quality.
Time taken for a full build
15-20 minutes
The build process for Shopify's monolith takes between 15 to 20 minutes.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
Ruby On Rails
Core framework for building Shopify's applications.
Database
Mysql
Relational database management system used for data storage.
Backend
Redis
Used for queues and background jobs.
Tools
Docker
Containerization platform used for deployment orchestration.
Tools
Kubernetes
Container orchestration platform for managing application deployment.
Tools
Buildkite
Continuous integration platform used for managing builds and tests.
API
Graphql
Used for efficient data fetching in the Shopify Admin.

Key Actionable Insights

1
Implementing sharding can significantly improve database performance and scalability.
By isolating merchants on different database shards, Shopify can handle increased loads without affecting overall performance. This strategy is particularly effective during high-traffic events like flash sales.
2
Utilizing feature flags allows for safer deployments and quicker rollbacks.
Feature flags enable developers to release new features gradually, minimizing risks associated with large-scale changes. This practice is crucial for maintaining service reliability during updates.
3
Adopting a pod architecture can enhance application reliability and reduce downtime.
By deploying isolated pods, Shopify has minimized the impact of outages, ensuring that issues affect only specific regions rather than the entire platform.

Common Pitfalls

1
Relying on shared resources can create single points of failure.
Shopify experienced a major outage due to a single Redis instance that affected all shards. This incident highlighted the importance of isolating resources to prevent widespread disruptions.

Related Concepts

Distributed Systems
Microservices Architecture
Continuous Integration And Deployment
Scalability Patterns