How Shopify Uses Kubernetes

Shopify uses SkyPilot, an open-source framework, to manage GPU-intensive ML training workloads across multiple cloud providers (Nebius and GCP).

The article discusses the evolution of product classification at Shopify, detailing the transition from basic categorization to a comprehensive understanding of products using Vision Language Model...

The article discusses Shopify's Merlin machine learning platform, focusing on its online inference capabilities for real-time predictions.

The article discusses Shopify's approach to performance testing at scale, particularly during the critical Black Friday Cyber Monday (BFCM) period.

This article narrates the journey of Cailyn Edwards, a former farmer who transitioned into a successful career as a Senior Infrastructure Security Engineer at Shopify through the Dev Degree program.

The article discusses Shopify's innovative compensation system, Flex Comp, which allows employees to customize their total rewards.

This article discusses how Shopify scaled its Server Pixels service to enhance event processing performance by 170% using Go worker pools.

The article discusses Shopify's transition to using Spin, a cloud-based development tool, to enhance the development process for the Shopify Fulfillment Network (SFN) team.

The article discusses the Spin infrastructure team's transition to Container Optimized OS (COS) and the challenges faced with Kubernetes, systemd, and cgroups.

Shopify's journey to cloud development involved transitioning from a local development tool called 'dev' to a more scalable and flexible cloud-native solution named Isospin.

The article discusses the experiences and lessons learned from running Apache Airflow at scale within Shopify.

The article discusses Shopify's initiative to unify its traffic management through a new system called Möbius.

Shopify's new machine learning platform, Merlin, is designed to enhance the efficiency of data scientists by providing a robust infrastructure and tools for machine learning workflows.

This article provides seven actionable tips for optimizing Apache Flink applications, focusing on performance and resiliency.

This article serves as a comprehensive guide for aspiring engineers looking to secure an internship at Shopify.

The article discusses a significant incident at Shopify caused by the expiration of Let’s Encrypt’s old root certificate on September 30, 2021, which nearly led to a global outage.

This article discusses the recent MySQL upgrade at Shopify, detailing the motivations behind the upgrade, the challenges faced during the process, and the internal tools developed to streamline fut...

Shopify's Data Reliability team developed custom verification, benchmarking, and profiling tooling to enhance Trino query execution speed and reliability.

The article discusses the challenges faced by Shopify's KateSQL, a Database-as-a-Service platform, in managing MySQL instances on Kubernetes, particularly focusing on a performance issue caused by ...

The article discusses Shopify's approach to shard balancing within its MySQL database infrastructure, emphasizing the importance of maintaining balanced database utilization to prevent failures and...

The article discusses the unique patterns developed by Shopify for their Ruby on Rails applications, focusing on tools and architectures that enhance developer productivity and maintainability.

The article discusses Shopify's efforts to enhance the performance of Trino, a distributed SQL query engine, to provide faster query execution times for data scientists.

The article discusses Shopify's evolution from a traditional data warehouse to a more dynamic data platform using Change Data Capture (CDC) and event streaming technologies.

The article discusses the management of project-wide SSH keys in Google Cloud Platform (GCP) and the potential security risks associated with persistent SSH keys.

This article discusses the optimization of Shopify's server-side storefront rendering engine, focusing on techniques such as simplifying data access, implementing caching layers, and reducing memor...

This article discusses how Shopify's Data Platform Engineering team scaled their data platform to handle the unprecedented data volume during Black Friday and Cyber Monday (BFCM).

The article discusses Shopify's capacity planning strategies for handling increased traffic during the Black Friday and Cyber Monday (BFCM) shopping period.

The article discusses the importance of securing Docker images through code signing and Binary Authorization, particularly in the context of Kubernetes.

This article discusses Shopify's journey in implementing DNS traffic management to enhance the resiliency of their services.

The article outlines Shopify's evolution in managing TLS certificates, detailing the transition from manual provisioning to a fully automated system that now serves over 1 million merchants.

The article discusses the software release culture at Shopify, emphasizing the importance of developer autonomy, feedback mechanisms, and an efficient release process.

Shopify has transitioned to building all new mobile apps using React Native, a decision driven by the increasing mobile purchase rates and the need for a unified development approach.

This article discusses five common data stores—Relational Database, Non-Relational (NoSQL) Database, Key-Value Store, Full-Text Search Engine, and Message Queue—highlighting their characteristics a...

This article discusses how Shopify manages MySQL backups and restores at a petabyte scale using Google Cloud Platform's Persistent Disk snapshots.

The article discusses a new kubectl plugin for the ingress-nginx Kubernetes Ingress Controller, which simplifies the upgrade and debugging processes for users.

The article discusses Shopify's engineering efforts in preparation for the legalization of cannabis in Canada on October 17, 2018.

The article reviews Shopify's Bug Bounty program for 2018, highlighting its achievements, including a significant reduction in triage time and increased payouts to hackers.

The article discusses Shopify's journey of running Apache Kafka on Kubernetes, detailing the transition from data centers to cloud infrastructure and the benefits of using Kubernetes for managing K...

The article discusses Shopify's transition to a more scalable ingress solution using ingress-nginx, highlighting the challenges faced with the previous Google Cloud Load Balancer Controller and the...

The article provides an in-depth look at Shopify's tech stack and engineering practices, detailing how the platform scales to support over 600,000 merchants and 80,000 requests per second.

Shopify is collaborating with Google to enhance its cloud infrastructure, focusing on scalability and performance for its commerce platform.

The article discusses Shopify's transition to a Pods architecture to enhance scalability and resilience after facing limitations with database sharding.

The article discusses how Shopify utilizes Grafeas and Kritis to manage container security and governance at scale.

This article discusses the development of a dynamic mobile Continuous Integration (CI) system at Shopify, highlighting the transition from a static setup to a scalable, efficient architecture.