SkyPilot at Shopify: Multi-cloud GPUs without the pain

Key Actionable Insights

1

Use a policy plugin architecture to inject organizational logic into your ML job scheduling system. By intercepting requests before they reach a cluster, you can validate labels, route to different providers, and inject configurations — all without changing the user-facing interface. This keeps the abstraction in the routing layer rather than the interface layer.

Shopify's SkyPilot plugin handles routing, cost tracking, and configuration injection transparently, so engineers just write YAML and run sky launch without worrying about cloud-specific details.

2

Enforce cost attribution at the platform level by requiring a cost-owner label on every job submission. Making cost tracking mandatory (rejecting jobs without it) ensures complete visibility into GPU spend without relying on manual processes or after-the-fact audits.

Shopify requires showback_cost_owner_ref on every job. Teams see their costs in dashboards and self-correct usage patterns, eliminating the need for finance teams to chase down cost attribution.

3

Implement shared caches for model weights and package dependencies across GPU jobs. The first download of a large model like llama-70b gets cached on shared storage, and subsequent jobs start instantly without re-downloading, which saves significant time at scale.

Shopify mounts /mnt/uv-cache for Python packages and /mnt/huggingface-cache for model weights automatically, which adds up when running hundreds of jobs that share common dependencies.

4

Deploy a GPU utilization reaper service that automatically terminates jobs running below a utilization threshold (e.g., 20%) for extended periods. This recovers wasted GPU resources from stuck or runaway training jobs, but make sure to exempt interactive development environments from this enforcement.

Shopify's GPU reaper catches runaway training jobs while exempting dev environments, since low utilization during active debugging is expected and normal behavior.

5

Use Kueue for Kubernetes-native fair-share GPU scheduling with a priority class hierarchy. Define priority tiers (emergency, interactive, automated-low-priority, lowest) that allow emergency jobs to preempt batch work and interactive sessions to schedule faster than automated pipelines.

Kueue solved Shopify's fair scheduling problem better than anything they could have built in-house, handling quota-based scheduling and preemption across teams automatically.

6

When designing multi-cloud GPU infrastructure, keep training data replicated across cloud providers so jobs run where the data already exists. This eliminates cross-cloud data transfer latency and keeps data under your control regardless of which provider runs the workload.

Shopify replicates training datasets across clouds — when training on Nebius, data comes from local volumes; same for GCP — so the routing decision doesn't create a data movement problem.