Advancing Our Chef Infrastructure: Safety Without Disruption

Key Actionable Insights

1

Split production environments by availability zone to contain blast radius. Rather than having all production nodes share a single configuration source, map each node to a zone-specific environment during bootstrapping. This ensures that bad configurations only affect a subset of your fleet initially, giving you time to detect and fix issues before they propagate everywhere.

This is especially critical during scale-out events where dozens or hundreds of new nodes could simultaneously pick up a broken configuration from a shared environment.

2

Use a canary environment that updates frequently with the latest changes, separate from the rest of production that follows a slower release train. Prod-1 receiving every new version hourly catches issues close to when they're introduced, while prod-2 through prod-6 only advance after the previous version has fully propagated. This dual-speed approach balances early detection with safety.

Testing large cumulative changes makes it harder to identify which specific change caused a problem. Small, frequent canary updates make root cause analysis significantly easier.

3

Replace fixed-schedule cron-based configuration runs with signal-driven triggers. When environments update at variable times due to staggered rollouts, a fixed schedule can't predict when new changes will be available. A signal-based system that only triggers runs when actual updates exist improves both efficiency and safety.

Slack built Chef Summoner to poll S3 for signals from Chef Librarian, using configurable splay values to stagger runs within each environment and avoid load spikes.

4

Always build a fallback recovery mechanism for critical deployment infrastructure. If your deployment trigger service itself is deployed by the same system it manages, a broken version could prevent its own fix from being rolled out. Bake in an independent safety net (like a cron job) that can trigger deployments when the primary system fails.

Slack's fallback cron checks Chef Summoner's local state and triggers Chef directly if no run has occurred in 12 hours, ensuring the system can always self-heal.

5

Use S3 as a lightweight signaling mechanism between infrastructure services rather than building complex pub/sub systems. By having Chef Librarian write JSON signals to S3 on each promotion and having Chef Summoner poll those signals, Slack created a decoupled, durable communication channel that doesn't require managing additional messaging infrastructure.

The S3 signal contains all necessary metadata including version, cookbook versions, splay configuration, and commit hash, making it a self-contained deployment manifest.

6

When considering major infrastructure migrations like Chef Policyfiles, evaluate the migration cost against incremental improvements to existing systems. Sometimes improving the current system with environment splitting and progressive rollouts achieves similar safety gains without requiring dozens of teams to rewrite their configurations.

Slack chose incremental improvement over a disruptive migration, reserving the complete rebuild (Shipyard) for when they had the resources and design to do it properly.