Advanced Rollout Techniques: Custom Strategies for Stateful Apps in Kubernetes

In a previous blog post—A Simple Kubernetes Admission Webhook—I discussed the process of creating a Kubernetes webhook without relying on Kubebuilder. At Slack, we use this webhook for various tasks, like helping us support long-lived Pods (see Supporting Long-Lived Pods), and today, I delve once more into the topic of long-lived Pods, focusing on our…

Slack logo
Slack
•
Clément Labbe
•13 min read•advanced•
--
•View Original
AWSChefConsulEnvoyKubernetesReactVault

Overview

The article discusses advanced rollout techniques for stateful applications in Kubernetes, focusing on the development of the Bedrock Rollout Operator at Slack. It addresses the limitations of existing StatefulSet update strategies and presents a custom solution that enhances deployment control and integrates with internal tools.

What You'll Learn

1

How to implement a custom rollout strategy for StatefulSets in Kubernetes

2

Why the OnDelete strategy can lead to version leaks in StatefulSets

3

How to integrate Slack notifications into Kubernetes deployments

Prerequisites & Requirements

  • Understanding of Kubernetes StatefulSets and custom resources
  • Familiarity with Slack API and Consul for service discovery(optional)

Key Questions Answered

What are the limitations of the default StatefulSet update strategies?
The default StatefulSet update strategies, OnDelete and RollingUpdate, have significant limitations. OnDelete requires manual intervention to update Pods, while RollingUpdate can be slow, allowing only one Pod to be updated at a time, which is inefficient for large applications with hundreds of Pods.
How does the Bedrock Rollout Operator enhance StatefulSet deployments?
The Bedrock Rollout Operator allows for faster, controlled rollouts of StatefulSets by enabling percent-based deployments, rollbacks, and integration with Slack for notifications. It manages a custom resource named StatefulsetRollout, which includes additional parameters for deployment control.
What is the version leak issue in StatefulSets?
The version leak issue occurs when some Pods in a StatefulSet run an older version while others run a newer version due to the OnDelete strategy. If an old version Pod is terminated for any reason, it may be replaced by a new version Pod, leading to an unintended rollout of the new version.
What integrations does the Bedrock Rollout Operator support?
The Bedrock Rollout Operator integrates with several internal tools, including Consul for service discovery, Vault for secrets management, and Slack for real-time notifications about rollout statuses. This integration enhances deployment visibility and control.

Key Statistics & Figures

Number of Kubernetes clusters at Slack
Over 200
This scale highlights the complexity and need for efficient rollout strategies in large environments.
Number of stateful services at Slack
Nearly 100
Managing a significant number of stateful services necessitates robust deployment strategies.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Orchestration
Kubernetes
Used for managing containerized applications and deploying StatefulSets.
Service Discovery
Consul
Used for service discovery and health checking during rollouts.
Secrets Management
Vault
Used for rendering secrets in conjunction with deployments.
Communication
Slack API
Used for sending notifications about rollout statuses.

Key Actionable Insights

1
Implement a custom rollout strategy using the Bedrock Rollout Operator to gain more control over StatefulSet deployments.
This approach allows for faster rollouts and the ability to pause or rollback deployments, which is crucial for maintaining application stability during updates.
2
Utilize Slack notifications to keep teams informed about the status of deployments.
Real-time updates on rollout progress can enhance team communication and ensure that all stakeholders are aware of deployment statuses, reducing potential confusion.
3
Be aware of the version leak issue when using the OnDelete strategy for StatefulSets.
Understanding this limitation can help teams plan their rollout strategies more effectively and avoid unintended consequences during updates.

Common Pitfalls

1
Using the OnDelete strategy without understanding its implications can lead to version leaks.
This occurs when Pods running different versions coexist, potentially causing confusion and instability during deployments.

Related Concepts

Kubernetes Statefulsets
Custom Resources In Kubernetes
Service Discovery With Consul
Secrets Management With Vault