•Jayme Bird, Manish Gill, Manas Alekar, Francesco Ciocchetti & Ashwath Singh•20 min read•beginner•
--
•View OriginalOverview
The article discusses the implementation of Make Before Break scaling mechanics for ClickHouse Cloud, focusing on the transition from a single StatefulSet architecture to a more flexible MultiSTS approach. It highlights the challenges faced during live migrations and the advantages of this new scaling method in improving resource management and reducing downtime.
What You'll Learn
1
How to implement Make Before Break scaling in ClickHouse Cloud
2
Why StatefulSets can limit scaling flexibility in Kubernetes
3
When to use live migrations for seamless upgrades
Prerequisites & Requirements
- Understanding of Kubernetes StatefulSets and ClickHouse architecture
- Familiarity with Temporal for orchestration(optional)
Key Questions Answered
How does Make Before Break scaling improve resource management?
Make Before Break scaling allows for the immediate addition of new pods with increased capacity before terminating old ones. This approach eliminates rolling restarts, reducing resource crunch during scaling operations and ensuring that the cluster can handle traffic without disruption.
What challenges arise during live migrations in ClickHouse Cloud?
Live migrations face challenges such as handling unfinished DDL queries, maintaining connections with external table engines, and ensuring proper synchronization of metadata. These issues can disrupt service if not managed carefully, requiring robust orchestration and error handling.
What is the role of the migration controller in ClickHouse Cloud?
The migration controller orchestrates the transition from SingleSTS to MultiSTS by managing the creation of new replicas and the removal of old ones. It ensures that customer queries continue to function during the migration process, maintaining service availability.
Why is it important to implement Maintenance Mode during migrations?
Maintenance Mode allows SQL queries to continue processing while disabling cloud operations that could interfere with migrations. This minimizes the risk of disruptions and ensures a smoother transition during critical updates or scaling operations.
Key Statistics & Figures
Maximum time for replica eviction during scaling
1 hour
This is the worst-case scenario for terminating a replica gracefully before it can be restarted with new resources.
Total time to resize a 3 replica cluster
over 3 hours
This includes the time taken for each replica to restart and reinitialize after eviction.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Orchestration
Kubernetes
Used to manage StatefulSets and orchestrate the scaling of ClickHouse server pods.
Orchestration
Temporal
Utilized for managing migration workflows and ensuring durable execution of migration tasks.
Key Actionable Insights
1Implement Make Before Break scaling to enhance ClickHouse Cloud performance.This approach reduces downtime and resource strain during scaling operations, making it essential for maintaining high availability in cloud environments.
2Utilize Maintenance Mode during migrations to prevent disruptions.By temporarily disabling cloud operations while allowing queries to run, you can ensure a seamless experience for users during critical updates.
3Leverage Temporal for orchestrating complex migration workflows.Temporal's durable execution and scheduling capabilities can simplify the management of migrations, making it easier to handle large-scale transitions without downtime.
Common Pitfalls
1
Failing to account for ongoing queries during pod eviction can lead to service disruptions.
It's essential to wait for long-running queries to complete or to handle them appropriately to avoid impacting user experience.
2
Not implementing a proper Pod Disruption Budget can result in too many replicas being restarted simultaneously.
This can cause significant downtime and resource strain, highlighting the need for careful planning in scaling operations.
Related Concepts
Kubernetes Statefulsets
Clickhouse Architecture
Cloud Migration Strategies
Orchestration With Temporal