Overview
The article discusses how ClickHouse Cloud optimized its Kubernetes pod allocation in AWS EKS by changing the scheduling policy to MostAllocated, resulting in significant cost savings. By implementing a custom scheduler, they increased resource utilization and reduced the number of EC2 nodes required, achieving over 20% reduction in infrastructure costs.
What You'll Learn
1
How to optimize Kubernetes pod scheduling to improve resource utilization
2
Why changing the scheduling policy can reduce infrastructure costs
3
How to implement a custom Kubernetes scheduler in EKS
Prerequisites & Requirements
- Understanding of Kubernetes scheduling and resource management
- Familiarity with AWS EKS and Kubernetes custom resources(optional)
Key Questions Answered
How did ClickHouse improve pod allocation in AWS EKS?
ClickHouse improved pod allocation by changing the Kubernetes scheduler's scoring policy from LeastAllocated to MostAllocated. This adjustment allowed for more efficient packing of pods onto nodes, increasing overall resource utilization and reducing the number of EC2 nodes required.
What were the results of changing the scheduling policy?
After implementing the MostAllocated scheduling policy, ClickHouse increased cluster utilization to 70% and achieved over 20% reduction in EC2 costs. This change also led to a decrease in the number of EC2 nodes by about 10%, confirming significant cost savings.
What challenges did ClickHouse face when optimizing EKS node utilization?
ClickHouse faced challenges such as low pod utilization due to the default LeastAllocated scheduling policy, which favored nodes with higher available resources. This led to sparse pod distribution and increased costs due to underutilized nodes.
How did ClickHouse ensure minimal disruption during the rollout?
To ensure minimal disruption, ClickHouse implemented a PodDisruptionBudget and rolled out the custom scheduler gradually, starting with smaller instances and regions. This careful approach avoided impacting customer workloads during the transition.
Key Statistics & Figures
Cluster utilization
70%
Increased from previous utilization levels after implementing the MostAllocated scheduling policy.
Reduction in EC2 costs
over 20%
Achieved as a result of improved pod allocation and reduced number of EC2 nodes.
Reduction in number of nodes
about 10%
Resulted from more efficient packing of ClickHouse pods onto fewer nodes.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Orchestration
Kubernetes
Used for managing ClickHouse clusters in AWS EKS.
Cloud Service
AWS Eks
Elastic Kubernetes Service used for hosting ClickHouse clusters.
Key Actionable Insights
1Implementing a custom scheduling policy can lead to significant cost savings in cloud environments.By switching to the MostAllocated policy, ClickHouse achieved over 20% reduction in EC2 costs, demonstrating the financial benefits of optimizing resource allocation.
2Monitoring node utilization is crucial for identifying inefficiencies in cloud infrastructure.ClickHouse's analysis of EKS node utilization revealed that many nodes were underutilized, prompting the need for a more efficient scheduling strategy.
3Gradual rollout of changes can minimize disruption to customer services.ClickHouse's phased approach to implementing the new scheduler ensured that customer workloads remained stable while optimizing resource usage.
Common Pitfalls
1
Relying on the default Kubernetes scheduler can lead to inefficient resource utilization.
The default LeastAllocated policy may result in sparse pod distribution, which increases costs and reduces overall efficiency. Customizing the scheduling policy is essential for optimizing resource usage.
Related Concepts
Kubernetes Scheduling
Resource Management In Cloud Environments
Cost Optimization Strategies