Overview
The article discusses strategies to avoid CPU throttling in a containerized environment, particularly at Uber, where stateful workloads run on a large fleet of hosts. It highlights the transition from CPU quotas to cpusets, which has led to improved latency and reduced core allocation.
What You'll Learn
1
How to implement cpusets for CPU isolation in containerized environments
2
Why switching from CPU quotas to cpusets can improve latency
3
When to use CPU pinning to avoid throttling in stateful workloads
Prerequisites & Requirements
- Understanding of Linux kernel scheduling and resource allocation
- Familiarity with container orchestration platforms like Kubernetes(optional)
Key Questions Answered
How does switching to cpusets affect CPU allocation and latency?
Switching to cpusets allows for CPU pinning, which isolates containers to specific cores, reducing latency variance. This change has led to a fleet-wide core allocation reduction of up to 11% due to decreased resource requirement variance.
What are the downsides of using cpusets for CPU management?
While cpusets improve latency stability, they have limitations such as the inability to allocate fractional cores, potential time theft by system-wide processes, and the need for defragmentation of core allocations over time.
Why do CPU quotas lead to throttling in multi-threaded containers?
CPU quotas can cause throttling because multi-threaded containers may consume their allocated quota too quickly, leading to increased latencies for requests that typically complete in milliseconds, sometimes exceeding 100 ms.
Key Statistics & Figures
Core allocation reduction
up to 11%
This reduction was achieved by switching from CPU quotas to cpusets, leading to less variance in resource requirements.
P99 latencies improvement
around 50%
This improvement was noted due to the removal of heavy throttling when cpusets were implemented.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Key Actionable Insights
1Implementing cpusets can significantly stabilize latency for stateful workloads.By ensuring that each container runs on dedicated cores, you can avoid the performance penalties associated with CPU throttling, leading to more predictable application behavior.
2Monitor CPU allocation and performance metrics regularly.Regular monitoring helps identify potential issues with resource allocation and allows for timely adjustments, ensuring optimal performance across your containerized applications.
3Consider the trade-offs between using cpusets and CPU quotas.While cpusets provide better latency consistency, they require careful management of core assignments and may not allow for resource bursting, which can be beneficial in certain scenarios.
Common Pitfalls
1
Allocating CPUs incorrectly can lead to significant performance degradation.
If containers are scheduled across different physical sockets, it can result in increased latencies due to remote memory access, which should be avoided by understanding CPU topology.
Related Concepts
CPU Isolation Techniques
Resource Allocation Strategies
Performance Optimization In Containerized Environments