Ensuring Balanced GPU Allocation in Kubernetes Clusters with Time-Based Fairshare

NVIDIA Run:ai v2.24 introduces time-based fairshare, a new scheduling mode that brings fair-share scheduling with time awareness for over-quota resources to…

NVIDIA logo
NVIDIA
•
Ekin Karabulut
•11 min read•advanced•
--
•View Original
KubernetesPrometheusYAML

Overview

The article discusses the introduction of time-based fairshare in NVIDIA Run:ai v2.24, a scheduling mode that enhances GPU allocation fairness in Kubernetes clusters by considering historical resource usage. This approach addresses the challenges of resource contention between teams with different workload patterns, ensuring equitable access to over-quota GPU resources.

What You'll Learn

1

How to enable time-based fairshare in NVIDIA Run:ai and KAI Scheduler

2

Why time-based fairshare improves GPU resource allocation fairness

3

When to apply time-based fairshare for optimal resource utilization

Key Questions Answered

What is time-based fairshare in NVIDIA Run:ai?
Time-based fairshare is a scheduling mode that allocates GPU resources based on historical usage rather than just current demand. It ensures that teams that have been waiting for resources receive priority, balancing access to over-quota resources over time.
How does time-based fairshare improve GPU resource allocation?
Time-based fairshare improves GPU resource allocation by tracking historical usage and adjusting the effective weight of queues. This allows teams that have been starved of resources to receive a boost, ensuring fair access to over-quota resources across different workloads.
What are the benefits of using time-based fairshare?
The benefits of time-based fairshare include protected critical workloads, burst access for teams with occasional large jobs, fair sharing over time, and improved treatment of large workloads, preventing any single team from monopolizing resources indefinitely.
How does the scheduler calculate fair share with time-based fairshare?
The scheduler calculates fair share by comparing the proportion of over-quota resources consumed by each queue against the proportion they should have received based on their configured weights. Adjustments are made to balance future allocations accordingly.

Technologies & Tools

Software
Nvidia Run:ai
A platform that integrates time-based fairshare for GPU resource scheduling in Kubernetes clusters.
Software
Kai Scheduler
An open-source scheduler that implements time-based fairshare for managing GPU resources.

Key Actionable Insights

1
Implement time-based fairshare to enhance resource allocation fairness in your Kubernetes clusters.
This approach allows for equitable access to GPU resources, especially in environments where workloads vary significantly, ensuring that no team is consistently starved of resources.
2
Utilize the KAI Scheduler's time-based fairshare simulator to model different queue configurations.
This tool helps visualize how resource allocations change over time, allowing administrators to experiment with settings before applying them to production environments.
3
Adjust the K-value parameter in time-based fairshare to control the speed of weight adjustments.
A higher K-value results in faster corrections of resource imbalances, which can be crucial in dynamic environments where workloads fluctuate frequently.

Common Pitfalls

1
Failing to track historical usage can lead to unfair resource allocation.
Without memory of past usage, the scheduler may favor teams that continuously submit smaller jobs, leaving larger jobs waiting indefinitely. Implementing time-based fairshare mitigates this issue.