RoCE networks for distributed AI training at scale

AI networks play an important role in interconnecting tens of thousands of GPUs together, forming the foundational infrastructure for training, enabling large models with hundreds of billions of pa…

Meta logo
Meta
•
Adi Gangidi
•11 min read•advanced•
--
•View Original

Overview

The article discusses the design and implementation of RDMA over Converged Ethernet (RoCE) networks at Meta for large-scale distributed AI training. It highlights the challenges and solutions in building a robust network infrastructure to support the increasing demands of AI workloads, particularly for large language models.

What You'll Learn

1

How to design a dedicated backend network for distributed AI training

2

Why RDMA over Converged Ethernet (RoCEv2) is preferred for AI workloads

3

How to implement effective congestion control in high-speed networks

4

When to utilize path-pinning for routing in AI training workloads

Prerequisites & Requirements

  • Understanding of distributed systems and networking concepts
  • Experience with AI training workloads and GPU clusters(optional)

Key Questions Answered

What are the main challenges in building networks for distributed AI training?
The main challenges include managing high communication demands, ensuring low latency and high bandwidth, and accommodating the unique traffic patterns of AI workloads. These factors necessitate a reevaluation of traditional data center network designs to support the coordination of tens of thousands of GPUs effectively.
How does the AI Zone topology enhance GPU connectivity?
The AI Zone topology employs a two-stage Clos architecture that allows for non-blocking connectivity among GPUs. This design facilitates scalability and high availability, ensuring that the network can handle the extensive interconnections required for large AI models without bottlenecks.
What is the impact of using Enhanced ECMP with queue pair scaling?
Enhanced ECMP, combined with queue pair scaling, significantly increases flow entropy, leading to performance improvements of up to 40% for collective operations like AllReduce. This approach allows for better traffic distribution and reduces congestion in high-demand scenarios.
How is congestion controlled in RoCE networks for AI workloads?
Congestion is managed through a combination of techniques, including receiver-driven traffic admission and careful coordination between the collective communication library and the network. This approach limits in-flight traffic and ensures stable performance even under high load conditions.

Key Statistics & Figures

Performance improvement from Enhanced ECMP and queue pair scaling
up to 40%
This improvement was observed specifically for the AllReduce collective operation.
Reduction in training performance degradation due to RTSW uplink upgrades
more than 30%
This degradation occurred when uneven traffic distribution and congestion affected the training performance.

Technologies & Tools

Networking
Rdma Over Converged Ethernet Version 2 (rocev2)
Used as the inter-node communication transport for AI workloads.
Software
Nccl
Utilized for collective communication in GPU-to-GPU communication architecture.

Key Actionable Insights

1
Implement a dedicated backend network for distributed AI training to optimize performance.
By segregating the frontend and backend networks, you can tailor the infrastructure to better meet the specific demands of AI workloads, leading to improved efficiency and reduced latency.
2
Utilize path-pinning for routing to enhance the performance of AI training jobs.
This method can help manage traffic more effectively by ensuring that packets follow predetermined paths, which is crucial for maintaining high throughput in environments with predictable flow patterns.
3
Adopt Enhanced ECMP with queue pair scaling to improve network performance.
This strategy can significantly increase the number of concurrent flows, which is essential for handling the bursty nature of AI workloads and ensuring that network resources are utilized efficiently.
4
Focus on receiver-driven traffic admission to manage congestion effectively.
This approach allows for better control over in-flight traffic, particularly in high-speed networks, which is critical for maintaining performance during peak loads.

Common Pitfalls

1
Relying solely on path-pinning can lead to uneven traffic distribution if jobs are not fully allocated to racks.
This situation can cause congestion and degrade performance, highlighting the need for dynamic job allocation strategies that consider network conditions.
2
Using default DCQCN settings without adjustments for 400G networks can result in performance degradation.
As seen in the article, transitioning to higher speeds requires careful tuning of congestion control mechanisms to avoid introducing new bottlenecks.

Related Concepts

Distributed AI Training Infrastructure
Network Topology Design For AI Workloads
Congestion Control Mechanisms In High-speed Networks