In MLPerf Inference v3.0, NVIDIA made its first submissions to the newly introduced Network division, which is now part of the MLPerf Inference Datacenter suite.
Overview
The article discusses NVIDIA's submissions to the newly introduced MLPerf Inference Network division, highlighting the integration of NVIDIA InfiniBand and GPUDirect RDMA technology to enhance end-to-end inference performance in data center setups. It details the architecture, performance metrics, and technologies that contribute to NVIDIA's success in the MLPerf Inference v3.0 benchmarks.
What You'll Learn
1
How to utilize GPUDirect RDMA for low-latency, high-throughput tensor communication
2
Why the Network division in MLPerf Inference is crucial for benchmarking data center setups
3
How to implement a query dispatch library (QDL) for efficient inference requests
Prerequisites & Requirements
- Understanding of machine learning inference and data center architecture
- Familiarity with NVIDIA TensorRT and InfiniBand technology(optional)
Key Questions Answered
What are the key components of the NVIDIA Network division in MLPerf Inference?
The NVIDIA Network division consists of Frontend nodes that generate queries and Accelerator nodes that process these queries. These nodes communicate through high-speed networks like InfiniBand, which enhances performance by integrating networking components directly into the Accelerator nodes.
How does NVIDIA achieve high performance in MLPerf Inference v3.0?
NVIDIA achieved 100% of single-node performance on ResNet-50 and 94% on BERT workloads by leveraging GPUDirect RDMA technology and ConnectX-6 InfiniBand smart adapter cards, which provide extremely high bandwidth and low latency.
What optimizations are used in the NVIDIA Network division submission?
Key optimizations include transaction trackers per IBConnection for lock-free bookkeeping, multiple QP support per NIC for scalability, and the use of INLINE transfers for small messages to improve efficiency. These optimizations enhance both performance and resource utilization.
Key Statistics & Figures
ResNet-50 performance
100%
Achieved in the Network division using GPUDirect RDMA technology.
BERT (Low Accuracy) performance
94%
Achieved in the Network division compared to Closed division submissions.
BERT (High Accuracy) performance
90%
Achieved in the Network division compared to Closed division submissions.
Technologies & Tools
Backend
Nvidia Tensorrt
Used for optimized inference engine.
Networking
Infiniband
Provides high-throughput, low-latency communication for data transfers.
Networking
Gpudirect Rdma
Enables direct memory access for efficient data transfers between GPUs and NICs.
Networking
Nvidia Connectx-6
Smart adapter cards used to achieve high network performance.
Key Actionable Insights
1Implementing GPUDirect RDMA can significantly reduce latency in data transfers between nodes in a data center.This is particularly beneficial for applications requiring real-time processing, such as AI inference workloads, where every millisecond counts.
2Utilizing a NUMA-aware implementation can optimize resource allocation across CPUs, GPUs, and NICs.This ensures that the closest resources are used for processing, which can lead to improved performance in multi-node setups.
3Adopting a query dispatch library (QDL) can streamline the process of managing inference requests across multiple nodes.This approach abstracts the complexity of load balancing and resource management, allowing developers to focus on optimizing inference performance.
Common Pitfalls
1
Failing to implement NUMA-aware resource allocation can lead to suboptimal performance in multi-node environments.
This occurs because resources may not be utilized efficiently, resulting in increased latency and reduced throughput.
2
Neglecting the importance of buffer management can cause memory bottlenecks during high transaction loads.
Without proper management, the system may struggle to handle the volume of queries, leading to dropped requests and degraded performance.
Related Concepts
Machine Learning Inference Optimization
Data Center Networking Technologies
High-performance Computing Architectures