Per-Tensor and Per-Block Scaling Strategies for Effective FP8 Training

In this blog post, we’ll break down the main FP8 scaling strategies—per-tensor scaling, delayed and current scaling, and per-block scaling (including the…

NVIDIA logo
NVIDIA
•
Karin Sevegnani
•9 min read•advanced•
--
•View Original
Transformer

Overview

This article discusses FP8 scaling strategies, including per-tensor and per-block scaling, essential for maintaining numerical stability and accuracy during low-precision training. It highlights the importance of these strategies in the context of NVIDIA Nemotron experiments and provides insights into advanced recipes like Micro-Scaling FP8 (MXFP8).

What You'll Learn

1

How to implement per-tensor scaling strategies for FP8 training

2

Why delayed scaling improves numerical stability in FP8 training

3

How to utilize Micro-Scaling FP8 (MXFP8) for efficient training

4

When to apply per-block scaling for large models

Prerequisites & Requirements

  • Understanding of low-precision training techniques
  • Familiarity with NVIDIA NeMo framework(optional)

Key Questions Answered

What is per-tensor scaling and why is it important?
Per-tensor scaling assigns unique scaling factors to each tensor, which is crucial due to FP8's narrow dynamic range. This approach prevents numerical instability and ensures accurate training by tailoring the scaling to each tensor's statistics.
How does delayed scaling enhance FP8 training stability?
Delayed scaling computes scaling factors based on maximum absolute values observed over several iterations, smoothing out outliers and reducing abrupt changes. This history-based approach helps maintain stability during training.
What is Micro-Scaling FP8 (MXFP8) and how does it work?
Micro-Scaling FP8 (MXFP8) is a hardware-level solution that implements blockwise scaling for efficient FP8 training. It divides tensors into blocks, each with its own scaling factor, optimizing performance while minimizing quantization errors.
What are the benefits of per-block scaling in large models?
Per-block scaling allows for dedicated scaling factors for smaller blocks within a tensor, accommodating local statistical variations. This method enhances precision and stability, especially in complex models with diverse value distributions.

Technologies & Tools

Framework
Nvidia Nemo
Used for implementing FP8 training recipes and scaling strategies.
Hardware
Mxfp8
A hardware-level solution for efficient and stable FP8 training.

Key Actionable Insights

1
Implement per-tensor scaling to enhance the accuracy of FP8 training workflows.
This method tailors scaling factors to individual tensors, which is vital for maintaining numerical stability in low-precision training, especially in large models.
2
Utilize delayed scaling to mitigate the impact of outliers during training.
By basing scaling factors on historical maximum values, you can achieve smoother training processes and avoid abrupt changes that could destabilize model performance.
3
Adopt Micro-Scaling FP8 (MXFP8) for advanced FP8 training configurations.
This approach maximizes the utilization of FP8's dynamic range and minimizes quantization errors, making it particularly effective for large-scale deep learning applications.

Common Pitfalls

1
Relying solely on a single scaling factor for diverse tensors can lead to numerical instability.
This happens because different regions within a tensor may have varying statistical behaviors, which a single scale cannot accommodate. Using per-block scaling helps mitigate this issue.

Related Concepts

Low-precision Training Techniques
Numerical Stability In Deep Learning
Advanced Scaling Strategies For AI Models