Scaling data ingestion for machine learning training at Meta

Many of Meta’s products, such as search, ads ranking and Marketplace, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastru…

Meta logo
Meta
•
Aarti Basant
•13 min read•intermediate•
--
•View Original
ApachePyTorch

Overview

The article discusses how Meta scales its data ingestion infrastructure to support machine learning training for various AI models. It highlights the challenges posed by the rapid growth of data and the need for efficient data processing to improve model performance while managing power constraints in data centers.

What You'll Learn

1

How to optimize data ingestion pipelines for machine learning training

2

Why feature flattening improves data reading efficiency

3

How to implement last-mile transformations for AI training models

4

When to apply tiered storage solutions for large datasets

Prerequisites & Requirements

  • Understanding of data ingestion and machine learning concepts
  • Familiarity with distributed file systems like Tectonic(optional)

Key Questions Answered

How has Meta improved its data ingestion throughput?
Meta has achieved a 3-4x increase in data ingestion throughput over the last two years by optimizing its data ingestion infrastructure and implementing new last-mile transformation pipelines. This allows for better handling of the growing volume of data required for training AI models.
What is the role of the Data PreProcessing tier (DPP) in Meta's infrastructure?
The Data PreProcessing tier (DPP) serves as the reader tier for data ingestion and last-mile transformations, fetching data from Tectonic clusters, decrypting it, extracting features, and converting it to tensor formats for AI training. This disaggregated architecture allows for independent scaling of data ingestion and training hardware.
What optimizations have been made to improve memory consumption in data reading?
To optimize memory consumption, Meta implemented DPP client-side rebatching, allowing smaller batches to be read concurrently while appending them on the training node. This approach mitigates out-of-memory errors and improves throughput by 20-40% on reader nodes.
How does feature flattening enhance data reading efficiency?
Feature flattening represents each feature as a stream on disk, enabling more efficient reading of subsets of features. This new storage format has yielded data reading efficiency improvements of 2-2.3x in production workloads, significantly benefiting models that selectively read features.

Key Statistics & Figures

Data ingestion throughput improvement
3-4x
Achieved over the last two years through infrastructure optimizations.
Power budget improvement
35-45%
Resulting from optimizations in the data ingestion infrastructure.
Data reading efficiency gains from feature flattening
2-2.3x
Observed in production workloads that selectively read a smaller subset of features.

Technologies & Tools

Storage
Tectonic
Meta’s exabyte-scale distributed file system that serves as a disaggregated storage infrastructure for AI training models.
Data Format
Dwrf
A hybrid columnar format used for encoding AI training datasets.

Key Actionable Insights

1
Implement feature flattening in your data ingestion pipeline to improve reading efficiency.
By representing features as contiguous streams, you can reduce the overhead of reading unnecessary data, which is particularly beneficial for models that only require a subset of features.
2
Utilize the Data PreProcessing tier (DPP) to scale data ingestion independently from training hardware.
This allows for flexibility in managing resources and ensures that your training models can handle large datasets without being bottlenecked by data ingestion speeds.
3
Consider tiered storage solutions for datasets that require single-pass processing.
By using SSDs for caching high-reuse features, you can significantly reduce data access times and improve overall training efficiency.

Common Pitfalls

1
Overreading features during data ingestion can lead to inefficiencies and increased I/O demands.
This often occurs when models are designed to read more data than necessary, resulting in wasted resources. To avoid this, implement feature reordering and flattening to ensure that only required features are accessed.

Related Concepts

Data Ingestion Optimization Techniques
Machine Learning Model Training Strategies
Distributed File Systems And Their Benefits