5 ways Facebook improved compression at scale with Zstandard

Two years ago, Facebook open-sourced Zstandard v1.0, a landmark data compression solution that offers best-in-kind performance. Since then, we’ve released a variety of enhancements and advanc…

Meta logo
Meta
•
Felix Handte
•20 min read•advanced•
--
•View Original
CassandraMySQL

Overview

The article discusses how Facebook has enhanced the Zstandard compression algorithm since its open-sourcing, focusing on five key improvements that optimize compression performance at scale. These enhancements include multithreading, long range mode, automatic level determination, and adaptations for various use cases like large payloads, data warehouses, and databases.

What You'll Learn

1

How to implement multithreaded compression using Zstandard

2

Why long range mode improves compression efficiency for large datasets

3

How to optimize block sizes for better compression ratios in data warehouses

4

When to use dictionary compression in database systems for improved performance

Prerequisites & Requirements

  • Understanding of data compression algorithms
  • Familiarity with Zstandard library and its command-line tool(optional)

Key Questions Answered

How does multithreading improve compression speed in Zstandard?
Multithreading allows Zstandard to utilize multiple CPU cores, significantly speeding up the compression process without sacrificing the compression ratio. This is particularly beneficial in scenarios where compression can become a bottleneck, such as during backup operations.
What are the benefits of using long range mode in Zstandard?
Long range mode increases the maximum window size for compression to 2 GB, allowing for better compression ratios by capturing more redundancy in large datasets. This mode can reduce backup sizes by 16% and improve decompression speed significantly.
What improvements does Zstandard offer over traditional compression methods like zlib?
Zstandard provides better compression ratios and faster decompression speeds compared to zlib. Facebook reported improvements ranging from a 30% better compression ratio to 3x faster speeds after switching to Zstandard.
When is dictionary compression most effective in database systems?
Dictionary compression is effective when dealing with small fields scattered across a database, as it allows the compressor to start with a predefined state based on common patterns in the data, leading to better compression ratios and reduced latency.

Key Statistics & Figures

Improvement in compression ratio
30 percent better
Reported after switching from zlib to Zstandard.
Reduction in backup size using long range mode
16 percent
Achieved by applying long range mode during backup processes.
Increase in storage efficiency with Btrfs
3x more storage
Realized after enabling Zstandard compression on development servers.
Improvement in read speed for HHVM binaries
2x faster
Observed after switching to Zstandard in SquashFS.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Compression Algorithm
Zstandard
Used for data compression across various applications at Facebook.
Filesystem
Btrfs
Utilized for transparent compression in development environments.
Database
Rocksdb
Serves as the underlying storage engine for various databases at Facebook.
Filesystem
Squashfs
Provides a read-only compressed filesystem for efficient data delivery.

Key Actionable Insights

1
Implement multithreading in your compression processes to enhance performance.
By utilizing multiple cores, you can significantly reduce the time required for compression tasks, especially in high-load scenarios like backups or large file transfers.
2
Consider using long range mode for compressing large datasets to maximize efficiency.
This mode can drastically reduce file sizes and improve processing times, making it ideal for applications that handle large volumes of data.
3
Adjust block sizes in your data warehouse to optimize compression ratios.
Larger block sizes can yield better compression with Zstandard, but be mindful of memory usage and access patterns to maintain performance.
4
Utilize dictionary compression for databases to enhance query performance.
By capturing common data patterns, dictionary compression can lead to faster access times and reduced storage costs, particularly in environments with frequent read operations.

Common Pitfalls

1
Overlooking the importance of block size in compression settings can lead to suboptimal performance.
Choosing a block size that is too small may hinder compression efficiency, while a size that is too large can increase memory usage and slow down access times. It's crucial to find a balance based on your specific use case.

Related Concepts

Data Compression Techniques
Performance Optimization Strategies
Advanced Compression Algorithms