Overview
The article discusses how Uber Engineering evaluated various JSON encoding and compression algorithms to optimize trip data storage. It highlights the challenges of managing large volumes of trip data and presents findings on the most effective methods for reducing data size while maintaining performance.
What You'll Learn
1
How to evaluate different JSON encoding protocols for performance
2
Why compression algorithms are essential for managing large datasets
3
When to use IDL-based versus non-IDL-based encoding protocols
Prerequisites & Requirements
- Understanding of JSON and data serialization concepts
- Familiarity with compression libraries like zlib and Bzip2(optional)
Key Questions Answered
What are the best encoding protocols for JSON data?
The article evaluates three IDL-based encoding protocols (Thrift, Protocol Buffers, Avro) and seven non-IDL-based protocols. It concludes that Protocol Buffers with zlib compression offers the best balance of size and speed for JSON data.
How much storage space can be saved by compressing JSON data?
By using MessagePack with zlib, the size of trip data can be reduced from 20 KB to approximately 2,822 bytes per trip. This results in a significant storage savings, allowing a 1 TB disk to last almost a year compared to just 30 days without compression.
What compression algorithms were tested for JSON data?
The article tested three lossless compression libraries: Snappy, zlib, and Bzip2. Each was evaluated for its speed and compression efficiency when applied to JSON data.
What were the performance results of the encoding and compression tests?
The tests revealed that Thrift with Bzip2 provided the best compression ratio, while Thrift with zlib was the fastest in terms of encoding speed. Protocol Buffers with zlib also performed well, compressing data to just above 8% of its original size.
Key Statistics & Figures
Data size reduction
86%
This reduction is achieved by using MessagePack with zlib compared to raw JSON.
Storage duration on 1 TB disk
347 days
This is the expected lifespan with MessagePack/zlib compared to just 30 days without compression.
Encoding speed of Thrift
1548 ms
This is significantly faster than the native JSON encoder, which takes 6535 ms.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Data Serialization
Messagepack
Used for encoding trip data to reduce size.
Compression
Zlib
Used to compress JSON data for storage efficiency.
Data Serialization
Thrift
One of the IDL-based protocols evaluated for encoding efficiency.
Data Serialization
Protocol Buffers
Another IDL-based protocol tested for performance.
Data Serialization
Avro
An IDL-based protocol included in the evaluation.
Key Actionable Insights
1Consider using MessagePack with zlib for JSON data storage to achieve significant space savings.This combination reduces the size of trip data from 20 KB to about 2,822 bytes, allowing for much longer storage durations on existing hardware.
2Evaluate both IDL-based and non-IDL-based encoding protocols based on your data requirements.IDL-based protocols provide better validation and structure but may require more overhead in schema management, while non-IDL-based protocols offer flexibility.
3Benchmark different encoding and compression combinations to find the best fit for your application.The article emphasizes the importance of testing various protocols on real data to determine the most efficient solution for your specific use case.
Common Pitfalls
1
Relying solely on JSON for data storage can lead to rapid storage exhaustion.
As demonstrated, raw JSON can consume significant disk space quickly, especially with high data influx, making it essential to explore compression options.
2
Not benchmarking different encoding protocols may lead to suboptimal performance.
The article highlights the importance of testing various combinations to find the most efficient solution tailored to specific data needs.
Related Concepts
Data Serialization Techniques
Compression Algorithms
Performance Optimization In Data Storage