Overview
LinkedIn has integrated Google Protocol Buffers (Protobuf) with Rest.li to enhance microservices performance, achieving significant reductions in latency and improvements in resource utilization. The article discusses the challenges faced with JSON serialization and the successful implementation of Protobuf as a more efficient alternative.
What You'll Learn
1
How to evaluate serialization formats for microservices
2
Why Google Protocol Buffers can improve performance over JSON
3
How to implement symbol tables for schema management in Rest.li
Prerequisites & Requirements
- Understanding of microservices architecture and RESTful APIs
- Familiarity with Google Protocol Buffers and Rest.li(optional)
Key Questions Answered
What challenges did LinkedIn face with JSON serialization?
LinkedIn encountered performance bottlenecks with JSON due to its verbose nature, leading to increased network bandwidth usage and higher latencies. Serialization and deserialization latencies were suboptimal, particularly in garbage collected languages like Java and Python, prompting the search for a more efficient alternative.
How did LinkedIn integrate Protobuf into Rest.li?
The integration involved creating symbol tables to map PDL field names to Protobuf field numbers, allowing Rest.li to continue using its existing data modeling while leveraging Protobuf's efficiency. This was achieved through runtime generation of symbol tables and a controlled rollout to minimize disruption.
What performance improvements were observed after using Protobuf?
The integration of Protobuf resulted in up to a 60% reduction in latency for services with large payloads and an average throughput increase of 6.25% for response payloads and 1.77% for request payloads across all services, with no significant degradations compared to JSON.
Key Statistics & Figures
Reduction in latency
up to 60%
Observed in services with large payloads after integrating Protobuf.
Average throughput increase for response payloads
6.25%
Measured across all services after the rollout of Protobuf.
Average throughput increase for request payloads
1.77%
Measured across all services after the rollout of Protobuf.
Technologies & Tools
Serialization
Google Protocol Buffers
Used to replace JSON for improved performance in Rest.li.
Framework
Rest.li
An open source REST framework used for microservice development at LinkedIn.
Key Actionable Insights
1Evaluate serialization formats regularly to ensure optimal performance in microservices.As seen with LinkedIn's transition from JSON to Protobuf, regular evaluations can uncover performance bottlenecks and lead to significant improvements in latency and resource utilization.
2Implement symbol tables for managing schema changes in microservices.Using symbol tables allows for a seamless integration of new serialization formats without disrupting existing services, as demonstrated in LinkedIn's approach to integrating Protobuf.
3Consider the programming language support when selecting a serialization format.LinkedIn prioritized wide programming language support in their evaluation, ensuring that the chosen format would work across their diverse tech stack.
Common Pitfalls
1
Relying solely on JSON without considering performance impacts.
JSON's verbosity can lead to increased latencies and resource consumption, making it essential to evaluate alternatives like Protobuf for performance-critical applications.
Related Concepts
Microservices Architecture
Restful Apis
Serialization Formats
Performance Optimization Techniques