Overview
This article discusses how LinkedIn successfully merged two identity service systems to reduce latency and cost-to-serve. The authors detail the motivations, implementation steps, and performance improvements achieved through this integration.
What You'll Learn
1
How to merge microservices to improve performance and reduce costs
2
Why maintaining a single service architecture can simplify operations
3
How to utilize A/B testing frameworks for gradual changes in critical systems
Prerequisites & Requirements
- Understanding of microservices architecture and service-oriented design
- Familiarity with A/B testing frameworks like T-REX(optional)
Key Questions Answered
What were the main motivations for merging the identity services?
The primary motivations included addressing scalability challenges, reducing operational overhead, and simplifying the architecture. The team realized that separating the identity data service from the midtier service added unnecessary complexity and latency, prompting the merger.
What steps were taken to implement the merger of the identity services?
The implementation involved four key steps: using the data service's REST API as a local library, gradually ramping changes using A/B testing, decommissioning the data service hosts, and cleaning up the codebase to improve craftsmanship. This approach minimized disruption while ensuring performance parity.
What performance improvements were observed after merging the services?
Post-merger, the 90th percentile latency for fetching profiles dropped from 26.67ms to 24.84ms, representing a 6.9% improvement. Additionally, improvements were noted across all percentiles, with p50, p90, and p99 latencies improving by 14%, 6.9%, and 9.6%, respectively.
How did the merger impact the cost-to-serve for identity services?
The merger resulted in significant cost savings by decommissioning the data service cluster, which saved over 12,000 cores and over 13,000 GB of memory. This reduction in physical resources translated into substantial annual savings for LinkedIn.
Key Statistics & Figures
Reduction in latency
6.9%
The 90th percentile latency for fetching profiles dropped from 26.67ms to 24.84ms after the merger.
Improvement in p50 latency
14%
The merger improved the p50 latency significantly, contributing to overall performance gains.
Cores saved
12,000 cores
The decommissioning of the data service cluster resulted in substantial resource savings.
Memory saved
13,000 GB
The merger led to significant reductions in memory usage, impacting cost-to-serve positively.
Technologies & Tools
Database
Espresso
Used as the data store for identity services.
Tool
T-rex
An A/B testing framework used to manage the rollout of changes during the merger.
Key Actionable Insights
1Consider merging microservices when operational overhead and complexity outweigh the benefits of separation.This approach can lead to reduced latency and cost savings, as demonstrated by LinkedIn's identity services merger.
2Utilize A/B testing frameworks to manage high-risk changes in production systems.This allows for gradual implementation and quick rollbacks, minimizing disruption to users.
3Regularly evaluate the architecture of your services to identify potential inefficiencies.As systems evolve, what once seemed like a beneficial separation may become a liability, necessitating re-evaluation.
Common Pitfalls
1
Failing to properly evaluate the impact of merging services can lead to unforeseen complexities.
It's crucial to assess both the benefits and potential drawbacks of such architectural changes to avoid introducing new issues.
2
Underestimating the importance of maintaining performance parity during service mergers.
Without careful planning and execution, merging services can degrade performance, negatively impacting user experience.
Related Concepts
Microservices Architecture
Service-oriented Architecture
Performance Optimization Strategies
Cost Management In Software Services