Overview
The article discusses FollowFeed, LinkedIn's new feed infrastructure designed to enhance performance and relevance for its users. It details the challenges faced with the previous system, Sensei, and highlights the improvements achieved with FollowFeed, including reduced latency and increased data retention.
What You'll Learn
1
How to implement a feed infrastructure that supports high availability and low latency
2
Why co-locating computation with data can improve query performance
3
How to utilize Kafka for asynchronous data ingestion in distributed systems
Prerequisites & Requirements
- Understanding of distributed systems and caching concepts
- Familiarity with Kafka and RocksDB(optional)
Key Questions Answered
What improvements does FollowFeed offer compared to Sensei?
FollowFeed provides significant enhancements over Sensei, including a fivefold reduction in feed query's 99th percentile latency, a 20 times larger data retention index, and a 50% reduction in operational costs. These improvements allow LinkedIn to better handle its growing user base and content volume.
How does FollowFeed ensure high relevance in content delivery?
FollowFeed employs complex algorithms for scoring and ranking content, which are essential for delivering personalized and relevant feeds to users. This is achieved through real-time computation and the ability to process large volumes of records efficiently.
What role does Kafka play in FollowFeed's architecture?
Kafka is utilized for asynchronous data ingestion in FollowFeed, allowing for real-time updates of content records. This enables efficient handling of the feed data stream, ensuring that the index nodes can consume and process data effectively.
What caching strategies are implemented in FollowFeed?
FollowFeed uses a read/write-through caching mechanism with Guava to optimize latency. This involves maintaining a cache of deserialized content records to reduce the overhead of deserialization during query processing.
Key Statistics & Figures
p99 latency for mobile news feed
140ms
This latency is five times faster than the previous system, Sensei.
Data retention capacity
20 times larger
FollowFeed can handle significantly more data compared to Sensei.
Operational cost reduction
50%
The migration to FollowFeed resulted in halving the overall capital expenditure compared to Sensei.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Data Streaming
Kafka
Used for asynchronous data ingestion into FollowFeed's index nodes.
Database
Rocksdb
Serves as the embedded persistent key-value store for FollowFeed.
Caching
Guava
Utilized for caching deserialized content records to optimize query performance.
Key Actionable Insights
1Implementing a fan-out-on-read model can significantly enhance the efficiency of your data retrieval processes.This model allows for dynamic querying of data at read time, reducing the need for excessive data duplication and enabling faster iterations for A/B testing relevance algorithms.
2Utilizing an embedded database like RocksDB can improve performance for applications requiring low-latency access to data.RocksDB's design for fast storage solutions like SSDs makes it an excellent choice for systems like FollowFeed that need to handle large volumes of data efficiently.
3Incorporating robust monitoring and recovery tools is essential for maintaining the integrity of stateful systems.By implementing reliable checkpointing and backup strategies, you can ensure that your system can recover quickly from failures and maintain high availability.
Common Pitfalls
1
Over-reliance on pre-materialization can lead to excessive data duplication and increased storage costs.
It's essential to evaluate the trade-offs between pre-materialization and on-demand querying to maintain efficiency and reduce operational complexity.
Related Concepts
Distributed Systems
Caching Strategies
Data Ingestion Techniques
Relevance Algorithms