Overview
The article discusses the development of LinkedIn's Activity Graph, a system designed to understand the relationships between member-generated content. It details the challenges faced in ensuring relevant content is displayed in user feeds, particularly regarding spam and low-quality content.
What You'll Learn
1
How to build a graph of Activities and their relationships
2
Why accurate data is crucial for machine learning models in feed systems
3
How to prevent cycles in a graph structure during data propagation
Prerequisites & Requirements
- Understanding of graph theory and data structures
- Familiarity with distributed databases like NoSQL(optional)
Key Questions Answered
How does LinkedIn ensure relevant content appears first in user feeds?
LinkedIn's Activity Graph helps manage the relationships between member-generated content, ensuring that spam and low-quality content are filtered out before being displayed. This system allows for better ranking of organic content, improving user experience by prioritizing relevant posts.
What is the decoration step in LinkedIn's feed system?
The decoration step involves resolving identifiers for Activities to ensure full data availability. If any referenced URN is marked as spam or low-quality, the top-level record is not decorated, preventing it from being displayed in the feed.
What challenges arise from spam and low-quality content in feeds?
Spam and low-quality content can lead to poor member experiences, as they might appear in prominent positions in user feeds. This necessitates accurate classification and removal of such content to maintain the integrity of the feed.
How does LinkedIn prevent cycles in the Activity Graph?
To prevent cycles, LinkedIn implements validation checks that determine if adding a new edge would create a cycle. This is done by tracking the ancestors of nodes and ensuring no ancestor is already present in the new edge being added.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Espresso
Espresso is used as LinkedIn's distributed NoSQL document database for storing nodes and edges in the Activity Graph.
Messaging
Kafka
Kafka is utilized for sending updates and feature data across the system, ensuring that all relevant nodes have the latest information.
Key Actionable Insights
1Implement a robust validation system to prevent cycles in graph structures.Cycles can lead to significant performance issues and data propagation failures. By ensuring that each new edge is validated against existing paths, you can maintain the integrity of your graph.
2Utilize materialized views to enhance data retrieval efficiency in distributed systems.Materialized views can simplify complex queries and improve performance by pre-computing paths or relationships, which is especially useful in systems where data is frequently accessed.
3Incorporate a feedback loop for machine learning models based on user interactions.Accurate data is essential for training models that predict user preferences. By ensuring that the data seen by users aligns with what models are trained on, you can improve the relevance of content displayed.
Common Pitfalls
1
Failing to account for spam and low-quality content can lead to poor user experiences.
When spam content is not filtered effectively, it can dominate user feeds, leading to dissatisfaction and reduced engagement. Implementing robust content classification systems is essential to avoid this issue.
Related Concepts
Graph Theory
Distributed Databases
Machine Learning In Content Recommendation Systems