Overview
This article discusses the evolution of LinkedIn's Endorsements infrastructure, focusing on the integration of GraphDB to enhance the relevance of suggested endorsements. It outlines the challenges faced during the migration and the solutions implemented to ensure real-time updates and performance.
What You'll Learn
1
How to integrate GraphDB with existing endorsement systems
2
Why maintaining a SQL database as a source of truth is crucial for real-time updates
3
How to manage high write loads in endorsement systems using messaging queues
Prerequisites & Requirements
- Understanding of graph databases and their applications
- Familiarity with Hadoop and Kafka(optional)
Key Questions Answered
How does LinkedIn ensure real-time updates for endorsements?
LinkedIn maintains a SQL database as the source of truth for endorsements, allowing for immediate updates. This is combined with GraphDB to serve endorsements, ensuring that users see the most current data despite the eventual consistency of the graph.
What challenges arise from using GraphDB for endorsements?
Migrating to GraphDB introduced challenges such as eventual consistency, where updates are not immediately visible. LinkedIn addresses this by continuing to query the SQL database for real-time data while gradually loading the endorsements dataset into GraphDB.
What is the process for bootstrapping the endorsements data?
GraphDB re-bootstraps itself weekly by running Hadoop jobs that fetch and filter raw endorsements data stored on HDFS. This ensures that the graph remains up-to-date and reflects all endorsements accurately.
How does LinkedIn handle high write loads for endorsements?
LinkedIn uses a messaging queue to manage high write loads, where updates are emitted as events to a Kafka cluster. This allows for controlled processing of endorsement updates without overwhelming the system.
Key Statistics & Figures
Number of updates to the endorsements database
More than a hundred updates per second
This high frequency necessitates efficient handling mechanisms to ensure that the system remains responsive.
Size of the final dataset added to the graph
On the order of 400GB
This reflects the scale of data processing involved in managing endorsements on LinkedIn.
Number of nodes added to the endorsements dataset
200M nodes every two weeks
This controlled loading strategy helps manage system performance during the migration to GraphDB.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Graphdb
Used for serving endorsements and managing relationships between users and their skills.
Data Processing
Hadoop
Used to run jobs that fetch and process endorsements data for loading into GraphDB.
Messaging
Kafka
Used to handle updates to endorsements in a controlled manner through a messaging queue.
Stream Processing
Samza
Used to process events emitted to Kafka and update the GraphDB accordingly.
Key Actionable Insights
1Implement a messaging queue to handle high-frequency updates in your systems.Using a messaging queue like Kafka can help manage the load of frequent updates, ensuring that your system remains responsive and efficient during peak times.
2Consider maintaining a dual-database approach for real-time data access.By using a SQL database as a source of truth alongside a graph database, you can achieve both immediate data visibility and the advanced querying capabilities of graph databases.
3Regularly re-bootstrap your data to ensure consistency and accuracy.Implementing a routine for data re-bootstrapping helps maintain the integrity of your graph database, especially in systems with high update rates.
Common Pitfalls
1
Failing to manage eventual consistency in distributed systems can lead to outdated information being displayed to users.
This can be avoided by maintaining a reliable source of truth, such as a SQL database, to ensure that users always see the most current data.
2
Overloading the graph database with too much data at once can lead to performance issues.
Implementing a controlled loading strategy, such as adding nodes incrementally, can help mitigate this risk.
Related Concepts
Graph Databases
Endorsements Systems
Real-time Data Processing
Distributed Systems Architecture