Candidate Generation in a Large Scale Graph Recommendation System: People You May Know

Parag Agrawal

•

Parag Agrawal

•13 min read•advanced•

--

•View Original

EmbeddingGraph Neural NetworksNeural Networks

Overview

This article discusses the Candidate Generation (CG) stage of LinkedIn's People You May Know (PYMK) recommendation system, detailing the various techniques used to generate relevant candidate pools for users. It highlights graph-based, similarity-based, and heuristics-based methods, emphasizing the importance of these techniques in enhancing user engagement and retention.

What You'll Learn

1

How to apply graph-based techniques for candidate generation in recommendation systems

2

Why personalized PageRank is essential for enhancing recommendation relevance

3

How to implement embedding-based retrieval for candidate generation

4

When to use heuristics-based filters in generating candidate recommendations

Prerequisites & Requirements

Understanding of graph theory and recommendation systems
Familiarity with machine learning frameworks for embedding generation(optional)

Key Questions Answered

What are the main techniques used in LinkedIn's candidate generation?

LinkedIn's candidate generation employs three main techniques: graph-based methods, which utilize network proximity; similarity-based methods, which focus on profile and skill similarities; and heuristics-based methods, which apply simple rules to filter candidates. Each technique contributes to creating a diverse and relevant candidate pool for users.

How does personalized PageRank enhance candidate relevance?

Personalized PageRank (PPR) enhances candidate relevance by restricting the teleportation process to a viewer's immediate network neighborhood, allowing for deeper connections to be considered. This method ensures that even candidates several hops away can still receive high scores, making it effective for under-connected users.

What role does negative sampling play in the recommendation system?

Negative sampling is crucial for efficiently training the recommendation model by selecting a subset of negative instances that help refine the embeddings. This method avoids the computational burden of evaluating all members, allowing the model to focus on relevant candidates while improving recall.

What metrics are used to evaluate the candidate generation process?

The primary evaluation metric for the candidate generation process is Recall@k, which measures the effectiveness of retrieving relevant candidates. Additionally, entropy is used as a secondary metric to ensure diversity in the candidate pool across various categorical attributes.

Key Statistics & Figures

Number of LinkedIn members

Over one billion

This large user base makes candidate generation a complex and resource-intensive task.

Technologies & Tools

Machine Learning

Two Tower Neural Network

Used for learning embeddings of LinkedIn members to facilitate similarity-based candidate retrieval.

Key Actionable Insights

1
Implement graph-based candidate generation techniques to leverage network proximity for recommendations.
Using graph walks to identify candidates within a user's network can significantly enhance the relevance of recommendations, especially for users with fewer connections.

2
Utilize personalized PageRank to improve the quality of candidate recommendations.
By focusing on a viewer's immediate network, PPR allows for deeper connections to be considered, which is particularly beneficial for users who may not have many direct connections.

3
Incorporate embedding-based retrieval methods to enhance candidate similarity assessments.
Learning member embeddings through neural networks can help in identifying candidates with similar profiles, skills, and experiences, thus increasing the chances of successful connections.

4
Apply heuristics-based filters to refine candidate recommendations based on recent interactions.
Using simple rules to filter candidates who have interacted with the viewer can help surface relevant recommendations that might otherwise be overlooked.

Common Pitfalls

1

Relying solely on popular members as negative instances can bias the recommendation model.

This bias occurs because popular members are often over-represented, which can lead to embeddings that do not accurately reflect the needs of less active users. It's essential to balance negative sampling to include a diverse range of candidates.

Related Concepts

Graph Theory In Recommendation Systems

Machine Learning For Embedding Generation

Diversity In Candidate Selection