Overview
This article discusses LinkedIn's approach to A/B testing, specifically focusing on the efficient assignment of variants at scale. It outlines the importance of a robust randomization algorithm and compares different methods for variant assignment, highlighting the hash-based approach as a cornerstone of their experimentation system.
What You'll Learn
1
How to implement a hash-based variant assignment for A/B testing
2
Why independence of variant assignment is crucial for accurate A/B testing results
3
How to identify and troubleshoot sample size ratio mismatches in experiments
Prerequisites & Requirements
- Understanding of A/B testing principles and randomization algorithms
- Experience with data analysis and statistical methods(optional)
Key Questions Answered
What are the main characteristics of a good variant assignment function?
A good variant assignment function should ensure that variants are assigned according to the desired split, that assignments are deterministic for each member, and that there is no correlation between assignments across multiple experiments. This ensures valid and reliable A/B testing results.
How does LinkedIn handle variant assignment at scale?
LinkedIn employs a hash-based variant assignment method, which allows for deterministic assignments and reduces reliance on external data stores. This method significantly improves performance and consistency, enabling the handling of trillions of evaluations per day.
What are the differences between RNG and hash-based variant assignment methods?
The RNG method requires frequent calls to a remote data store, which can lead to bottlenecks and inconsistencies. In contrast, the hash-based method allows for local evaluations, significantly reducing the need for network requests and improving speed and reliability.
What is sample size ratio mismatch (SSRM) and how is it detected?
Sample size ratio mismatch occurs when the observed sample sizes of variants do not match the expected ratios. LinkedIn uses the Chi-squared goodness-of-fit test to monitor for SSRM, hiding reports when detected to prevent misleading conclusions.
Key Statistics & Figures
Trillions of evaluations handled per day
35 trillion
This scale demonstrates the efficiency of LinkedIn's variant assignment methods.
Assignment storage write QPS for RNG
1,770,000 - 25,000,000
This highlights the high demand on storage systems when using RNG methods.
Cache read Network QPS for hash-based assignment
Up to 1,000,000
This shows the reduced load on network resources compared to RNG methods.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Kafka
Used for transferring variant assignment data to Hadoop for batch report generation.
Backend
Hadoop
Used for processing and storing large volumes of variant assignment data.
Key Actionable Insights
1Implementing a hash-based variant assignment can significantly improve the efficiency of A/B testing frameworks.This method reduces the dependency on external data stores, allowing for faster evaluations and better scalability, especially in high-traffic environments like LinkedIn.
2Regularly monitor for sample size ratio mismatches to maintain the integrity of A/B testing results.Using statistical tests like the Chi-squared test can help identify issues early, ensuring that experiments yield reliable data for decision-making.
3Ensure that variant assignments are independent across multiple experiments to avoid confounding results.This independence allows for accurate analysis of each experiment, which is crucial for understanding the impact of different variants.
Common Pitfalls
1
Sample size ratio mismatches can lead to unreliable A/B test results.
This often occurs due to dynamic targeting or biased code implementation. It's crucial to diagnose the root cause and apply fixes to ensure valid experimental outcomes.
Related Concepts
A/B Testing Principles
Randomization Algorithms
Statistical Methods For Data Analysis