Using experiments to tackle a large refactor with confidence
Overview
The article discusses Airbnb's transition of its search page from the Flight framework to React, highlighting the challenges faced during the refactor and the use of the Experiment Reporting Framework (ERF) to validate changes. It emphasizes the importance of minimizing regressions to protect core business metrics during significant code changes.
What You'll Learn
1
How to use the Experiment Reporting Framework to validate code changes
2
Why minimizing regressions is crucial during a major refactor
3
How to implement split-testing between old and new code using ERF
Prerequisites & Requirements
- Understanding of React and component-based architecture
- Familiarity with Experiment Reporting Framework (ERF)(optional)
Key Questions Answered
How did Airbnb ensure the refactored search page did not regress in performance?
Airbnb utilized the Experiment Reporting Framework (ERF) to conduct split-tests between the original and refactored React code. This allowed them to monitor key metrics and identify regressions, ensuring that any negative impacts on user engagement were quickly addressed.
What challenges did Airbnb face when refactoring the search page to React?
The Airbnb engineering team faced challenges due to the complexity of the search page, which included various external experiments and customizations. Additionally, the original code lacked comprehensive tests, making it difficult to ensure that the refactor would not introduce regressions.
What role did the Experiment Reporting Framework (ERF) play in the refactoring process?
ERF was crucial for simplifying experiment setup, data analysis, and results visualization. It allowed the team to analyze the impact of the refactored code on key business metrics and segment results by various dimensions, helping to identify regressions effectively.
What specific regressions were identified during the refactoring of the listing card component?
The team discovered that the refactored React listing card did not open the listing page in a new tab on mobile devices, leading to a drop in views. This regression was identified through ERF, allowing the team to implement a fix and improve user engagement metrics.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Frontend
React
Used as the canonical front-end view framework for the refactored search page.
Tools
Experiment Reporting Framework (erf)
An in-house tool for experiment setup, data analysis, and results visualization.
Key Actionable Insights
1Implement a robust testing framework before undertaking major refactors to minimize risks.Having comprehensive test coverage can significantly reduce the chances of regressions during code changes, ensuring smoother transitions and maintaining user engagement.
2Utilize split-testing to validate changes in real-time and gather actionable data.By comparing the performance of the original and refactored code, teams can make informed decisions about the impact of their changes on user behavior and business metrics.
3Segment performance metrics by user demographics and device types to identify specific issues.This approach allows teams to pinpoint regressions that may affect certain user groups, enabling targeted fixes that enhance overall user experience.
Common Pitfalls
1
Neglecting to monitor key metrics during a refactor can lead to undetected regressions.
Without proper monitoring tools like ERF, teams may miss critical performance drops that could negatively impact user engagement and business outcomes.
2
Overlooking the complexity of existing components can result in incomplete feature implementations.
It's essential to fully understand the behavior of existing components before refactoring to ensure that all functionalities are preserved in the new implementation.
Related Concepts
Component-based Architecture
A/B Testing
User Engagement Metrics