Overview
The article discusses Pinterest's transition to executing end-to-end UI tests before code submissions, emphasizing the benefits of shifting testing left in the development process. It highlights improvements in test pass rates and developer productivity through better coordination and ownership of UI tests.
What You'll Learn
1
How to implement pre-submit UI tests to improve code quality
2
Why ownership of tests is crucial for maintaining test reliability
3
How to balance speed and cost in test execution
4
When to use tools like Flank and bluepill for test execution
Prerequisites & Requirements
- Understanding of UI testing concepts
- Familiarity with testing tools like Flank and bluepill(optional)
Key Questions Answered
What benefits did Pinterest see from shifting UI tests to pre-submit?
Pinterest improved its UI test pass rate from less than 50% to over 90% by executing tests before code submissions. This shift also reduced the time spent on debugging and allowed more teams to contribute to the testing platform.
How does Pinterest manage test ownership and accountability?
Pinterest assigns specific teams as owners of UI tests, ensuring they monitor alerts and decide on test updates or removals. This accountability helps maintain the quality and relevance of the test suite.
What strategies does Pinterest use to balance testing speed and costs?
Pinterest runs tests on every code change and hourly from the main branch, aiming for a reasonable execution time of around 30 minutes. They also implement strategies like parallel execution and minimizing test wait times to enhance efficiency.
What is the role of the Stability Enforcer in Pinterest's testing process?
The Stability Enforcer automatically silences tests that show signs of instability, such as failing over 20% of the time in recent runs. This helps prevent excessive errors from reaching developers and maintains a stable testing environment.
Key Statistics & Figures
UI test pass rate
>90%
This improvement was achieved after shifting UI tests to be executed before code submissions.
Weekly builds
700 builds
Pinterest runs approximately 700 builds each week, executing tests on every change.
Test execution time goal
30 minutes
The target is to keep test execution within a reasonable time frame to maintain developer productivity.
Technologies & Tools
Testing Tool
Flank
Used for executing Android tests efficiently through smart fragmentation.
Testing Tool
Bluepill
Utilized for running iOS tests in parallel on AWS EC2 instances.
Key Actionable Insights
1Establish clear ownership for each UI test to enhance accountability and reliability.By assigning specific teams to monitor and maintain tests, you can ensure timely updates and resolutions, which is crucial for maintaining a healthy test suite.
2Implement a pre-submit testing strategy to catch issues early in the development cycle.This approach not only improves the quality of code being merged but also reduces the time developers spend on debugging after changes are made.
3Utilize tools like Flank and bluepill for efficient test execution across platforms.These tools help in managing test execution effectively, especially when dealing with a large number of tests across different environments.
Common Pitfalls
1
Failing to maintain a stable test suite can lead to a high number of silenced tests.
When tests are frequently silenced, it indicates underlying issues with test quality or team accountability, which can undermine the effectiveness of the testing process.
Related Concepts
Shift-left Testing
Test Automation Best Practices
Continuous Integration And Deployment Strategies