Overview
The article discusses Pinterest's transition to running a large end-to-end UI test suite before every commit in their Android and iOS repositories. This shift-left approach has significantly improved test maintenance, reduced failure resolution time, and increased the test suite's pass rate from less than 50% to over 90%.
What You'll Learn
1
How to implement pre-submit UI tests to catch issues early in the development process
2
Why establishing ownership of tests is crucial for maintaining test quality
3
How to balance test speed and cost to enhance developer productivity
4
When to silence tests and the importance of resolving them promptly
Prerequisites & Requirements
- Understanding of UI testing principles and practices
- Familiarity with testing frameworks and CI/CD tools(optional)
Key Questions Answered
How did Pinterest improve their UI test pass rate?
Pinterest improved their UI test pass rate from less than 50% to over 90% by shifting UI tests to run before commits. This proactive approach allowed for quicker identification and resolution of issues, enhancing overall test reliability and developer productivity.
What challenges did Pinterest face when implementing pre-submit UI tests?
Pinterest faced several challenges, including establishing clear ownership of tests, balancing test speed and cost, enhancing developer experience with UI tests, and insulating the main branch from failures. Each challenge required strategic solutions to maintain a stable and efficient testing environment.
What metrics are important for maintaining a healthy test suite?
Key metrics include the main branch pass rate, test speed, the number of silenced tests, and the overall number of tests. These metrics help ensure that the test suite remains effective and responsive to changes in the application.
How does Pinterest handle flaky tests in their UI testing process?
Pinterest employs a tool called Stability Enforcer to automatically silence tests that show signs of instability, such as being over 20% flaky in the last 20 runs. This helps to reduce the burden of monitoring and ensures that only reliable tests impact the development process.
Key Statistics & Figures
Test suite pass rate
>90%
Increased from less than 50% after implementing pre-submit UI tests.
Weekly builds
700 builds per week
This includes tests run on every published diff and hourly health checks.
Tests per build
300 tests per build
This is the average number of tests run for each build at Pinterest.
Technologies & Tools
Testing
Firebase Test Lab
Used for running Android tests through a sharding tool called Flank.
Testing
Bluepill
Utilized for running iOS tests on macOS instances in AWS EC2.
Testing
Flank
A sharding tool for Android tests that intelligently distributes test runs.
Testing
Pinpill
A custom scheduler built on top of bluepill to efficiently utilize available simulators.
Key Actionable Insights
1Establish clear ownership for each UI test to ensure accountability and timely resolution of failures.By assigning a single team to maintain each test, Pinterest can monitor issues more effectively and ensure that tests are updated or removed as necessary, preventing test suite deterioration.
2Implement pre-submit UI tests to catch issues before they reach the main branch, improving overall code quality.Running tests before commits allows developers to address failures immediately, reducing the time spent on investigation and increasing the reliability of the codebase.
3Utilize metrics to track the health of your test suite, focusing on pass rates and test speed.Monitoring these metrics helps identify potential issues early and ensures that the testing process remains efficient and effective, ultimately supporting developer productivity.
Common Pitfalls
1
Failing to establish clear ownership of tests can lead to unresolved issues and deteriorating test quality.
Without designated teams responsible for maintaining tests, failures may go unaddressed, causing a backlog of issues and reducing the overall effectiveness of the test suite.
2
Neglecting to monitor test metrics can result in an unstable testing environment.
If teams do not track metrics such as pass rates and test speed, they may miss critical signs of test suite deterioration, leading to increased developer frustration and reduced productivity.
Related Concepts
Shift-left Testing
UI Testing Best Practices
Test Ownership And Accountability
Test Metrics And Monitoring