Overview
The article discusses the implementation of test retries in Shopify's Android monorepo to combat flaky tests and improve Continuous Integration (CI) pass rates. By employing retries, the team significantly enhanced their CI pass rate from 35% to 99.95%, demonstrating the effectiveness of this approach in maintaining developer trust and system reliability.
What You'll Learn
1
How to implement test retries to improve CI pass rates
2
Why addressing flaky tests is crucial for developer trust
3
When to use retriable versus fatal failure strategies in CI
Prerequisites & Requirements
- Understanding of Continuous Integration and automated testing concepts
- Familiarity with Android development and CI/CD practices(optional)
Key Questions Answered
How did Shopify improve their CI pass rate for Android apps?
Shopify improved their CI pass rate from 35% to 99.95% by implementing test retries for flaky tests. This approach allowed them to handle failures more effectively, ensuring that tests that passed after a retry were flagged for further investigation, thus maintaining developer trust in the CI system.
What are retriable and fatal failures in CI?
Retriable failures are those that can be retried within the same job without reinitializing the test environment, while fatal failures require a complete reload of the environment. Understanding the difference helps in optimizing CI processes and reducing unnecessary delays.
What impact do flaky tests have on developer trust?
Flaky tests can lead to a lack of trust in the CI system, as developers may assume failures are false positives. This can result in decreased confidence in the testing infrastructure, making it essential to address flakiness to maintain a reliable development environment.
What strategies did Shopify use to reduce test flakiness?
Shopify implemented several strategies to reduce test flakiness, including identifying unreliable components, improving their reliability, and applying caching to minimize interactions with external services. These measures contributed to a more stable CI environment.
Key Statistics & Figures
Initial CI pass rate
31%
The pass rate in Shopify's Android app pipeline before implementing retry mechanisms.
Final CI pass rate
99.95%
The pass rate achieved after implementing test retries and addressing flakiness.
Pass rate increase
from 35% to 99.95%
The improvement in CI pass rate due to the implementation of test retries.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Infrastructure
Docker
Used for setting up test environments in CI.
Build Tool
Gradle
Used for managing dependencies and builds in the Android project.
Key Actionable Insights
1Implement test retries to enhance CI pass rates and developer confidence.Retries can significantly improve the reliability of your CI system, especially when dealing with flaky tests. By ensuring that tests are retried, you can reduce the number of false negatives and maintain developer trust.
2Identify and categorize failures as retriable or fatal to optimize CI processes.Understanding which failures can be retried without reinitializing the environment can save time and resources. This categorization allows teams to focus on fixing the most problematic areas of their CI pipeline.
3Regularly monitor and flag tests that pass after retries for further investigation.By tracking tests that succeed only after retries, teams can proactively address underlying issues, leading to a more stable and reliable testing environment.
Common Pitfalls
1
Relying on flaky tests can undermine developer trust in CI.
When tests frequently fail without clear reasons, developers may start to ignore CI results, assuming failures are false positives. This can lead to a culture of distrust in automated testing processes.
2
Neglecting to categorize failures can slow down CI processes.
Failing to distinguish between retriable and fatal failures can result in unnecessary delays and resource usage, as fatal failures require more extensive recovery actions.
Related Concepts
Continuous Integration Best Practices
Automated Testing Strategies
Flaky Test Management
CI/CD Optimization Techniques