A couple quick wins and a bigger bet to speed up our CI test durations
Overview
The article discusses strategies to reduce the duration of Python test suites, which were previously taking around 20 minutes to run. It highlights three main approaches: increasing concurrency, eliminating redundant initialization, and prioritizing tests based on recent changes.
What You'll Learn
1
How to increase concurrency in Python test suites to reduce run times
2
Why skipping redundant initialization can speed up local test runs
3
How to implement an incremental test runner that only executes affected tests
Prerequisites & Requirements
- Understanding of Python testing frameworks, specifically pytest
- Familiarity with AWS CodeBuild and Docker(optional)
Key Questions Answered
How can increasing concurrency improve Python test performance?
Increasing concurrency allows more tests to run in parallel, which can significantly reduce overall test suite duration. The article notes that by restructuring database setup code, the team doubled the machine size, effectively halving CI test times.
What strategies can be used to skip redundant initialization in tests?
The article suggests retaining the database state between test runs instead of recreating it from scratch. This approach can save 10-30 seconds per test run, depending on the environment, by applying only pending migrations when possible.
What is the benefit of prioritizing tests based on recent changes?
Prioritizing tests that are relevant to recent code changes can drastically reduce feedback time. The article describes an incremental pytest runner that executes only 30-40% of the test suite, providing actionable information in minutes rather than the full test duration.
Key Statistics & Figures
Initial CI test duration
20 minutes
The starting point for test duration before optimizations were implemented.
Reduction in CI test times after optimizations
Halved
Achieved by doubling the machine size and improving concurrency.
Percentage of test suite executed by incremental runner
30-40%
This is the average amount of tests run by the incremental pytest runner, significantly speeding up feedback.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Testing Framework
Pytest
Used for running tests and implementing the incremental test runner.
CI/CD
AWS Codebuild
Platform used for running the test suite in a Docker environment.
Containerization
Docker
Used to host the application and its dependencies during testing.
Database
Postgresql
Database used for the application's data storage and testing.
Key Actionable Insights
1Implementing higher concurrency in your test suite can lead to significant time savings.By coordinating database schema changes among test workers, you can avoid redundant work and memory issues, which allows for more parallel test execution.
2Consider skipping redundant initialization tasks to streamline local test runs.By maintaining the database state and applying only necessary migrations, developers can reduce the setup time for tests, enhancing productivity.
3Adopting an incremental test runner can provide faster feedback for code changes.This approach allows developers to focus on relevant tests, reducing the time spent waiting for results and enabling quicker iterations.
Common Pitfalls
1
Failing to manage database state can lead to increased test times.
Many teams recreate the database for each test run, which can be inefficient. By maintaining the database state, teams can save significant time.
2
Overlooking the importance of test prioritization can slow down development.
Running all tests regardless of changes can lead to wasted time. Implementing a system to run only relevant tests can enhance productivity.
Related Concepts
Continuous Integration
Test-driven Development
Performance Optimization Techniques