Overview
The article discusses Uber's strategy for shifting end-to-end (E2E) testing left in their development process to improve efficiency and reduce operational costs. It highlights the challenges faced with traditional testing methods and introduces the Backend Integration Testing Strategy (BITS) as a solution to enhance testing reliability and speed.
What You'll Learn
1
How to implement the Backend Integration Testing Strategy (BITS) for E2E testing
2
Why isolating production and testing traffic is crucial for reliable testing
3
How to manage test suites effectively using UI tools
4
When to apply automated quarantining of tests to maintain CI/CD flow
Prerequisites & Requirements
- Understanding of microservices architecture
- Familiarity with testing frameworks and CI/CD tools(optional)
Key Questions Answered
What is the Backend Integration Testing Strategy (BITS) at Uber?
BITS is a company-wide initiative that allows developers to test code changes in isolated environments without deploying to production. This strategy enables on-demand deployment and routing to test sandboxes, allowing individual commits to be tested in parallel, which helps maintain the integrity of the production environment.
How does Uber ensure data isolation during testing?
Uber ensures data isolation by tagging requests with tenancy identifiers and routing testing traffic to logically separated datastores. This method prevents testing traffic from affecting production data and allows for safe testing of changes without interference.
What challenges does Uber face with E2E testing?
Uber faces challenges such as maintenance of test states, reliability and speed of tests, and ensuring debuggability and actionability of test results. These challenges arise from the complexity of their microservices architecture and the need for effective communication across teams.
How does Uber manage test failures in their CI/CD pipeline?
Uber automatically marks tests as non-blocking if their pass rate drops below 90% during placebo runs. This allows the CI/CD pipeline to continue functioning while alerts are sent to the owning team for investigation, ensuring that critical tests remain prioritized.
Key Statistics & Figures
E2E test average pass rate
90%+
This pass rate is achieved per attempt across several thousand E2E tests conducted at Uber.
Reduction in incidents per 1,000 diffs
71%
This reduction was achieved in 2023 as a result of improved testing strategies.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Kafka
Used for messaging and communication between services in the BITS architecture.
Workflow Engine
Cadence
Utilized for orchestrating workflows in the BITS architecture.
Tracing
Jaeger
Employed for capturing and indexing test execution traces.
Key Actionable Insights
1Implementing BITS can significantly enhance testing efficiency by allowing parallel testing of code changes.This approach minimizes the risk of introducing bugs into production and ensures that developers can validate their changes without impacting others.
2Utilizing data isolation techniques is essential for maintaining the integrity of production data during testing.By tagging requests and routing traffic appropriately, teams can safely test new features without the risk of affecting live user data.
3Automated test management tools can streamline the process of tracking test health and coverage.Real-time management of test suites allows developers to quickly identify issues and maintain high-quality standards in their codebase.
Common Pitfalls
1
Over-reliance on E2E tests can lead to maintenance challenges and slow down development.
Many teams may skip E2E testing due to its complexity, which can result in undetected issues in production. It's crucial to find a balance between E2E and other testing methods to ensure comprehensive coverage.
Related Concepts
Microservices Architecture
Testing Strategies
Continuous Integration/Continuous Deployment (ci/Cd)
Backend Integration Testing Strategy (bits)