Overview
The article discusses the challenges and solutions Netflix faced when testing its application on the Android platform. Key points include the importance of automation, device diversity, and maintaining a small, efficient team to ensure rapid releases and high-quality streaming experiences.
What You'll Learn
1
How to automate testing for an Android application using a modified test framework
2
Why device diversity is crucial for application testing on Android
3
How to leverage crowd-sourcing for testing network performance during international launches
Prerequisites & Requirements
- Understanding of Android application architecture and testing frameworks
- Familiarity with automation tools like Selenium and Android Instrumentation Framework(optional)
Key Questions Answered
What challenges did Netflix face when entering the Android ecosystem?
Netflix encountered challenges such as the need for rapid releases every 6-8 weeks, the diversity of hundreds of Android devices, and the goal of maintaining a small, happy team. These factors complicated the testing and quality assurance processes.
How does Netflix automate testing for its Android application?
Netflix developed a modified test framework that uses a bridge between JavaScript and native code to automate key scenarios like login and playback. This approach allows for effective testing despite the complexities of using an HTML5 interface.
What criteria does Netflix use to select devices for testing?
Netflix categorizes devices based on playback pipeline architecture, processor capabilities, operating system versions, and user popularity among Netflix subscribers. This ensures comprehensive coverage in testing.
Why is subjective quality analysis still necessary despite automation?
While automation can catch issues like buffering, subjective analysis by testers is essential for verifying aspects like seamless resolution switching and HD quality, which are difficult to automate effectively.
Key Statistics & Figures
Number of different devices streaming Netflix on Android daily
around 1000
This statistic highlights the diversity of devices that Netflix must accommodate in its testing process.
Frequency of releases
every 6-8 weeks
This rapid release cycle emphasizes the need for efficient testing and automation strategies.
Number of devices used for daily smoke automation testing
around 10 phones and 4 tablets
This streamlined approach allows Netflix to efficiently test across a representative sample of devices.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Frontend
Javascript
Used for automating interactions within the HTML5 interface of the Netflix application.
Testing
Android Instrumentation Framework
Considered for automating testing scenarios on Android devices.
Key Actionable Insights
1Implement a modified test framework that bridges JavaScript and native code to enhance automation in testing Android applications.This approach allows for effective testing of complex user interactions in applications that utilize HTML5 interfaces, ultimately improving release cycles.
2Regularly update your device testing list to reflect changing market conditions and user preferences.By maintaining an up-to-date list of devices, you can ensure that your application performs optimally across the most popular devices used by your audience.
3Utilize crowd-sourcing platforms for testing during international launches to gain real-world performance insights.This strategy provides additional assurance that backend systems function correctly under various network conditions, enhancing overall application reliability.
Common Pitfalls
1
Relying solely on automation for quality assurance can lead to missed subjective quality issues.
Automation is effective for catching many technical issues, but it cannot replace the nuanced understanding that human testers provide, especially for subjective quality aspects like video resolution and playback smoothness.
Related Concepts
Automation In Software Testing
Device Diversity In Application Development
Crowd-sourcing For Performance Testing