Downloaded 86 times
Uploaded byGeoffrey Goetz
Preparing your QA team for mobile testing
The document discusses the challenges and strategies for preparing a QA team for mobile testing, emphasizing the importance of understanding mobile-specific non-functional requirements and the complexities introduced by diverse devices, OS versions, and user behaviors. It highlights the fragmentation in the Android ecosystem and the need for thorough testing scenarios that consider multitasking interruptions, performance under various conditions, and the interaction between applications and device hardware. Additionally, it mentions the necessity of effective communication and version control to address issues that arise during testing.
Preparing your QA team for mobile testing
- 1. Preparing Your QATeam for Mobile Testing Where Reality and New Beginnings Meet
- 2. @ggeoffreggeoffre.com 20 years ofexperience in the development, marketing, sales, and leadership of computer software A C T S
- 3. Where did theterm "bug" come from? 66 years ago on September 9th 1947, operators of the Mark II Aiken Relay Computer being tested at Harvard University, found something curious trapped between points at Relay #70, Panel F.
- 4.
- 5.
- 6.
- 7.
- 8. Bugs not Defects! Outsidethe scope of most acceptance or unit testing Tend to design around them not to them Defensive programming techniques employed More likely hardware based than software based Environment plays a major role discovering them Magnified greatly with mobile based development
- 9.
- 10. Three Eras ofComputing mainframe personal ubiquitous
- 11. Three Human toComputer Ratios many one one one one many : : :
- 12. Leaving Other ErasBehind Single, closed computing systems Three "major" user upgradable operating systems Maybe five “major” browsers Virtualization possible to contain testing options
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19. Mobile Testing Challenge DeviceManufactures and Device Models Carrier Networks and Network Speeds OS Versions and API Updates Different Features for Different Phones
- 20. Android Fragmentation 11,868 DistinctAndroid devices seen this year 3,997 Distinct Android devices seen last year 682,000 Devices surveyed for this report 47.5% Samsung's share of those devices 8 Android versions still in use 37.9% Android users on Jelly Bean http://opensignal.com/reports/fragmentation-2013/
- 21. Android Fragmentation 18,796 DistinctAndroid devices seen this year 11,868 Distinct Android devices seen last year 682,000 Devices surveyed for this report 43% Samsung's share of those devices 7 Android versions still in use 46% Android users on Jelly Bean http://opensignal.com/reports/2014/android-fragmentation/
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 36. iOS Omneity 12* DistinctiOS devices seen this year 8 Distinct iOS devices seen last year 10M Events/hr surveyed for this report 100% Apple's share of those devices 3 iOS versions still in use 80.7% iOS users on iOS 7 (97.1% iOS 6+7) http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
- 37. iOS Omneity 17* DistinctiOS devices seen this year 12 Distinct iOS devices seen last year 10M Events/hr surveyed for this report 100% Apple's share of those devices 3 iOS versions still in use 80.7% iOS users on iOS 7 (97.1% iOS 6+7) http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 47. 700 Million iOSDevices
- 48. 800 Million iOSDevices
- 49. 900 Million AndroidDevices
- 50.
- 51.
- 52.
- 53. Planning for theExperience
- 54. NFRs of Mobile Immutablerequirements set before the project even begins Dictated to you by many different outside organizations Typically misunderstood during the design phase Often ignored by functional based acceptance testing Not typically caught by code quality unit testing Highly improbable if not impossible to avoid absolutely Things that every mobile user deals with every single day
- 55. Examples of MobileNFRs Wireless Network Access, Latency and Speed Screen Size, Aspect Ratio, Resolution and Orientation Multi-tasking Interruptions, Time Limits and Unexpected Shutdowns User Specified Configurations and Privacy Settings Processor Performance and Availability of On-Device Storage
- 56. Examples of MobileNFRs GPS Speed, Accuracy and Availability Photo Resolution, Focal Length, Lowlight Ability Platform Versions, Releases and Updates Code Signing, Packaging and Distribution Methods
- 58. Network Reachability Degraded performancedepending on network latency Slows down performance Some areas are not connected at all Stops functionality cold Bandwidth utilization and data plans Frustrates users and impacts runtime costs Users are actually Mobile and moving around! Don’t always access the network from the same WiFi
- 60.
- 62. Screen Size andOrientation Layout of Information Can all of the information be viewed Scrolling To Access Information Usability of hidden navigational components Logic Triggered by Switching Refresh screen data with each rotation Deployment to Tablet Devices
- 64. Multitasking Interruptions Interruption fromother Apps Incoming Phone Calls Tweets, Posts, Email etc… Background Operations Downloads, Tracking, Music External Connectivity Bluetooth, WiFi Screen Sharing
- 66. Battery Life Using lotsof device features GPS, Network, Video, etc... User disabling device features to prolong battery life Affects use cases and therefore should affect testing scenarios User deletes Apps to find out which one is causing problems
- 68. Accessing Online APIs Keepin mind that users do not have to update Loosely coupled APIs with optional fields helps with flexibility Layers of services may simplify end user support, but complicates operations (Seperate API Layer for each User Group or Platform) Large APIs accounting for all needs get updated more frequently than specialized smaller APIs for a single Apps needs.
- 70. Rewrite Master Testing Strategy Connectedand disconnected states Internet facing test environments Multiple sized devices for each platform version Multitasking scenarios per platform Track battery life in different use case scenarios Plan for live field testing to gauge responsiveness Review instrumentation results for memory use
- 71.
- 72. Primary Test Cases Rotatethe device (on every single screen) Receive a phone call (at the worst possible times) Switch to/from another app (at the worst possible times) Resolution Specific Content and Graphics Disconnect from the network entirely (airplane mode) 1
- 73. Edge Test Cases Switchbetween wi-fi and wireless networks (device specific) Battery dies (device shuts off instantly) Apply an update (with data change) 2
- 74. Special Test Cases Backgroundprocessing of developed app App to app communications Specific interaction with device hardware Notification or event based triggering Specific interaction with platform APIs 3
- 75. Frustration comes from… Testercan no longer reproduce the error Tester can reproduce on physical device, but developer cannot in the simulator Developer requires physical access to testers device in order to reproduce and diagnose the error API environment is down or inaccessible App needs to be removed and re-installed Device needs to be reset (power, network, settings, full)
- 76. Learning How MobileWorks Shutdown Apps for a Fresh Start Hard Reset to Recover from Catastrophic Failures Back-up and Restore the Device to Factory Settings Delete and Re-Install Apps How to Create Multi-Tasking Interruptions Understanding of the Platform’s Notification System Privacy and Accessibility Settings
- 77. Some things thatcan help Version number of build, traceable back to the snapshot of the code branch, located within the deployed app on the test device Screen shots and even video of error occurring on the test device Simplistic API Test Interface to verify if it is the backend or the app Adequate supply of physical test devices that can be checked in/out by all team members
- 78. Some things thatcan help Add validation criterial to every story card Make version control and change control information of APIs available to all team members Create an API up/down status page
- 79. Important Communications Report theDevice Type - Hardware and OS Versions Capture the complete software version information at time of failure App version traceable back to code branch Fresh install or update from prior version? API Environment and Version Perform a Standardized Smoke Test First Steps to Reproduce Reproducible on second like device Reproducible on second un-like device Reproducible within Simulator/Emulator
- 80. Agile Considerations introduction ofnon-functional story cards addition of non-functional tests to story cards SCRUM Considerations co-location and field testing is not practical together everyone plays every role (anyone can play any role) personal devices may not be fair game test devices
- 81. What about betatesters? They are dynamic and certainly fall into the system based compatibility testing space, but… are more often than not selective function based testers with their own agenda non-dedicated resources that have higher priorities in life outside of testing your app in a controlling position as some of your most valuable customers
- 82. Manual is tooexpensive… On the day of the launch, purchase the app from the store and… ensure that all of the last minute content changes were applied correctly access the system and log on to a real account download the updated version from the store to see if all of your previously stored data is still accessible