Whitepaper: Mobile Testing Best Practices

Download Whitepaper: Mobile Testing Best Practices

Introduction

In this white paper we will discuss some best practices to be applied when developing and performing Quality Assurance of Mobile Applications. Belatrix Software Factory is a leading Software Services Outsourcing company with its main offices and delivery centers based in Mendoza, Argentina and Lima, Peru. We provide Software Development, Quality Assurance, Localization and Graphics Design Services using the "Agile Near-Shore Model".

Mobile Quality Assurance Services

While many of the same practices of Software Quality Assurance apply to performing Quality Assurance on mobile applications, the reality is that mobile applications are quite different in some critical ways, which requires some special considerations.

How and Where to Test.

Having exactly each device model that is supposed to be compatible with the application is the only way you can give 100% assurance that the application will behave as expected. So for example in the case of iPhone as you cannot go back in versions you have to have the same hardware model multiplied by the number of software versions you want to support. This can get very expensive and complex very quickly.

Unfortunately, while helpful for initial testing, none of the alternatives such as emulators, simulators, and so on is 100% reliable so far, not even the emulators / simulators that come with the official SDKs. While most companies would probably want to target all devices out there, the reality is that because of the actual cost of sourcing, managing and maintaining all the physical devices the only realistic approach is to select a list of devices that your company will ensure compatibility combined with a well-managed user Beta testing program

The only cost-effective approach to cover as many devices as possible other than the ones the company wants to support by default is that the company engages groups of Beta testers that test the application on their devices and report any issues or incompatibilities. This will result in a compatibility database that can be used to adapt or fix the application.

Another important aspect is that companies should, if at all possible, avoid supporting devices or operating systems that the respective vendors and manufacturers no longer support. Trying to test on unsupported devices and O.S. versions increases costs exponentially.

Some good mobile testing practices.

• Having access to all physical devices that the application will be supported on. (Not just emulators or simulators).
• Taking into consideration the additional time and/or personnel needed to test all the application's modules on all the devices.
• Having an explicit list of devices on which quality assurance will be performed. Basically this means that the company is committing to support certain devices and Operating Systems on which the application will perform as designed, the users will be informed that even though the application might work on other devices that the company is not responsible for any unexpected issues. This is the approach taken by most leading vendors nowadays, including Skype for example.

Besides all the above, we must measure the quality of the applications developed regardless of the platform on which they will be executed. For this we may follow some or all of the following strategies:

Interface Testing

• Validation of buttons, text inputs ...
• Validate each screen in a comprehensive manner (does it do what it is supposed to do?)
• Validate navigation flow (e.i. "from the list of restaurants screen, Can I access bookmarks?)
• What happens when you change the orientation of the device?What if it is too fast?

Usability

• Is it easy to navigate between screens or does it require weird steps?
• Does it show redundant information or information of low value to the user?
• Can it display text properly in the language selected?
• ALL (repeat, ALL) user interactions with the system should generate some kind of feedback!
• Clicks on buttons
• Calls over the Internet
• Any potentially slow operation: always assume the worst case!
• Verification of the functionality OFFLINE / ONLINE. Does the user lose information that is sent to a server while in a non-coverage area?

Performance

• What if the connection to a server is done over 3G instead of WiFi? What about on 2G?
• What about whether to send / receive much information from a Web service?
• Are the images we use the "right" size?
• Optimize: redundant code => CPU cycles => + battery consumption => unhappy users
• Does the App comply with the response times required by users?
• Are there memory leaks?
• Are all resources being freed (GPS, camera)?

Security

• Any sensitive data being sent or received must be encoded: Encryption + Authentication (HMAC-SHA)
• Detection of areas in the app that can receive malicious information. Corollary: the server is not necessarily our friend.
• Do not assume good behavior: the world is evil, cruel and vile!
• Any file you saved in the application can reach the hands of unintended third parties
• Multiuser support without interfering with the data between them

Services

• The Mobile Apps should act as clients, not servers.
• We must validate what happens when a service goes down, if it returns malformed responses, control errors ...
• What if the service takes too long to respond?
• What happens if you try to access a service offline? What if I get disconnected during transmission of info?

Low-level resources

• Is the App generating "garbage" of some sort? Temporary files not cleaned, local database growing too much...
• Are we correctly using the sensors? Are we freeing the GPS? Do we make more calls to the server than needed?
• Do we use too much memory or are we not releasing it properly?

Operational

• Are we "backing up" necessary information in the app?
• Are we prepared for "shock" mode? If the battery dies, we need a Recovery and Save Plan.
• If you upgrade to a new version of the App on the corresponding "Market", is data lost?
• What if the user gets a call while using the app? What if there comes an SMS?
• Does the battery seem to go down dramatically with the use of our App?

Compatibility, and multi-device environments

• Have you validated the app on devices, OS versions and models agreed?
• How does your application behave on different screens and resolutions?
• What if the integration server changes? Does it support a seamless transition?
• Can our application affect other third-party systems? Can it jeopardize other systems or degrade them?

About Belatrix’s QA Services

Belatrix’s Quality Assurance department provides testing and quality assurance services for projects developed within the company (internal projects) or by other companies (external projects). It also supplies mechanisms to improve existing practices by developing standardized processes for the entire company and by applying ISO™ standards, international standards, and lessons learned from CMMI™-SEI.

Belatrix’s QA team specializes in high value-added QA services such as Load Testing, Scripted Testing, etc. Because of this we are extremely selective in our recruitment process and only hire top software engineers that have a passion for Quality Assurance.

Some of the Testing Strategies we follow are:

• Functional Testing
• Risk-Based Testing
• Graphic User Interface
• Acceptance, Usability and Accessibility
• Performance (Stress and Load)
• Regression
• Installation and Configuration
• Security
• API Testing (N.Unit and MS-VSTester Edition)
• Automation (Python, VBscript, Apodora, Mercury QTP)
• Smoke Test

We invite you to CONTACT US to learn how a relationship with Belatrix will give your company a distinctive advantage through low cost, disciplined, and high quality software development and quality assurance services. 

Download Whitepaper: Mobile Testing Best Practices in PDF.