• April 10, 2016

Startup Mobile App Testing 101

Startup Mobile App Testing 101

As of summer 2015, there were 1.6M Android apps and 1.5M iPhone apps. While startups make up a huge portion of the app market, those smaller teams have a big challenge ahead of them to create an experience that breaks through the noise and competition from millions of apps from other startups and bigger players with more financial resources.

The sad truth is that no matter how great the business & app idea, a bad user experience can tarnish any great idea in an instant. The quality of your app needs to be bullet proof, meaning during & after development, it’s analyzed, used, and tested on as many platforms, devices and use cases as possible – to ensure no user gets left behind.

The first question most startups ask: can I test in-house? If you have a solid dev team that’s proficient in QA, you might pull together a detailed process that uses every employee on staff and digs into the deepest corners of your app. But the reality is that you likely don’t have access to all devices and platforms that your customers might use.

Further, all members of your team, including non-developers, have been so thick in the creation process, weighed down by subjective and cognitive bias, that bugs, UX flaws, etc., are bound to slip through the cracks. It’s best to bring in outside solutions & users to test various features, flows and experiences.

Finally, to deliver the best user experience, mobile apps must be continuously updated to offer users the best experience – which means that your development, testing and deployment process must be buttoned up to get it right with each update. A strategy should be put in place early and iterated on as you learn and grow.

Here are the key terms and strategies that must be incorporated into your app testing strategy, to ask of your service provider, and to know of self-service tools:

Automated vs. Manual: Since mobile development is designed to be agile, testing automation is often looked at as a fast and valuable option with good ROI. Many testing solutions offer public and private clouds with the latest devices & platforms for manual and automated testing across thousands of interactions. If you have a strong internal dev team, you can also write your own test scripts. There are many commercial and open-source  options offering automation frameworks, tools and integration. Manual testing should give you a sense of what it’s like to be a customer of your app, from a real user, within the typical use cases as well as the outlying use cases, and everything in-between. Think: what happens when a user receives a call or text, in-app notifications, if wifi is spotty, low battery, etc. BlazeMeter, a company which offers performance testing as code, listed out the advantages and best practices for both automated and manual testing.

Device & Platform Fragmentation: According to TechTarget, mobile device fragmentation is a phenomenon that occurs when some mobile users are running older versions of an operating system, while other users are running newer versions. It’s pretty difficult for startups to acquire all devices and platforms, also backdating to take into account those that haven’t updated operating systems in a while. These different devices & platforms can vary greatly – resolution and layout size, speed, memory, etc. Deepanshu Agarwal wrote a helpful primer breaking down the technical differences when testing iOS & Android. For instance, here’s his high-level breakdown of Google vs. Apple:

  • Google’s Android is based mainly on Linux OS which forms the kernel of Android and “above” that kernel, a whole framework exists. The kernel and the core libraries are written in C/C++ and the API, through which one can develop his/her applications, uses mainly Java. Android’s source code is released by Google under open source licenses, even if most Android devices ultimately ship with a combination of open source and proprietary software.
  • Apple’s iOS is distributed exclusively forApple hardware. It is the operating system that powers many of the company’s iDevices. It’s a closed-source operating system. The iOS kernel is XNU and is mainly written in C/C++ and Objective-C. The applications must be written and compiled specifically for iOS and the 64-bit ARM architecture or previous 32-bit one. Xcode is the primary development environment for the iOS.iOS applications, like many of the higher-level frameworks and applications that are part of iOS and OS X, are written in Objective-C or the newly introduced SWIFT language.

Functionality: You might think you have your basic app functions down – after all, this is the basis on which you’ve built your app and how your users will interact with it – but it’s also the most important to make sure there are no mistakes or misses. This aspect of testing hits on all links, ecommerce functions, images, texts, controls, etc. InfoStretch, a company that helps enterprises launch mobile & IoT initiatives, detailed functional testing best practices, including having an issue tracking system to centralize results from tests and optimize task completion.

Usability: Before you launch your app, you’ll want to watch and listen to real users actually interacting with your app’s UX/UI. This goes beyond baseline functionality, and will help you understand what aspects of your app that users love, which you can continue to improve, and which aspects of your app users struggle with, and perhaps you should scrap all together or rethink from a usability perspective. KISSMetrics released a list of 17 testing tools for mobile UX. Be prepared for this aspect of testing by coming up with different user scenarios that you absolutely want tested, which results you want highlighted, and how to review (analytics reports, heatmaps, user testimonials & surveys, etc.) and act upon results.

Privacy & Security: Users are more and more guarded about their personal information, identification and credit card numbers, which are often required to conduct business in-app. You will need to identify any vulnerabilities and make sure your app follows any recommended regulation guidelines. If a customer doesn’t feel like their information is secure, they will go elsewhere, and if there are hacks, your customers likely won’t come back. Most likely, you can identify the concern areas with a security consultant and then conduct automated testing on areas that could be attacked. Applause, an app quality company in Boston, released a white paper that focuses on all things app security testing – such as how mobile data travels over many different networks, how to protect your users’ information, and concerns for native vs. mobile web.

Localization: If your business is expanding to new territories, you need to make sure users in those territories feel as though they are using an app made specifically for them. This includes translation, but also extends to cultural usability expectations. Business2Community dives into the localization best practices a startup should keep in mind for their app (and thus what to test for), including writing, coding and design implications.

Performance, Reliability & Load Testing: Slow load times and app crashes are unacceptable in the mobile app world as user attention spans shrink daily and loyalty is often given to the app that is most reliable at the moment. Load testing typically measures behavior under both normal and anticipated peak load conditions so that the startup understands maximum operating capacity of their app, any bottlenecks, and which particular aspect of the app is causing the breakdown. This could impact battery drain and on-device and memory card storage. SOASTA, which offers a digital performance management platform, created a useful mobile app testing checklist around performance.

For startups, app testing is often viewed as the final leg in a long race to build the best app. However, Renu Rajani of Capgemini, a consulting firm, created a quick yet conclusive list of valuable recommendations to enhance usability throughout the lifecycle of both your business and your app:

  1. Test Early, Test Often
  2. Test competitor products before developing your product
  3. Test wireframe early in the development process
  4. Test user journeys through live users for real customer experience
  5. Conduct cross-browser/cross-screen tests
  6. Test on multiple platforms

Looking for the app testing partner for your business? Maybe you aren’t sure whether you need a self-service tool or manual testing services. Fill out this quick form and we’ll find the solution for you.