User Testing, particularly User Testing of mobile applications, gains significance along with the flood of mobile applications. Over five million applications are currently available on Android and iOS systems (combined).
The market for mobile applications grows, and the creation of mobile applications itself, performing user interface testing, and user testing of mobile applications starts to be a part of the needs of mobile device users.
It's hard today to indicate a human need that a mobile application wouldn't try to fulfill — more or less successfully.
Mobile apps need to be more and more perfect in terms of technology and functions. It's inevitable. Additionally, they also need to be useful and refined regarding user experience. Otherwise, they sentence themselves to a complete lack of interest from users.
And this, in turn, requires creators of mobile applications to fit into a very complex structure of mobile app users.
UX research leads to learning about and understanding this structure. In the case of mobile applications, this includes, among others, usability testing and user testing.
We should say at the outset that performing user testing is also dictated by the growth of the number of mobile devices.
A few thousand manufacturers compete with each other on the global market, and their offers include mobile devices with various parameters.
The length of a diagonal, resolution, tactility, and operating system, to name a few of the most obvious aspects, significantly influence how a mobile app will work on a given device. And thus what mobile user experience it will offer.
The number of manufacturers, models, types, and technologies is a considerable challenge for mobile application developers.
Just like the need to face the requirements of platforms, that shouldn't be forgotten because they're a significant element in this puzzle. We wrote about this in the article "Designing mobile applications for Android and iOS."
The success of a mobile app is determined by many factors, from technological and functional, to those related to the user experience of a mobile app.
According to the study results conducted by eMarketer, the vulnerability of applications to uninstallation is very high, and each use of an app involves the risk of deleting it if it doesn't meet expectations.
With so many variables, trying to discover what users need and what problems they may face is pure guesswork without UX research, particularly without user testing of mobile applications.
So what is user testing of mobile applications? What dilemmas does this research method involve? What are the most popular tests of mobile user testing?
We'll deal with these questions and problems in this article.
We cordially invite you to read on!
Do you want to perform UX Research?
What is user testing of mobile applications?
Let's start with an explanation, a definition of user testing.
User Testing is one of the research methods used in user experience research, and it involves observing the interaction of mobile or web application users with a selected system fragment (e.g., a function).
In practice, user testing involves performing tasks determined in a research scenario, usually in a realistic environment.
According to the description of usability testing presented on the blog of a research company Nielsen Norman Group in the article "Usability Testing 101," during a testing session, a facilitator (also called a moderator) asks a respondent to perform tasks. The facilitator observes the participant's behavior and how they interact with an interface.
The main goal of user testing is to identify problems occurring in the created product or digital service, discover ways of fixing them and, above all, learn about the behavior and preferences of the end users.
Moreover, the purpose of user testing is to diagnose whether a web or mobile app is ready to be offered to users and whether it has issues regarding usability.
User testing primarily allows you to find errors with varying levels of criticality, eliminate usability problems, and rationalize the decision-making process regarding optimization and application development.
Specific tasks, real users representing a given Persona or a user segment, a realistic environment, and context enable you to achieve relevant results, reliability, and utility.
If you take the fierce competition in the mobile application market into consideration, user testing is a necessity that you can't ignore if you don't want to sentence your product to failure.
And without UX research, it's a certainty.
What types of tests are performed in the case of mobile applications?
User testing of mobile applications is a variant of user testing focused on examining mobile application interfaces with users.
It's worth noting that the subject of UX research (as mentioned by researchers from NN Group in the above-cited article), particularly usability testing or user testing, is not users but interfaces.
As Kate Moran writes, a researcher affiliated with NN Group, the term user testing may imply that researchers test a respondent.
As she rightly observes, in user testing, the user is never studied, only the interface. User testing is supposed to mean tests with users.
Specifying the meaning of this term is important because it defines the subject of research, and it's always the interface of a mobile or web application, which, thanks to research and the role of feedback, is supposed to be improved.
It needs to be clearly stated that users are never the problem but the interface. The research situation focuses on it, and it's supposed to suit the users and never the other way around.
In the case of mobile applications, the research subject is a bit more complex because mobile apps can be installed on different devices.
Moreover, mobile applications are "written" for various operating systems; they're used in more diverse contexts and operated in a more complex way.
The repertoire of gestures is much broader than a traditional operation with a mouse that involves moving it, clicking, and scrolling.
Usability and a positive mobile user experience are also much more dependent on such technological variables like the following:
- Quality, speed, and stability of Internet connection
- Speed of a processor, size of RAM
- Availability of Wi-Fi, 3G, 4G, 5G
- Durability and capacity of a battery
- Display parameters (size, resolution, brightness, technologies, type of matrix, aspect ratio)
- Quality of the tactile technology
- Quality of technology and solutions typical for mobile devices (e.g., cameras, GPS)
- Quality of a device itself.
The list of factors determining the success or failure of a mobile application in the market is much longer. You need to orchestrate all these requirements, expectations, and determinants to distinguish yourself from the competition.
How a mobile application works on a given device or operating system, or, generally speaking, how it cooperates with the hardware and software of a given device, is also crucial.
For example: How quickly does it start? How much memory does it take? How much strain does it put on RAM? How much energy does it consume? What level of security, stability, performance, and speed of working does it offer?
How does it work in different environments? (e.g., under high and low sunlight conditions, high and low temperatures, or high and low humidity).
The above matters are essential for UX/UI designers, researchers, and moderators performing mobile user testing.
To a certain degree, they're multi-dimensional and codependent challenges that expect a positive resolution.
Although testing of mobile applications is not simple, it has many benefits that compensate for the effort that needs to be put into conducting it.
User testing of mobile apps is very helpful in discovering the following:
- Unknown behavior patterns
- Needs and expectations
- Easiness/difficulty in operating a given function
- Easiness/difficulty of processes and use cases.
Mobile user testing — the most important challenges
According to statistics from Statista.com published in the article "Number of smartphones sold to end users worldwide from 2007 to 2021," in 2021, 1.5 billion mobile devices were sold globally.
From the perspective of usability testing of mobile devices, it means the need to adapt applications to at least the majority of popular brands and models.
Because there is a strong market expectation — mobile apps should be universal, they should work smoothly without an issue, if not on every device, model, or operating system, then at least on most of them.
Each mobile application must be compatible with the majority of globally available mobile devices to achieve success (not only regionally).
And this means the necessity to design tests that will make it possible to check whether this ideal is achievable and to what degree.
The matter is further complicated by operating systems which, even if you take into account only the three most important platforms, iOS, Android, and Windows, are constantly developed and updated.
The compatibility of a mobile app with every operating system's version is a serious challenge in terms of technology, costs, and research.
Another research challenge is the variety of screen sizes and aspect ratios. Sizes and ratios have a considerable influence on mobile user experience.
The size of a screen influences how an app is displayed and how it presents on various devices. And ultimately, how it works.
The requirement for adapting a mobile application to each new screen specification that appears in the market seems reasonable.
In practice, this means adapting buttons, navigation, and individual screens to diverse sizes and ratios and maintaining consistency on every device.
The type of mobile application is also a variable that is a design-research challenge.
Mobile applications can be divided into three main types:
- Native apps are created for one specific operating system. This means that, for example, a native application developed for Android won't work on other operating systems.
- Web apps are designed to be responsive versions of websites, meaning they can work on any mobile device or operating system.
- Hybrid apps are a combination of features of the above types.
As you can guess, each application type has advantages and limitations.
The choice of a particular solution depends on many factors; however, regardless of the type, it's necessary to test applications in terms of capabilities and limitations.
When thinking about just a few of these issues, it becomes clear that testing mobile applications is essential if you want to provide users with a trouble-free experience.
Because there is a big and real risk that a mobile app will work exceptionally well on some devices and on others, its way of working and offered experience will be unsatisfactory.
How to conduct user testing of mobile applications?
Let's start with an essential comment.
User Testing of mobile applications as a research method is similar to user experience research on websites.
Nevertheless, mobile user testing is a separate method that is continuously developed.
The character of mobile devices and mobile apps requires adapting research methods.
When testing interfaces of mobile applications, creating tasks, and leading in-depth interviews, it's important to remember their character.
Above all, you should remember that usability and experience should be:
- Separately tested on each operating system, diagonal and resolution
- Tested with simple and quick tasks
- Tested continuously during the mobile app development cycle (before development: on prototypes, wireframes, before release: on MVPs, after release: on a ready product that users use).
Let us remind you that the usability of a digital product is expressed in the easiness/difficulty of operation, of achieving goals, of understanding the way of working and learning how it works, of achieving goals and effectiveness of fulfilling needs.
In turn, mobile user experience is how users see a mobile application. User experience research of mobile applications involves measuring their emotions, assessments, reactions, behaviors, and preferences.
User testing of mobile applications allows you to gain confidence in clients' satisfaction, directly influencing an organization's sales revenue, profits, and earnings.
How to test an application with the participation of users?
User experience research, qualitative or quantitative, moderated or unmoderated, needs to have a precisely defined goal, including user testing of mobile applications.
While designing the goals of user testing and tasks that users will need to perform, it's worth remembering the most important attributes of usability.
Namely, the ease of learning, efficiency, memorability, error tolerance, and satisfaction.
The ease with which a user can learn to operate a mobile app is crucial because it strongly affects the first impression. We wrote about this effect in the article "Designing a good First Impression."
You should keep this in mind because in fractions of seconds and after a few first interactions with the app's interface, most users formulate very solid and strong beliefs.
Particularly regarding whether they like the mobile application, whether it complies with industry standards, whether it's trustworthy, whether it meets standards, and whether it's easy to use or understand.
The ease with which a user can learn how to use an app is simply the speed with which they can learn, remember, and control the operation of available functions.
Efficiency is the speed of performing tasks and achieving goals when the user has learned how to use the app.
Memorability refers to the ease with which the user can use functions after not using the app for a while.
The number of mistakes the user makes in the app is connected to the clarity of the interface. The goal of user testing of mobile applications can be, among others, discovering which elements are error-prone and which cause uncertainty and disorientation.
The pleasure of using the app and its functional and aesthetic attractiveness is a source of satisfaction, attachment, motivation, and loyalty. Thus, it's necessary to remember this aspect during the usability testing of mobile applications.
Discovering the reactions of real users during tasks allows you to determine the usability of an app more accurately.
The goal of user testing can be, as we mentioned, examining one of the five dimensions of usability.
It's worth precisely determining the goal of testing because clear purposes lead to clear and easy-to-measure results. Thus, you can be sure that you studied what you wanted to study.
Clearly defined goals also influence the efficiency, smoothness, and effectiveness of conducting research, its credibility, and reliability, and they translate into the trust that the results inspire.
The task users will need to perform should be equally specific, clear, simple, understandable, realistic, and feasible.
The task shouldn't contain hints, suggestions, or allusions.
In short, it should be formulated in a language that doesn't suggest a way of performing it, its goal, or the problems that it tries to diagnose. Above all, the task shouldn't even indicate that the problem exists.
Tasks in user testing of mobile applications should come from the actual goals of users, meaning goals that are important to them and that they would want to achieve.
They also should involve the relevant elements of an application, for example, crucial information, navigation, and an element of critical importance to the user or business owners (e.g., mandatory form fields of a credit card form).
In a formal sense, the task should be formulated as short as possible. It also should indicate a concrete interaction that the user should perform (e.g., enter data into their profile).
One of the most important features, attributes of mobile applications is using them in very different contexts: in private, during work, at home, or in public.
Thus, it's recommended to perform user testing of mobile applications in as typical contexts as possible, in a context where the app is/will be used by users.
If in marketing content is king, then when it comes to research and mobile application testing, the king is context.
Although testing done in a laboratory allows you to better control and observe the selected variables, the situation of performing such tasks is far from natural.
It's worth remembering that mobile apps are used in dynamically changing circumstances. Interaction with them is based on micro tasks and discontinuity.
Interactions are evoked in various ways — internally as an expression of a need and externally (e.g., through alerts, notifications, reminders) as an expression of a reaction to a stimulus.
When conducting user testing of mobile applications, it's worth remembering the note made by Oliver Lindberg in the article "Conduct Usability and User Testing for Mobile Apps Like a Pro," who wrote that the key to usability testing of mobile applications is to try to replicate the natural conditions of a mobile environment as faithfully as possible.
User testing of mobile applications. Summary
- Along with the huge increase in mobile applications, user testing of mobile apps has become a necessity.
- The increased number of mobile devices also determines the need for mobile application testing.
- User testing is one of the UX research methods, and it involves observing users' interactions with a selected interface fragment.
- User testing involves performing tasks determined in a research scenario, usually in a realistic environment.
- The main goal of user testing is to identify problems occurring in the developed product or digital service.
- User testing is used to discover the behaviors and preferences of end users.
- User testing primarily allows you to find errors with varying levels of criticality, eliminate usability problems, and rationalize the decision-making process in terms of optimization and application development.
- The list of factors determining a mobile application's market success or failure is very long.
- Mobile applications should work smoothly and without problems on all devices, models, and operating systems, or at least on most of them.
- User testing of mobile apps is essential if you want to provide users with a trouble-free experience.
- There is a big risk that a mobile app will work exceptionally well on some devices, and on others, its offered experience will be unsatisfactory.
- The character of mobile devices and mobile apps requires adapting research methods.
- When testing interfaces of mobile applications, creating tasks, and leading in-depth interviews, it's important to remember their character.
- The usability of a digital product is expressed in the easiness/difficulty of operation, of achieving goals, of understanding the way of working and learning how it works, of achieving goals, and the effectiveness of fulfilling needs.
- When designing goals of user testing, you should remember the most important attributes of usability.
- Clearly defined goals of user testing influence the efficiency, smoothness, and effectiveness of conducting research, its credibility, and reliability, and they translate into the trust that the results inspire.
- It's recommended to perform user testing of mobile applications in as typical contexts as possible, in a context where the app is/will be used by users.
- The key to user testing of mobile applications is to try to replicate the natural mobile environment as faithfully as possible.
