If you're asking yourself: What's user testing? What's website user testing?
What's mobile app user testing? What's user testing in UX, and what place does it occupy among research methods and techniques? And so on.
Then this article is definitely for you, Dear Reader!
In this article, we will try to describe exhaustively and broadly what website or mobile app user testing is.
The reason is pretty important.
While writing about user testing in UX, it's worth being aware of an old but very adequate and relevant remark made by Jakob Nielsen, who noticed that the task success rate is the simplest usability metric that should be considered.
Simply put, Nielsen is very direct and doesn't beat around the bush. If a web or mobile application user cannot perform a task or achieve a set goal, then all the other elements are insignificant. They are simply irrelevant.
Even the most aesthetic, visually attractive application that answers a very important need but is not even remotely useful will not be of value to the user.
After all, why would anyone need a nice but unwieldy, too-light hammer that can't even hammer nails?
Hence, you should be aware of the role and importance of user testing in the process of creating, optimizing, and developing digital products.
UX research (from UX audits and user testing to usability testing) is an important element in this process, if not the most important.
Diagnosing and solving problems with usability will allow you to compete better, gain market advantage and deliver to the market a product aimed toward the needs, expectations, capabilities, experiences, and competencies of its diverse users.
Ok, let's get to the specifics. If you are curious about the capabilities, advantages, and benefits of user testing, we cordially invite you to read this article.
Are you looking for an experienced UX agency?
What is User Testing of applications?
User testing in UX as a part of UX Research primarily allows you to accurately and reliably identify real usability problems of an application (web and mobile).
It also helps identify design flaws that you can diagnose at the very early stages of digital product creation.
The goals of user testing are intuitively clear.
It's about detecting a problem. However, we need to emphasize that it's not always easy to recognize its existence and acknowledge its importance.
The next step is to propose a remedy based on research results.
Thanks to this, the general user experience will be much better, more satisfactory, and in line with expectations.
User testing involves tracking and observing the interaction of users from a given segment with a selected system fragment (such as a functionality).
With that, it is possible to see how users, in reality, use the system and handle its operation.
What works according to expectations and experiences, what is important to them, what is not attractive to them, and what is not needed.
This is achieved by monitoring the interaction of representative users with the product in a realistic environment.
By doing that, you can see what works and, more importantly, what doesn't.
By combining task-based tests, and that's precisely what user testing is with individual in-depth interviews (IDI), it is possible to discover reasons for dysfunctions and the deeper causes of problems.
According to the definition proposed in the article "What is...User testing", User testing is a process in which the interface and functions of the website, mobile application, or digital service are tested by target users who perform specified tasks in realistic conditions.
This process aims to evaluate the website or application's usability and decide whether the product is ready to be offered to users.
User testing and usability testing are methods for measuring, diagnosing, discovering, and detecting the ability of users to perform a given task in an application or achieve a set goal.
The goal that is crucial and necessary to achieve from their perspective.
According to another popular definition, user testing is a research method that allows you to see how easy/difficult it is for users of web or mobile applications to use the functionality in question.
For instance, it can be a contact form or a credit card form.
Interaction of real users with the system allows you to evaluate functionalities and look at their usability from the perspective of user needs.
While writing about user testing, we must not overlook a particularly crucial issue.
In the subject literature, a discussion about the difference between user testing and usability testing appears very often.
What is this distinction about? Is it true that when writing about user and usability testing, we have two different things in mind?
User Testing vs. Usability Testing
We loyally warn you that trying to answer the question: What is the difference between user testing and usability testing? Looks a bit like finding five differences between two identical images.
Purists will say these are two different methods, although maybe a little similar.
In turn, people with a more "liberal" approach to definitions and indicating differences and scopes will say that user testing and usability testing have a shared goal.
Regardless of the definition we stand by, we need to write that both methods, approaches serve to detect problems with functionalities and usability.
They help discover design flaws and deficiencies in the offered user experience.
After all, they both are used to improve the digital product, to make it more flawless, convenient, intuitive, easy to operate, and effective from the perspective of user needs.
Furthermore, user and usability testing are essential to designing websites and web and mobile applications.
Their shared goal is so important and crucial that there is no point in crossing swords, right?
User testing allows you to observe how target users interact with the product. It is often aimed toward a better understanding of user needs and validating the solution in question.
User testing aims to understand whether users need a given functionality and how they use it.
In contrast, usability testing focuses on identifying specific usability problems.
It concentrates on finding an answer to whether the digital product works properly.
In the article published on the Maze blog, "A Beginner's Guide to Usability Testing," its authors point out that user testing usually precedes usability testing.
As you can read in the article, the goal of user testing can also be "to determine whether future users need the product, tool or service in question. User testing can also be used by marketers to test target audiences and better understand their needs and frustrations."
In practice, user testing involves testing a selected part of the product through specific test tasks whose goal is to check if the functionality in question works in accordance with expectations and whether it is ready to hit the market.
It's also about discovering cognitive frictions and the cost of interaction.
If combined with individual in-depth interviews, user testing makes it possible to obtain better insight and knowledge about unknown users' behavior patterns, preferences, and opinions.
The collected feedback is the basis for creating a more adequate and satisfying digital product.
User testing is conducted in a remote or in-person formula. It can be moderated or unmoderated and performed in natural or laboratory conditions. Through single tasks or a set of tasks. On the finished product or the prototype.
User testing usually begins by defining as precisely and concretely as possible and in a closed manner the goal or goals of the tests.
The definition of goals is essential for choosing the best metrics that will allow you to make relatively unbiased, measurable, comparable, adequate, and conclusive evaluations and comparisons.
When selecting metrics, you should keep in mind the remarks made by Jakob Nielsen, who, in the article "Success Rate: The Simplest Usability Metric," rightly observed that "Metrics are great for assessing long-term progress on a project and for setting goals.
They are an integral part of a benchmarking program and can be used to assess if the money you invested in your redesign project was well spent".
The most common indicators and metrics are the time necessary to perform a task, the number of mistakes made, and the percentage of completed and uncompleted tasks. Navigation and interfaces are the most frequently studied things.
When comparing user testing with usability testing, we must say that these two approaches to research are primarily different in how their results are used.
User testing is usually meant to determine the "readiness" of the digital product for entering the market. It is used to check if the product can meet user needs.
In contrast, usability testing makes it possible to see to what extent these needs are fulfilled.
Usability tests allow you to optimize a digital product's functionality in terms of not only the usability as such but the pleasure from its use. As well as in regards to satisfaction and emotions it evokes.
As a rule, user testing is often conducted at the concept development stage and usability testing at the MVP or prototype stage.
In practice, however, these two types of research are rarely separated.
The strongly recommended dominant approach is not to favor one method over another but to use different methods, techniques, and research tools depending on set research goals.
User Testing of websites and applications
User testing is used both for testing web applications (such as websites) and, increasingly often, for testing mobile applications.
Regardless of whether you want to test web or mobile applications on a certain level, the research process will be the same.
Most of all, the research project will have to have the following:
- Specific research goal. Colloquially speaking, researchers will need to know "what," "why," "who," and "for what" they want to study.
- Specific research groups, respondents who were profiled according to various project-relevant angles.
- Clear schedule. A clearly set structure in terms of goal, scope, time, sequence, and interdependence of carried out studies.
Often user testing should allow you to find accurate answers to the following questions:
- What problem/s do users face?
- How can it/them be solved?
- How to satisfy a need in a better-tailored, adequate, and desired way?
- How do users deal with problems, what cost of interaction can they accept, and what is unacceptable, unprofitable, and discouraging to them?
- Which tasks make it possible to discover problems with usability faster, easier, and the most effectively?
- How to capture the intensity of the problem?
The last question is particularly problematic.
Let us refer to Jakob Nielsen again. As he notices in the abovementioned article, few levels of success can be distinguished.
The test task can:
- End in complete success. It is completed flawlessly and according to assumptions.
- End after overcoming one tiny problem. The respondent makes a mistake, but its importance and rank aren't critical.
- End after overcoming a severe problem. It involves making a serious and critical mistake.
- Cannot be completed.
User testing, as with every research method, is effective the more it is methodologically correct, carried out according to the art of research, and conducted in an individual manner.
That is, taking into account in the broadest and deepest possible way the character of the product, project, and segment of users in question.
User testing, as a rule, should operate with standard methods, techniques, and tools in a " non-standard " way to obtain as useful and simultaneously reliable, "real" results as possible.
The goal of the researchers is to design the study and research process in a way that enables them to obtain information, evaluations, feelings, attitudes, and suggestions characterized by the following:
- Wide scope
- Significant depth
- Reliability and validity
- Practicality (so they can be used in optimization work).
User testing, especially when combined with other methods (e.g., IDI), should provide knowledge about the following:
- Problems arising during the use of the application in question
- Impressions and emotions evoked by design
- Level of satisfaction/unfulfillment of needs
- Satisfaction and emotional attitudes
- Preferences and willingness to use.
User Testing, understood narrowly as a method involving the creation of diagnosing tasks, definitely doesn't realize the full potential of UX research.
The approach in which it is an element of a broader strategy is much better and more recommended.
It provides much more valuable information, which is much more useful in the design, optimization, and development processes.
Let's also add that user testing and usability testing are treated as obverse and reverse of the same research coin and are not so much in competition as they are complementary.
How to effectively conduct User Testing?
The quality of obtained results of user testing largely depends on the knowledge, experience, and reliability of researchers conducting it.
Remember, user testing should have the following:
- Clearly defined goals
- Realistically formulated questions and research hypotheses
- Realistically formulated test tasks
- Properly chosen metrics
- Adequately selected methods and research tools
- A properly identified group of respondents.
The fundamental objective of every test task is to check if it's feasible and challenging, which problems it involves, and what kind of involvement, experience, knowledge, and competencies it requires.
What cognitive frictions it causes, and what cost of interaction it involves.
In other words, test tasks make it possible to discover what conditions the app needs to meet to be considered useful.
What requirements must the respondent fulfill to achieve their goals in web or mobile application.
Test tasks also enable you to determine the efficiency of the flow, that is, how easy/hard it is for users to move and navigate in the application.
However, learning requires an appropriate research method which is always a certain narrower way of viewing. It provides insight into a certain slice of experience, behavior, action, and reaction.
It allows you to learn about the problem but never exhaustively and unambiguously.
That is why it is necessary to enrich test tasks with, for example, individual in-depth interviews and various mapping techniques such as heat maps, click maps, or eye-tracking.
Methods, techniques, and research tools should be tailored every time to the following:
- Research goals
- Stage of the project
- Character of the digital product
- Characteristics of target users
- Expectations regarding, in particular, what purposes are to be served by the testing process, the research process
- Time in which the tests must be performed
- Budget the organization has assigned for research
- The knowledge that the research team has, in particular, the results of previous studies or analogous ones conducted by other researchers
- Complexity and diversity of users of the application under research.
When conducting user testing, remember that the more thoroughly you learn about, define, and segment model users, the easier it will be to choose methods and research tools and formulate relevant test tasks.
Variables that, to a large extent, determine the behaviors, reactions, expectations, experiences, competencies, and biases include:
- Age
- Gender
- Education
- Profession
- Social class
- Place of residence
- Lifestyle
- Income.
The validity, reliability, and usefulness of user testing are also largely a matter of the size of the research group.
We will write more about the problem of the size of the research group in the article "Why is it enough to test 5 users?".
Although it is a common belief that only 5 respondents are needed to determine 80-85% of all usability problems, the issue, as we wrote in the above article, is not so obvious.
Indeed, in most cases, it is enough to study only 5 respondents, but it is not a rule that applies to all types of research.
The approach promoted by the Nielsen Norman Group, according to which a larger amount of respondents has little effect on the outcome of a study, has received rightful criticism.
It is also worth remembering these methodological objections and selecting the size of the research group in a more problematic and far less obvious way.
For user testing to bring the most useful and measurable benefits and not waste time, energy, and budget, it must be performed methodologically correctly.
Most of all, this type of research should be conducted with an awareness of the advantages and disadvantages, limitations, and the need to expand and deepen the research goals.
What is User Testing? Summary
- According to Jakob Nielsen's remark, the success rate is the simplest usability metric that should be taken into account in the creation, optimization, and development processes of digital products.
- Diagnosing and solving problems with usability will allow you to compete better, gain market advantage and deliver to the market a product aimed toward the needs, expectations, capabilities, experiences, and competencies of its diverse users.
- User testing is a process in which the interface and function of a website, mobile app, or digital service are tested by target users who perform specific tasks in realistic conditions.
- User testing and usability tests are methods for measuring, diagnosing, discovering, and detecting the ability of users to perform a given task in an application or achieve a set goal.
- User testing and usability testing are essential elements of a website, mobile app, and digital services creation.
- User testing allows you to observe how target users interact with the product.
- Most often, user testing is aimed toward a better understanding of user needs and validating a solution in question.
- In practice, user testing involves testing a selected part of the product through specific test tasks whose goal is to check if the functionality in question works in accordance with expectations and whether it is ready to hit the market.
- User testing is usually meant to determine the "readiness" of the digital product for entering the market.
- User testing checks if the product can meet user needs.
- User Testing, understood narrowly as a method involving the creation of diagnosing tasks, definitely doesn't realize the full potential of UX research.
- The fundamental goal of every test task is to check whether actual users can perform it, whether it poses a challenge, what kind of problems it involves, and what cognitive frictions it causes.
- Test tasks also help determine how easy/hard it is for users to move and navigate in the application.
