Website usability testing, application usability testing, and laboratory usability testing—analyzing user behavior and reactions is now a standard.
The necessity, indispensability, and usefulness of researching representative groups of users are indisputable.
Research helps improve websites and mobile applications and makes it possible to solve problems on websites that users encounter.
They can improve functionality, streamline processes, and support achieving goals on a website and performing typical tasks.
They're also extremely helpful in improving the user experience and meeting the target group's needs.
Of course, their diagnostic, analytical, and cognitive value depends on many factors, one of which is the appropriate choice of research method.
User experience research has developed and adapted from social science methodologies (such as psychology), several methods, and techniques that have proven helpful in discovering usability problems on a website.
One of the most fundamental divisions of these methods and techniques is their execution method, which can be in-person (face-to-face) or remote.
The first method is traditional and has quite a long history.
The second is becoming increasingly popular because of the pandemic and the dynamic development of remote usability testing tools (platforms that support conducting usability research).
Remote usability testing or in-person usability testing? In any location or a lab?
What are the advantages and disadvantages of in-person UX testing and remote UX testing? What should you pay attention to when choosing a research method?
Here are our questions. Today, we'll examine the nuances of the UX researcher's work more closely.
Do you want to perform UX Research?
What is usability testing, and what are UX tests?
While answering this question, I should mention a fundamental observation. It's about the approach to research, its results, and the reasons for doing it.
It's very common to treat testing (e.g., website usability testing) as a form of confirming an application's flawlessness, which makes such an approach look like a mere need for quality control.
Of course, research also has such a function, but reducing its significance, role, functions, goals, and results only to such narrowly understood needs (narrowly exploiting its potential) is always to the detriment of the digital product and the user experience.
The purpose of testing is to diagnose errors (such as an incorrectly operating function) and problems that aren't errors in the strict sense.
It's important to remember that usability testing aims to improve a product, not just to ensure it's flawless.
Usability testing should be implemented at the earliest possible stage of digital product development because it can improve the project and lower financial costs.
All right, but what exactly is usability testing? What is it all about?
Let's start with the definition of usability, which, according to ISO 924, is understood as the effectiveness, efficiency, and satisfaction with which users achieve goals in specific environments (e.g., mobile applications, web services, web applications).
Usability testing is a process based on scientific, academic methodologies and techniques that studies potential users' reactions.
Usually, they're asked to complete a list of tasks and interact with the application interface.
The time required to complete tasks and achieve set goals, the number of mistakes made, reactions (e.g., behavioral, emotional, cognitive), and many other indicators help determine various aspects of a digital product.
Indicators of usability tests can help define the following characteristics of a digital product:
- Usefulness
- Intuitiveness
- Ease of use
- Seamlessness
- Compatibility with the user perceptions
- User expectations
- User experiences
- User habits
- Design conventions
- User preferences
Even more importantly, usability testing allows us to understand the product's use cases, understand its future users more profoundly and multi-dimensionally, and offer them a product that aligns with their multidimensional preferences.
Usability testing plays a pivotal role in the development of user-centered digital products.
A positive definition of usability testing should also be supplemented with a negative definition that indicates what usability testing is not.
Usability testing is not:
- Aggregating opinions about a product — this is the domain of marketing research.
- An A/B comparative test because it doesn't answer the question of why variant A is better than variant B or vice versa.
- A variation of surveys, as surveys don't allow observation of activities.
- A variation of focus research aimed at learning about opinions but doesn't allow us to learn about behaviors.
- A variation of heatmaps, which involve creating a visual representation of the most popular and least popular sections, areas, and elements on a website.
Let's summarize: usability testing allows us to:
- Learn about behaviors (answer the question: What do users do?).
- Understand behaviors (answer the question: Why do users do it?).
- Identify the problems (answer the question: What is causing trouble for users?).
- Determine the level of satisfaction resulting from using a product (answer the question: What is the source of satisfaction/dissatisfaction of users?).
Usability testing can be divided into the following types of research:
- Moderated vs. Unmoderated
- Remote vs. In-Person
- Explorative vs. Comparative
Moderated usability tests
As the name suggests, moderated user testing, conducted in person or remotely, involves the presence of a moderator (a facilitator). A researcher fills the role of a facilitator by helping test participants throughout a study.
One advantage of this method is that a facilitator can help respondents overcome potential issues that may arise during testing. Moreover, they can react to test participants' ideas in real-time, which can deepen the research scope.
A drawback of this method is that it can be challenging for a moderator to stay unbiased, which can lead test participants to desired conclusions.
Unmoderated usability tests
In-person or remote unmoderated usability testing is characterized by the absence of a moderator/facilitator. In this type of testing, test participants interact with a product unassisted.
Respondents test a product at their own pace without external stimuli or distractions. However, if they encounter any issues or need clarification on parts of a test, no one is there to help and answer their questions.
Therefore, this type of test must be meticulously prepared to ensure that tasks are understandable and the instructions are easy to follow.
Explorative usability tests
Explorative usability testing involves discovering users' perceptions, perspectives, and reactions. It's usually performed at the initial development stage so the product design can be assessed regarding the elements that work and those that don't. Since the testing occurs at the beginning of development, we can save money and resources because the potential issues are identified early.
This type of research can help researchers and product designers identify design elements that can be improved or reworked. It can also help discover market niches, potential new functionalities, or new ideas.
During user testing, participants often need to perform various tasks and answer open-ended questions to discover their insights regarding a product.
Comparative usability tests
Comparative usability testing consists of comparing two (or more) product variants. Researchers can collect various data regarding a product, such as which of the tested versions offers a better user experience, which is more effortless to use, which one users prefer, etc.
Comparative tests allow us to make informed design decisions regarding the offered features and the development direction.
The most popular example of comparative usability testing is A/B testing, which requires a lot of participants.
Naturally, this basic division creates a matrix of different methods in which different types are combined.
Hence, for example, we can use moderated remote testing, unmoderated remote testing, unmoderated in-person testing, or moderated in-person testing.
In-person usability testing
In-person usability tests are conducted face-to-face in a laboratory. The user is asked to perform various types of tasks, either alone or in the presence of a researcher.
Like any research method, in-person usability testing has limitations and undeniable advantages.
The key advantages of in-person usability testing include the following:
- Collection of data from diverse sources (e.g., body language, facial reactions)
- Ability to simultaneously observe physical, emotional, and verbal reactions (holistic approach to behavior, user reactions)
- Ability to interact and deepen the testing with probing questions
- Improved capacity to control the dynamics and rhythm of the usability test and adjust its course to the individual needs and reactions of each test participant
- Relatively easier availability of participants
- Control over the test environment with the help of facilitators (no dependence on the quality of the Internet connection)
- Ability to conduct it at any stage of the design process — it doesn't require the creation of a clickable user interface
- In the moderated variant, the researcher can question the answers and check the reactions
- Collection of more direct insight into users' behaviors, needs, desires, and concerns
- Ability to mobilize researchers better, who must be prepared for very different circumstances
- Ability to test a project on a variety of devices
The most important disadvantages of in-person usability testing include the following:
- Higher costs of conducting the usability test (because of the need to hire a testing laboratory, organize transportation, etc.)
- Greater logistical problems associated with recruitment, execution of the study
- Longer duration of the entire research project
- Moderation and interpersonal skills of a researcher
- Influence of a researcher on a respondent and thus on their behavior (e.g., inexperienced researchers often unintentionally impose their beliefs and expectations on a user)
- In the case of many respondents, budgetary, organizational, logistical, and recruitment problems tend to intensify
Remote usability testing
Remote usability testing is gaining popularity and is slowly becoming one of the leading methods, mainly due to increasingly improved tools (dedicated remote testing platforms) and much better price and organizational attractiveness.
In the article "Remote Usability Tests: Moderated and Unmoderated," published on the NN Group blog, remote usability testing is defined as tests similar to traditional tests but differ in that the participant and moderator are in two different physical locations.
Study participants in remote user testing interact with a project (usually, the study lasts from 15 to a maximum of 30 minutes) in their own homes, offices, or other locations, and an expert observes their actions remotely.
Remote usability tests are performed using screen-sharing software or specialized testing platforms.
Key advantages of remote usability testing:
- Realism, the authenticity of a usability test — the natural context and the environment of the respondent have less influence on the course of the test and the respondent than tests in laboratory conditions
- Greater relevance of the information obtained (especially if it's destination-based testing)
- More flexibility in the selection of respondents
- Lower costs, shorter execution time, and greater ease of obtaining respondents (some platforms offer access to their own databases of study participants, who can be selected according to preferred criteria)
- Ability to examine more people
- Capability to quickly and accurately test selected elements of an application
- Learning about typical usability problems
- Ability to quickly test a prototype (with the proverbial one click)
- Shorter time between test creation and results
- Smaller organizational and logistical requirements (only personal equipment and Internet connection are necessary for the usability test)
- Ability to study users from different countries and cultures
The most important disadvantages of remote testing:
- One-way nature, the possibility of communication problems
- Greater risk of test failure due to factors beyond the control of the researcher and the respondent (such as a broken Internet connection)
- Less control over the test environment
- Need to conduct pilot studies (especially in the case of unmoderated remote usability testing)
- Meticulous and thoughtful planning of the study in every detail
- The limited duration of the test forces the selection of more straightforward tasks with a narrower scope
- Greater risk of testing financially motivated users, "professional testers" who perform tasks routinely
- Thinking aloud can be a rather difficult skill for respondents to maintain
- The difficulty in finding the optimal way of communication, the balance between listening to the respondents and questioning them
- Lack of real-time respondent support in unmoderated studies
- Hard-to-predict technical and operational issues with the research platform
Iterative usability testing
No matter which testing method is chosen, we must remember an essential point in the article "Usability Testing," published on the Interaction Design Foundation (IDF) blog.
Usability testing produces the best results if it's carried out methodically and according to specific guidelines, which include:
- A study plan — a precisely defined scope of the study; the researcher should know what they want to test, why they want to test it, and with what methods they want to conduct the test.
- User tasks — tasks should be prioritized, realistic (e.g., achievable in the available standard time of one or two-quarters of an hour), clear and understandable, and structured within a scenario of the most natural use of the digital product under study.
- Recruitment of study participants — the aim should be to define the target group precisely and select its representatives with the help of screening questionnaires.
- Test facilitation — we should create a suitable and optimal environment that doesn't create technical, functional, or communication issues.
It's also crucial to maintain fluidity in the conducted tests by following the guidelines concerning the following:
- User evaluations — quantitative (e.g., time taken to complete a task, success rates) and qualitative (e.g., facial reactions, changes in body language)
- Reporting — should result from a meticulous, critical, reliable, and exhaustive examination of all data obtained during the test
Performing usability tests: usability testing tools
How should we conduct an in-person or remote usability test? What tools and usability testing platforms can help us?
As we mentioned, usability testing tools are continuously improved, and new ones are also appearing on the market. They're indispensable parts of both types of usability testing (remote and in-person).
UserZoom
UserZoom is an in-person and remote user testing tool that allows us to conduct studies using various research methods, including tree testing, card sorting, and surveys. It enables us to perform tests on Android, iOS, and desktop devices.
Its features include, among others, session recordings, analyses, and visualizations of results.
Moreover, UserZoom offers research templates that can aid us with creating a research plan.
Testbirds
Testbirds is a dedicated usability testing tool for all types of devices (iOS, Android, desktop). It offers access to over 1 million testers, which makes it easy to find a suitable target audience and collect user feedback.
Testbirds enables us to enhance user research by combining various research methods and involving different analytics platforms.
The platform offers access to such features as exploratory testing, structured testing, competitor analysis, prototype evaluation, crowd surveys, and more.
PlaybookUX
PlaybookUX advertises itself as an all-in-one user feedback platform offering the possibility of performing quantitative and qualitative research.
Moreover, it provides features such as unmoderated testing, moderated interviews, transcripts, analytics, metrics, surveys, and much more.
The platform also allows researchers to recruit participants or invite existing testers and offers the option to localize the participant platform in over 50 languages.
Remote usability testing vs. In-person usability testing. Summary
- Usability is the effectiveness, efficiency, and satisfaction with which users achieve goals in specific environments.
- User behavior testing, UX testing with users, and remote usability testing are standard activities during the development of digital products, the usability of which is often compared to web analytics.
- Usability testing (of a website) helps improve functionality, streamline processes, and support users in achieving goals and completing tasks.
- The research's diagnostic, analytical, and cognitive value depends, among other things, on the correct choice of the appropriate research method.
- Usability testing (of a web application, for example) is often treated as a form, a process of confirming the flawlessness of an application or its quality.
- Reducing testing to such narrowly defined needs is always to the detriment of the digital product and user experience.
- Usability testing is a process based on scientific methodologies and techniques. In most cases, the software and its effectiveness after examination lead to constructive conclusions and enable the implementation of beneficial changes.
- Usability testing, UX research, and remote UX research with users help to understand product use cases, offer users a product that matches their multidimensional preferences, and rationalize the decision-making process.
- Of particular note is task-oriented testing, which is the ability to conduct remote testing, analysis, and observation in a natural environment. This can improve user experience and make websites more competitive.
- Usability testing allows us to learn about and understand behavior, identify problems, and determine the level of satisfaction that comes from using a product.
- In-person usability tests are conducted face-to-face in a laboratory, where the user is asked to perform various tasks.
- In remote usability testing, respondents interact with the project using dedicated research tools, and an expert observes their actions remotely.
- UX testing with users produces the best results when done methodically and according to guidelines.
