Usability Testing

Benefits of usability testing

Usability testing offers many benefits and provides data that can be turned into actions and tangible improvements. It can help us distinguish our brand and product by offering a positive user experience. And that's only the general advantage of conducting testing.

Minimizing the cost of the development process

Performing user testing enables us to save time and costs before we launch a product on the market by identifying any possible issues. Some development errors may be challenging to fix after introducing a product to the market, and they need to be fixed as soon as possible to counteract any negative user impressions.

Usability testing allows us to identify problems early and makes it possible for the development team to resolve them, thus ensuring the best possible product launch.

Fulfilling the needs and expectations of users

User testing helps us understand user needs and expectations more deeply. It also provides valuable user feedback that allows us to adapt the product more accurately to users' desires. This type of user research can show us where users encounter problems, misunderstandings, or cognitive frictions.

Increasing accessibility

The accessibility of digital products should always be important. Providing accessibility features enables us to offer products to an even wider audience regardless of disabilities. Therefore, it's crucial to include users with disabilities in our participant pool. This will allow us to make sure that the offered features work as intended and are helpful.

Improving user satisfaction and brand perception

As mentioned, performing usability testing allows us to fix errors before launch and gain insights into users' expectations and needs. This results in improved user satisfaction and a better perception of our brand—increasing its reputation and trustworthiness.

Offering a positive user experience is crucial to attracting new users and maintaining the existing ones.

Avoiding design assumptions

Usability testing provides us with actual data from real users, enabling us to make informed design decisions. We shouldn't rely on unproven assumptions and biases, as they won't reflect true user actions and behavior.

Additionally, we should be wary of researchers' biases during user research and ensure that they base the results on the actual user performance.

What can be studied with usability testing?

Usability tests can help answer questions such as why some people abandon a website or why it doesn't achieve the expected conversion result.

Through this research, we'll also discover what users find difficult, what frustrates them, and what they don't understand.

Usability testing (e.g., interface testing) helps to verify whether users understand correctly the following:

• Functions of a website or an application
• The usability scope of a website or an application
• Use all available functions
• Navigate a site with a sense of purpose and meaning

To what extent is it problematic/easy for them to:

• Find information
• Achieve goals.

Usability testing makes it possible to examine the following:

• Attractiveness (to what extent a product stands out from competing solutions)
• Effectiveness (to what extent a product helps to achieve desired goals)
• Aesthetics (to what extent contact with a product is a source of aesthetic pleasure)
• Ease of use (the speed with which it's possible to learn how a website or application works)
• Approachability (the speed with which it's possible to learn all the functions)
• Satisfaction (the strength of the emotional reactions triggered by the use of a product)
• Effectiveness (the speed at which a user achieves the set goal)
• Comprehensibility (unambiguity of the function-purpose relationship)

Usability testing helps study user reactions by considering the structural elements of a website, web application, or mobile application.

Usability tests make it possible to investigate problems that arise during interactions with the following:

• Forms
• Search engines
• Navigation
• Menu
• Home page and subpages
• Calculators (or similar tools)
• Product pages
• Media embedded on a site
• Galleries
• Widgets
• Layouts
• Information architecture

Advantages of usability tests: examples

As mentioned, usability testing has many benefits and advantages, which is why it is so widespread when it comes to identifying issues with user experience and usability.

The most fundamental advantages of usability testing include the following:

• Speed of execution (the research with a summary report can be completed in a few days)
• Appealing price (they are among the cheapest studies)
• The relative unproblematic nature of their conduct (they can be carried out stationary and remotely, moderated and unmoderated)
• Optimality (they capture most of the critical problems and make it possible to study users of a site without space limitations)
• Effectiveness (make it possible to correctly diagnose the causes of, for example, a low conversion rate in a short period)

The fundamental advantages should be supplemented by other equally important ones, such as:

• Diagnosis of specific sales and branding problems, even at the design stage
• Optimization based on reasonable information (e.g., A/B UX testing)
• Improvement of experience, satisfaction, and benefits of using a website or application
• An adequate product that fulfills real needs
• Reducing customer service costs
• Increasing customer loyalty
• Increasing ROI.

Types of usability testing

Developing research tools (mainly for collecting, analyzing, and presenting data) has contributed significantly to expanding the number of research methods and techniques.

We can divide usability testing (UX testing) into:

• In-person usability testing
• Remote usability testing
• Qualitative usability testing
• Quantitative usability testing

All these types of usability testing can be performed in a moderated or unmoderated manner.

In-person usability testing

In-person user testing is conducted face-to-face, usually in a laboratory or other location specifically prepared for conducting tests. The organization of an in-person usability test is much more costly because it requires a dedicated space, equipment, and a way to transport and accommodate research participants.

The drawback of this type of research is that the testing environment is far from natural and can affect the end results.

Remote usability testing

Remote testing is cheaper to conduct than in-person tests. This is primarily because it doesn't require renting a space and utilizes the users' equipment. It's also easier and faster to organize. Usually, remote usability testing is conducted using two formulas: moderated or unmoderated. These two methods boil down to the presence of a facilitator (a researcher) or their absence.

Moderated usability testing enables the facilitator and user to interact. The researcher is present during the study and can offer any clarifications if needed, although they should abstain from interrupting the test as much as possible.

Unmoderated usability testing is conducted without the facilitator's presence and with a dedicated research tool. This type of testing tool provides access to tasks and allows users to start testing without additional help. The goal is for the user to finish the test by themselves and on their own time to gather as realistic results as possible.

Qualitative usability testing

Qualitative usability testing focuses on gathering data regarding how users use a given product and its user interface. It aims to study user behavior and their reactions. This type of usability testing is the most popular because it allows us to see the reasons behind user's actions. It's also a great method for finding issues with the user experience.

Quantitative usability testing

Quantitative usability testing focuses on aggregating quantitative data, such as metrics. This type of research allows us to measure various UX metrics, such as time on task or task success. It provides us with numerical data that we can use in benchmarks or comparative usability testing. Quantitative data are a great addition to qualitative data; therefore, using them in tandem is beneficial.

The disadvantage of this research type is that it doesn't offer deeper insights and doesn't explain why certain choices were made.

In most cases, UX usability testing is performed using a task-based formula.

What does usability testing consist of?

Participants

Participants are an invaluable part of user testing. Research participants can consist of new users who have never interacted with a product before or existing ones who have some familiarity with it. When recruiting participants, it's important to remember that they should reflect the defined target audience.

During the user testing, participants are asked to think aloud while doing tasks. The "think aloud" method involves users talking aloud and explaining what they do and why. This allows researchers to gather user feedback and insights into user behavior.

Facilitators

The task of a usability test facilitator is to guide the research participant through the study. Their role in moderated usability testing is to ask questions, provide clarifications, and ask any follow-up questions if they find it necessary.

Facilitators need to be careful when offering help to participants and when asking questions because they risk influencing user behavior and their answers. Moderators don't have to do this consciously, and that's why it's crucial to be aware of it.

Tasks

Tasks in usability testing consist of activities a user needs to perform during testing. These tasks should reflect the actions a user may take in real life. For example, a task can ask the user to find a specific product in an online store. The goal of the user is to find the product and narrate the process.

How we formulate tasks is crucial. We should avoid situations where users misunderstand a given word or instruction and are confused about how to proceed. This can also result in users performing the task in a completely different way, contrary to expectations.

The issue of misunderstanding instructions can be fixed with the presence of a facilitator, but this can't be done in unmoderated usability testing. Therefore, we should pay special attention to the wording of research tasks.

The course of usability testing (UX testing)

Each usability research should be conducted based on a detailed scenario.

A correctly prepared scenario should stem from a thoughtful consideration of fundamental issues:

• The scope of research (what we want to find out)
• The method of research (from whom and how we want to find out)

Established assumptions form the basis for creating a research scenario and guide the selection of respondents and the research method or techniques.

Regardless of the method, technique, or tool with which they're performed, usability tests follow a similar course.

Each phase has its role and must not be overlooked. The accuracy of all the results depends significantly on the reliability of the execution of each step.

A typical usability test should include the following:

• The preparatory phase is used to determine the objectives of a study, research tools, target groups, and tasks to be performed by respondents.
• The respondent recruitment phase aims to recruit respondents (including reserve respondents).
• The research phase usually spans about 60 minutes per respondent.
• The analysis phase analyzes the collected data and observations.
• The synthesis phase draws firm conclusions and makes optimization recommendations.
• The reporting phase is used to inform individual stakeholders about the results and recommendations.

In task-based usability tests, respondents perform specific actions and are most often asked to do the following:

• Find a product subpage and make a purchase
• Locate a subpage dedicated to a given service and order it
• Contact a company using a form or widget
• Register in a store or service
• Edit their account after registration
• Try using a SaaS application (e.g., issuing an invoice in an e-accounting system)

Tasks can focus on the following:

• Execution of actions
• Problem-solving
• Understanding the operation of a website or application

During the usability study, moderators pay special attention to:

• Actions that were taken by respondents (e.g., clicks)
• Their reactions
• Time and ease of performing a task
• Means of achieving goals
• Number and type of mistakes made
• Obstacles encountered during the execution of a task.

The most popular research techniques include:

• Click-tracking (session recording)
• Eye-tracking
• Post-its and affinities
• Rapid iterative testing and evaluation

Click-tracking gathers information about how users navigate a website or app and identifies the elements they click on most often. This technique also allows us to discover any issues with the navigation process or product functionality.

For example, it can show us where rage clicks occur. These clicks happen rapidly, one after the other, and usually imply that a user was trying to click on an element that they thought was interactive. Thanks to this, we can improve such elements and reduce user confusion.

Eye-tracking is a method that tracks eye movement and provides data about website elements that users look at. It utilizes specialized equipment to determine which parts of the webpage catch users' attention.

Eye-tracking in usability testing allows us to optimize the placement of CTA buttons or other helpful elements from the users' perspective.

Affinity diagramming or Affinity mapping is a method for sorting findings and ideas into helpful and organized clusters. This method utilizes sticky notes to divide findings into groups that are focused on similar themes or concepts.

For example, we can divide our findings into those concerning the design of the user interface and those relating to more general users' feelings about a product.

Affinity mapping helps us prioritize clusters and focus on improving groups of elements one at a time. Additionally, it provides insight into user behavior and preferences and makes large amounts of data more manageable.

Rapid iterative testing and evaluation (RITE) is a technique used in usability testing to create and test prototypes of a product quickly. It can also be called iterative prototyping. We can use interactive prototyping to develop low and high-fidelity prototypes to gain early user feedback.

The RITE method enables us to save costs by identifying problems at an early development stage and ensures that the product vision is shared by all teams engaged in a project.

How to conduct usability testing?

Writing down the steps we need to take to plan usability testing will enable us to go through the planning stage smoothly and effortlessly. A well-made plan can spare us unnecessary anxieties and ensure that we stay on task. It's particularly helpful when it's our first approach to organizing usability testing.

Define the goal and target audience

The obvious goal of user testing is to assess a digital product's usability and user experience. To make this aim more precise, we should determine the target users we want to research.

We should determine who our target users are. What is their age, profession, and location? What do we want them to accomplish? What kind of functions do we want them to test? What is the expected timeframe?

These questions will help us determine the research goals and allow us to choose appropriate methods and metrics.

Find a suitable testing tool

We need the right tools to conduct usability testing. We should find a suitable research tool that will fulfill our needs and enable us to conduct tests at every stage of the product development process.

Online tools let us and our team work efficiently and make the research design process much easier and smoother. They allow for collaborative work with team members and make it simpler to perform remote user testing.

As a general rule, we should opt for tools that let us record testing sessions, test prototypes, select a ready-made research template, and choose various types of research methods.

Some examples of usability research tools include the following:

• Maze
• Userlytics
• UserZoom
• Hotjar
• User interviews

For more about testing tools, we recommend our article "UX research of mobile and desktop applications. Best user research tools".

Determine criteria

We should select the right metrics for evaluating user experience. They will help us measure user satisfaction and evaluate the study as a whole. They will also enable us to keep track of product performance.

Common examples of UX metrics include the following:

• Task success
• Time-on-task
• User satisfaction score
• Misclick rate
• System usability scale

The task success rate allows us to determine the number of successfully completed tasks. This metric can help us identify user experience or interface design problems. It's easy to calculate the task success rate by taking the total number of successfully completed tasks by all respondents and dividing the result by the number of all tasks completed by testers.

The average task success rate is around 78%. It's crucial for users to be able to complete tasks. If the success rate is significantly lower than the average, then the product might have severe problems with functionalities and user experience.

The time-on-task metric gives us insight into how quickly a user can complete tasks, allowing us to compare this time to the assumed number. If the user takes more time to finish a specific task, it's a sign that we should look at the activity closer. The problem might be caused by a confusing interface design or misinterpretation of instructions. It can also be an issue involving learnability. Learnability describes how easy it is for new users to learn how to operate an app or navigate a website. Of course, this factor will differ among users who had previous experience with similar products and testers who first encountered such products.

The user satisfaction score measures the level of satisfaction users feel after interacting with a product. It's usually expressed in percentages and uses the Likert scale. The user satisfaction score lets us see how users evaluate our product and how likely they are to recommend it.

The misclick rate calculates the number of clicks that occurred outside the interaction area. This metric enables us to see which elements look clickable from the users' perspectives. It also shows us how intuitive a website or application is.

The system usability scale measures the usability of websites and applications. It consists of ten questions that evaluate how the product is evaluated and perceived and how easy it is to use. The system usability scale is often used after the user finishes completing tasks so that they can evaluate the system as a whole.

Write a usability testing script

Now that we have goals, methods, and metrics in mind, it's time to write a usability testing script. The testing script includes the planned questions, tasks, and instructions that will be given to participants at the beginning of the test. This list will help us stay on track and will ensure that no question will be forgotten.

What's also important is that a usability test shouldn't last too long. It's recommended to conduct tests that last between 15 and 20 minutes. The longer the test, the more likely the participants will lose focus.

Recruit research participants

As mentioned, it's important to recruit representative users who will reflect the characteristics of our target audience. There are a couple of ways to acquire testers.

The first method is to use our existing user base and invite them to participate in usability testing. We should keep in mind that these types of users will already possess some level of familiarity with the site or application, which may reduce the number of detected usability problems. Nevertheless, it's one of the easier ways of recruitment.

The next method involves using a pool of research participants offered by many usability testing tools available on the market. Many platforms provide databases with users of varying demographics, allowing us to select the most suitable testers.

The last method is probably the most cost- and time-consuming: recruiting participants by ourselves. We can do that by posting advertisements on social media, putting notices on our website, or placing them inside an application. When recruiting in this way, we should provide potential participants with screener questions to make sure that they fit with our targeted users.

Conduct user testing

When performing in-person usability testing, we should ensure the participants know how the session will unfold and what to expect. Then, we should proceed with the session and start testing.

If the facilitator is present, they should take notes during the study. Usually, this task is performed by an assistant researcher so that the moderator can entirely focus on asking questions to the participant. In the case of remote and unmoderated research, note-taking can take place after the session and during the watch of the recording.

During the testing, facilitators should be careful not to ask leading questions or suggest responses.

Analyze the result of usability testing

When we receive the results of our usability test, it's time to analyze them and turn them into actions. Based on the usability testing findings, we will be able to improve the user experience in a way that benefits the users.

Moreover, if we organize these findings into a research report, we can effectively communicate them to our team members and other interested stakeholders.

Repeat the research

Usability testing is an iterative process that must be repeated to achieve improved results. We can test the product at different stages of the development process to continuously uncover issues. We can also test different variants of a design to see which one users prefer.

Pitfalls of usability testing

During the course of usability testing, we need to be careful not to make common mistakes. This will ensure the quality of testing and its findings.

Recruiting participants who don't match the target audience

As mentioned, it's crucial for effective usability testing to recruit participants who match the actual users of a website, application, or service. Otherwise, we won't gain insights, leading to effective improvements and wasting resources. Usability testing participants are meant to reflect our actual user base. They can't be a randomly selected group because their behavior patterns will differ from those of targeted users. Therefore, creating user personas to facilitate the testing and recruitment process is helpful.

A small number of research participants

We can perform usability testing with only five users; however, this number is the minimum necessary to gather sufficient data. We may find some issues with, for example, only three users, but it won't be enough to identify most of them. This may result in an impression that the design needs little improvement, whereas, in reality, it may contain some crucial functionality problems that the three testers won't discover. In general, it's recommended to conduct usability testing with 5 to 10 participants to uncover significant problems with user experience and interface design.

Leading questions and excessive intervening

Asking leading questions during usability testing may affect the tester's ability to answer them according to their own opinions and feelings. We should make sure that the questions don't imply the solution to tasks or the right answers. This should be taken care of during the writing of the research script. Facilitators should remain neutral and careful during the study and not allow their own biases to emerge. Tasks and questions are a vital part of usability testing; that's why they require special care and should be carefully considered. The answers to them will determine what kinds of findings we will acquire.

Excessive interrupting of the course of the study also affects the natural user behavior. The facilitator should take time to explain how the study will proceed and answer questions that the participant might have before the actual test begins. During the usability testing, moderators should only intervene if the participants encounter a problem during the test and can't solve it by themselves. In such a case, the researcher should help them get back on track and continue the study. Apart from this situation, the facilitator should remain silent, observe, and allow the tester to complete the study unassisted.

Neglecting user behavior during the testing

A common mistake in usability testing is ignoring user behavior, such as facial expression and body language, during the study. These observations can tell us a lot about the inner feelings of users that they won't necessarily express verbally. When performing in-person usability testing, researchers can observe these reactions to the product in real-time. In the case of remote testing, we need to review session recordings, although depending on the tools used, the quality of the material may vary, and it may be harder to spot nonverbal reactions.

The cost of usability testing

A few factors determine the cost of conducting usability testing. Considering them at the planning stage will help us optimize the use of resources and save time. A couple of factors significantly impact the final cost of usability testing. These include time, the cost of renting space for testing, the cost of participant recruitment, and compensation for testers.

Time

The longer the study, the more costs it generates. As mentioned, the time per session with one user should range between 15 and 20 minutes. This time needs to be multiplied by the amount determined by our sample size.

If we test only five users, the overall time of usability testing will range between 75 minutes (1 hour and 15 minutes) and 100 minutes (1 hour and 40 minutes). This time will increase proportionally to the number of participants. The increased number of testers will result in an increased number of facilitators and their assistants (in the case of moderated and in-person usability testing). Of course, we can use one facilitator and assistant for the testing, but the sample will need to be small.

In addition to the session time, we need to factor in the time for the preparation phase and analysis of the results, as well as any repeated testing.

Cost of renting a space

In the case of in-person usability testing, we need to add the costs of renting a space for testing. This space can be a simple conference room or a dedicated lab. We should also consider the cost of renting the equipment if necessary.

Cost of recruitment

The cost of recruiting participants also depends on various factors. First, we need to consider how we will recruit them. If we opt to use a usability testing platform that provides testers, then the cost may be included in the price of the tool. If we decide to recruit participants on our own, then we need to factor in the costs of advertisements or email campaigns that will notify potential testers about the planned usability testing.

Compensation for research participants

Compensation for research participants is a compelling incentive to encourage potential testers to participate in usability testing and, if applicable, reimburse them for travel expenses. We can offer monetary compensation, gift cards, vouchers, or gift baskets. It's also a good idea to adapt the type of compensation to research participants' preferences; some cultural aspects should also be considered beforehand.