Homepage > Journal > Quantitative research in UX
Journal

Quantitative research in UX

How you like that:

Quantitative research answers the questions: how much, how many, and allows you to establish in a specific and measurable manner the size, extent, scope, frequency, and intensity of a given phenomenon.

Quantitative and qualitative research (e.g., focus groups, expert audits, and heuristic analysis) should be a permanent element of every digital product design.

The use of the potential of both types of research allows you to learn about the needs of end users (e.g., customers in an online store) in a more profound, multidimensional, and reliable way.

It allows you to create solutions that are adequate to the needs of end users.

It enables you to adjust the design of a website or mobile application to their specific and discovered, through research, preferences.

Quantitative and qualitative research should be treated as complementary approaches and not as opposing ones.

What is the fundamental difference between quantitative and qualitative research? How do these user research methods differ?

Qualitative data provides answers to the following questions: Why? In what way? When? Where?

In turn, quantitative data provides answers to questions: How frequently? How much?

It may seem that qualitative research is more appropriate for user experience problems.

The quality of experiences and problems with usability refers to the impressions, opinions, emotions, attitudes, biases, and behavioral issues that are much more easily captured by this method.

So, is quantitative research even useful in UX/UI design and offering a positive user experience?

What value can quantitative user research bring to research, design, and optimization processes? How can you use quantitative methods in UX? What are the examples of qualitative and quantitative research methods?

Today, we'll introduce you to the issue of quantitative UX research. It's worth familiarizing yourself with it because it's an essential supplement to qualitative research.

It allows you to obtain a more complex, multidimensional, and complete picture of usability problems. It gives you insight into their scope and intensity.

As always, we cordially invite you to read the article!

Do you want to perform UX Research?

Quantitative research — what is it?

Admittedly, quantitative research doesn't answer why a given phenomenon occurs, its causes, or under what conditions it happens, but it shows for how many users a given problem is real. It provides numerical data and insight into user behavior.

Therefore it allows you to estimate the scale of a phenomenon.

Quantitative research vs. Qualitative research

What is qualitative research? For example, qualitative research can explain why a contact form has a low conversion rate.

The reason might be too many fields or incomprehensible labels.

In turn, quantitative research determines how many people consider the number of form fields problematic.

It also enables you to identify the change during the optimization process when the number of fields is reduced.

By repeating quantitative research regarding this issue, you will discover if the number of users facing this problem has increased or decreased.

Objective results will replace assumptions and unrealistic results.

Did You Know...

Only the combined results of qualitative and quantitative research (which, unfortunately, are not achievable in the short term) will enable you to make more rational and justified decisions regarding optimization.

The most essential quantitative UX research methods include the following:

In turn, qualitative methods of UX research include activities such as user interviews, focus groups, five-second testing, and diary studies.

Qualitative research methods are useful when you need to understand the emotions, behaviors, and attitudes of a digital product user more deeply.

Thanks to these research methods, UX/UI designers can improve the user experience of a digital product and help build applications that meet user needs.

Every UX research method is characterized by its own specificity in the following aspects:

  • Scope of application
  • Cost
  • Difficulty in acquiring data and recruiting respondents
  • Problems involved in analytical and interpretative work
  • Context and conditions in which the study is performed
  • Research utility and ability to compile results with other methods, types of research
  • Research goals
  • Time required to perform a study
  • Exploratory value

For example, card sorting is a method that gives you insight into the mental models of the website's users.

In particular, it allows you to see how they categorize and name phenomena.

Card sorting allows you to learn the terminology users use to describe and group phenomena and determine what a product is.

This method determines the most optimal wording and forms of labels and information and architecture structures used in digital products.

It is very affordable, and data collection is uncomplicated. Card sorting doesn't cause many problems regarding interpretation.

Currently, it's possible to conduct this type of research on users in stationary conditions and remotely through a dedicated tool.

Card Sorting is sometimes called an indirect method because it gives you both quantitative and qualitative results.

It allows you to learn the names, terminologies, and categories and will enable you to determine the frequency of their use.

It's extremely useful in designing information architecture and, above all, creating categories and names in menus that will better suit users' mental models.

Significant advantages of quantitative research

Naturally, the advantages of quantitative research are much broader and diverse.

By performing quantitative research, you gain insight into a given phenomenon in the following ways:

  • Objective, through tools and tasks that are standardized
  • Possible to compare results over time
  • Statistically relevant, hence the credibility of this research is high
Did You Know...

In UX research, quantitative research usually means performing quantitative usability tests. Quantitative usability testing involves performing tasks defined by researchers on a website.

Adopted by a researcher, UX metrics allow them to express the efficiency of task performance in numerical form. A metric or UX indicator can be time or amount.

Respondents are usually asked to perform various tasks on a website (e.g., find a product page, or use a contact form).

The UX metrics that allow you to indirectly evaluate usability most often include the following:

  • Time of performing a task
  • Number of completed tasks
  • Number of made mistakes during a task
  • Quantified user satisfaction ratings for websites or mobile apps

Admittedly, metrics won't indicate the causes. Still, they help you check or confront usability with competitive solutions (or previous designs) and thus determine whether the attractiveness for users increases or decreases.

As we've already mentioned, a limitation of quantitative research is the inability to discover the causes of quantitative results.

The conclusion you will draw will only relate to the frequency of the behavior.

Quantitative research results don't determine or define the problems users encounter. They don't suggest solutions.

Quantitative research examples

This kind of insight and knowledge can be gained from qualitative user research.

Furthermore, quantitative research requires more respondents, which, of course, raises the cost, increases the time needed to perform it, and raises organizational issues.

It also poses challenges for UX researchers to statistically process and interpret the results.

Nevertheless, an advantage that can't be underestimated is the far greater objectivity of this kind of research.

Did You Know...

The randomness and low credibility of results are reduced by well-proven techniques for performing research and a whole background of mathematical and statistical knowledge that helps guarantee the expected credibility.

It's also worth remembering Kate Moran's remarks in the article "Quantitative User-Research Methodologies: An Overview."

The researcher associated with Nielsen Norman Group observes very aptly that quantitative research should be an inseparable element of every UX researcher's toolkit.

The justification for this belief and approach sounds very compelling.

According to Kate Moran, quantitative research enables you to:

  • Obtain data that are significant in the decision-making process and make it possible to convince various stakeholders in a more concrete way
  • Compare, estimate, evaluate, and rank
  • Calculate and estimate the ROI level, which is especially important in optimization work.

Quantitative research also measures the following:

  • Effectiveness and efficiency of a digital product — most often, an indicator of high efficiency is the short time required to complete a task because time correlates with ease and intuitiveness of use.
  • User satisfaction during the product use — it's measured just after performing a task, e.g., through a Subjective Mental Effort Questionnaire (SMEQ).
  • Effectiveness (accuracy and completeness) — indicated by success rate and the number of mistakes made during a task.

Disadvantages of quantitative research

No user research method is perfect, and quantitative UX research is no exception. While it has many advantages and helpful uses, it also has some drawbacks.

As mentioned, quantitative data can tell you where the problem is and how common it is, but it won't reveal its cause or provide a solution.

This type of research method can also be tainted by confirmation bias, which occurs when researchers tend to interpret the data they receive according to their beliefs instead of results. Therefore, it's important to be wary of this.

Quantitative data lacks context because research participants also lack it. Respondents can't justify their responses or ask clarifying questions.

During the analysis of obtained data, researchers need to make a lot of assumptions about the ways users think and what prompted them to give a certain response. This issue ties in with the above-mentioned absence of context.

We've also already said that quantitative UX research requires a large sample size to achieve statistical significance. Because of that, it's very costly and cumbersome to organize and perform.

UX researchers must possess strong analytical skills to ensure that data are interpreted correctly and unbiasedly. An insufficient amount of knowledge can negatively impact the accuracy of the results.

When to perform quantitative UX research?

You can use quantitative research to:

  • Evaluate and analyze key performance indicators
  • Find new ideas
  • Compare your products with the competition
  • Determine which design variant is better
  • Understand customer needs
  • Discover design problems.

Other differences between qualitative and quantitative research

In the most general sense, the main difference between qualitative and quantitative research is its scope and depth.

Qualitative research vs. Quantitative research - differences and similarities

Quantitative research, compared to qualitative research, provides the following:

  • Larger scope
  • But lower depth.

Moreover, both types of research pose other challenges to researchers in terms of the following:

  • Size of the sample
  • Recruitment
  • Organization or performance of research
  • Research tools
  • Methods of analyzing and interpreting
  • Time required to conduct it
  • Moment — the performance of quantitative research is recommended at later stages of the design process because, with its help, the performance of a website or mobile application is tested

The reason behind doing quantitative research at later stages of the design is also its cost.

Did You Know...

Usually, to ensure statistical significance in quantitative research, you need at least several dozen respondents, while for qualitative research, you only need a few (at least 5).

It's also worth remembering, as Raluca Budiu, the author associated with NN Group, states in the article "Quantitative vs. Qualitative Usability Testing," that qualitative research results usually allow you to:

  • Gain insight into the strong and weak sides of the design
  • Prioritize problems.

And simultaneously, they:

  • Depend on the knowledge, experience, and reliability of a researcher
  • Are susceptible to the evaluation effect — various research will identify different sets (sometimes overlapping) of problems with usability.

Raluca Budiu also writes that both types of research must provide the following:

  • External validity — respondents need to be a representative group, and the course of the research should be as close to the natural conditions in which the tasks are performed as possible.
  • Internal validity — research should be conducted in precisely the same conditions, circumstances, and contexts to avoid the influence of unknown and uncontrollable variables.

How to conduct quantitative research on user experience?

The choice of the research method is naturally crucial.

Nevertheless, if you want to find out how frequently a given problem occurs, you need to define what problem you have in mind.

Focusing the study on a specific research goal enables you to obtain much more reliable results.

When you know what you want to study, you can proceed to the next step, in which you have to match the adequate metrics to the research problem that will be able to capture the scale of the issue.

Standard indicators, such as the average time needed to finish a task or the number of mistakes, can be combined with more study-specific ones.

Quantitative user experience research

Determining indicators can be difficult, especially for researchers with little experience.

A tool, namely Google's framework — Heart — recommended in the article "Google's HEART Framework for Measuring UX" written by Interaction Design Foundation may prove helpful.

Quantitative research, as well as qualitative research, can be performed stationary or remotely or be moderated or unmoderated.

The choice of a particular configuration depends on the following:

  • Research problem — its complexity and difficulty
  • Budget
  • Time scheduled for performing a task
  • Experience and competencies of researchers
  • Available research tools

In the next step, it's necessary to create a research scenario, a detailed plan, in which you should define tasks, goals, tools, and contexts.

With a ready UX research scenario, you can proceed to the phase of recruiting respondents according to the adopted criteria.

In particular, you should consider the representativeness of participants and the size of the sample—usually, it shouldn't be smaller than 30 respondents.

The last phases include performing the research and preparing results in a statistical sense.

Remember that recruitment in quantitative research is one of the biggest challenges UX researchers face due to the sample size.

Did You Know...

UX research requires thorough preparation. The choice of a research method is the core of the researcher's work, but the most challenging task is the selection of the sample. It's problematic regarding representativeness, gratification, and availability of respondents within the scheduled period.

In any research project, you should bet on optimal solutions and match the research method to the research problem, but you shouldn't exclude different methods if you have a limited budget and time.

Quantitative research vs. qualitative research UX

It's also important to remember that while conducting quantitative research on UX, the most important, according to the article "Usability Testing," factors that condition the size of the budget include the following:

  • The size of a research team
  • Number of respondents
  • Number of days necessary to study all of the participants
  • Costs of recruitment (e.g., costs of the specialized companies that recruit respondents according to imposed criteria)
  • Remuneration for participants
  • Costs of a laboratory and/or dedicated access to research platforms

How to recruit participants for quantitative research?

Since quantitative UX research requires a large sample, this type of research method is challenging to organize. To maintain the reliability and accuracy of results, you can't just pick random respondents. So, how to choose a suitable sample?

First, you need to choose a large, general "population," such as users of mobile applications. Then, you need to pick a group called a sample from among them that will match your research criteria. For example, one such criterion can be "users of mobile applications between the ages of 18-25."

Remember that your requirements for research participants should be adequate to the research goals you want to achieve. Additionally, your sampling method should generate a random sample to obtain relevant results.

Quantitative research in UX. Summary

  1. UX research, quantitative and qualitative research should be performed simultaneously during the design process or when creating websites or mobile applications.
  2. UX research (qualitative and quantitative methods) should be treated as complementary approaches.
  3. UX research, especially quantitative UX research, allows you to determine the phenomenon's size, scope, frequency, and intensity.
  4. The combined results of qualitative and quantitative research provide the most reliable, credible, and exhaustive approach to making rational design decisions.
  5. Quantitative UX research methods include benchmarking, A/B tests, card sorting, surveys, and eye-tracking.
  6. Qualitative user research methods include user interviews, focus groups, five-second testing, and diary studies.
  7. Performing quantitative research gives you insight into a phenomenon in an objective, comparable, statistically relevant way.
  8. Quantitative research most often involves conducting quantitative usability tests in practice.
  9. User experience research, or quantitative usability tests, consists of performing tasks defined by researchers on a website or mobile application.
  10. Quantitative research results don't determine or define the problems users encounter.
  11. Using mathematical and statistical knowledge reduces randomness and low credibility of results.
  12. Quantitative methods allow you to compare, estimate, evaluate, rank, and calculate the level of ROI.
  13. Compared to qualitative research, quantitative research provides a larger scope but lower depth.
  14. Although quantitative UX research has many benefits, it also has some drawbacks that you should keep in mind.
How you like that:
Journal / Redaktor
Author: Radek
UX Writer and researcher by education + experience. Collects The Story's knowledge and shares it on the Journal.
Reviewer: Dymitr Romanowski

Are you interested in working with us? Take a look at our Portfolio