How you like that:

Quantitative research answers the questions: how much, how many, and allows you to establish in a specific and measurable manner the size, extent, scope, frequency, and intensity of a given phenomenon.

Quantitative and qualitative research (e.g., focus groups, expert audits, and heuristic analysis) should be a permanent element of every digital product design.

The use of the potential of both types of research allows you to learn about the needs of end users (e.g., customers in an online store) in a more profound, multidimensional, and reliable way.

It allows you to create solutions that are adequate to the needs of end users.

It enables you to adjust the design of a website or mobile application to their specific and discovered, through research, preferences.

Quantitative and qualitative research should be treated as complementary approaches and not as opposing ones.

What is the fundamental difference between quantitative and qualitative research? How do these user research methods differ?

Qualitative data provides answers to the following questions: Why? In what way? When? Where?

In turn, quantitative data provides answers to questions: How frequently? How much?

It may seem that qualitative research is more appropriate for user experience problems.

The quality of experiences and problems with usability refers to the impressions, opinions, emotions, attitudes, biases, and behavioral issues that are much more easily captured by this method.

So, is quantitative research even useful in UX/UI design and offering a positive user experience?

What value can quantitative user research bring to research, design, and optimization processes? How can you use quantitative methods in UX? What are the examples of qualitative and quantitative research methods?

Today, we'll introduce you to the issue of quantitative UX research. It's worth familiarizing yourself with it because it's an essential supplement to qualitative research.

It allows you to obtain a more complex, multidimensional, and complete picture of usability problems. It gives you insight into their scope and intensity.

As always, we cordially invite you to read the article!

Do you want to perform UX Research?

DISCOVER OUR SERVICE

Quantitative research — what is it?

Admittedly, quantitative research doesn't answer why a given phenomenon occurs, its causes, or under what conditions it happens, but it shows for how many users a given problem is real. It provides numerical data and insight into user behavior.

Therefore it allows you to estimate the scale of a phenomenon.

Quantitative research vs. Qualitative research

What is qualitative research? For example, qualitative research can explain why a contact form has a low conversion rate.

The reason might be too many fields or incomprehensible labels.

In turn, quantitative research determines how many people consider the number of form fields problematic.

It also enables you to identify the change during the optimization process when the number of fields is reduced.

By repeating quantitative research regarding this issue, you will discover if the number of users facing this problem has increased or decreased.

Objective results will replace assumptions and unrealistic results.

Did You Know...

Only the combined results of qualitative and quantitative research (which, unfortunately, are not achievable in the short term) will enable you to make more rational and justified decisions regarding optimization.

The most essential quantitative UX research methods include the following:

Benchmarking
Web analytics
A/B tests in UX
Card sorting
Surveys
Eye-tracking

In turn, qualitative methods of UX research include activities such as user interviews, focus groups, five-second testing, and diary studies.

Qualitative research methods are useful when you need to understand the emotions, behaviors, and attitudes of a digital product user more deeply.

Thanks to these research methods, UX/UI designers can improve the user experience of a digital product and help build applications that meet user needs.

Every UX research method is characterized by its own specificity in the following aspects:

Scope of application
Cost
Difficulty in acquiring data and recruiting respondents
Problems involved in analytical and interpretative work
Context and conditions in which the study is performed
Research utility and ability to compile results with other methods, types of research
Research goals
Time required to perform a study
Exploratory value

For example, card sorting is a method that gives you insight into the mental models of the website's users.

In particular, it allows you to see how they categorize and name phenomena.

Card sorting allows you to learn the terminology users use to describe and group phenomena and determine what a product is.

This method determines the most optimal wording and forms of labels and information and architecture structures used in digital products.

It is very affordable, and data collection is uncomplicated. Card sorting doesn't cause many problems regarding interpretation.

Currently, it's possible to conduct this type of research on users in stationary conditions and remotely through a dedicated tool.

Card Sorting is sometimes called an indirect method because it gives you both quantitative and qualitative results.

It allows you to learn the names, terminologies, and categories and will enable you to determine the frequency of their use.

It's extremely useful in designing information architecture and, above all, creating categories and names in menus that will better suit users' mental models.

Significant advantages of quantitative research

Naturally, the advantages of quantitative research are much broader and diverse.

By performing quantitative research, you gain insight into a given phenomenon in the following ways:

Objective, through tools and tasks that are standardized
Possible to compare results over time
Statistically relevant, hence the credibility of this research is high

Did You Know...

In UX research, quantitative research usually means performing quantitative usability tests. Quantitative usability testing involves performing tasks defined by researchers on a website.

Adopted by a researcher, UX metrics allow them to express the efficiency of task performance in numerical form. A metric or UX indicator can be time or amount.

Respondents are usually asked to perform various tasks on a website (e.g., find a product page, or use a contact form).

The UX metrics that allow you to indirectly evaluate usability most often include the following:

Time of performing a task
Number of completed tasks
Number of made mistakes during a task
Quantified user satisfaction ratings for websites or mobile apps

Admittedly, metrics won't indicate the causes. Still, they help you check or confront usability with competitive solutions (or previous designs) and thus determine whether the attractiveness for users increases or decreases.

As we've already mentioned, a limitation of quantitative research is the inability to discover the causes of quantitative results.

The conclusion you will draw will only relate to the frequency of the behavior.

Quantitative research results don't determine or define the problems users encounter. They don't suggest solutions.

Quantitative research examples

This kind of insight and knowledge can be gained from qualitative user research.

Furthermore, quantitative research requires more respondents, which, of course, raises the cost, increases the time needed to perform it, and raises organizational issues.

It also poses challenges for UX researchers to statistically process and interpret the results.

Nevertheless, an advantage that can't be underestimated is the far greater objectivity of this kind of research.

Did You Know...

The randomness and low credibility of results are reduced by well-proven techniques for performing research and a whole background of mathematical and statistical knowledge that helps guarantee the expected credibility.

It's also worth remembering Kate Moran's remarks in the article "Quantitative User-Research Methodologies: An Overview."

The researcher associated with Nielsen Norman Group observes very aptly that quantitative research should be an inseparable element of every UX researcher's toolkit.

The justification for this belief and approach sounds very compelling.

According to Kate Moran, quantitative research enables you to:

Obtain data that are significant in the decision-making process and make it possible to convince various stakeholders in a more concrete way
Compare, estimate, evaluate, and rank
Calculate and estimate the ROI level, which is especially important in optimization work.

Quantitative research also measures the following:

Effectiveness and efficiency of a digital product — most often, an indicator of high efficiency is the short time required to complete a task because time correlates with ease and intuitiveness of use.
User satisfaction during the product use — it's measured just after performing a task, e.g., through a Subjective Mental Effort Questionnaire (SMEQ).
Effectiveness (accuracy and completeness) — indicated by success rate and the number of mistakes made during a task.

Disadvantages of quantitative research

No user research method is perfect, and quantitative UX research is no exception. While it has many advantages and helpful uses, it also has some drawbacks.

As mentioned, quantitative data can tell you where the problem is and how common it is, but it won't reveal its cause or provide a solution.

This type of research method can also be tainted by confirmation bias, which occurs when researchers tend to interpret the data they receive according to their beliefs instead of results. Therefore, it's important to be wary of this.

Quantitative data lacks context because research participants also lack it. Respondents can't justify their responses or ask clarifying questions.

During the analysis of obtained data, researchers need to make a lot of assumptions about the ways users think and what prompted them to give a certain response. This issue ties in with the above-mentioned absence of context.

We've also already said that quantitative UX research requires a large sample size to achieve statistical significance. Because of that, it's very costly and cumbersome to organize and perform.

UX researchers must possess strong analytical skills to ensure that data are interpreted correctly and unbiasedly. An insufficient amount of knowledge can negatively impact the accuracy of the results.

When to perform quantitative UX research?

You can use quantitative research to:

Evaluate and analyze key performance indicators
Find new ideas
Compare your products with the competition
Determine which design variant is better
Understand customer needs
Discover design problems.

Other differences between qualitative and quantitative research

In the most general sense, the main difference between qualitative and quantitative research is its scope and depth.

Qualitative research vs. Quantitative research - differences and similarities

Quantitative research, compared to qualitative research, provides the following:

Larger scope
But lower depth.

Moreover, both types of research pose other challenges to researchers in terms of the following:

Size of the sample
Recruitment
Organization or performance of research
Research tools
Methods of analyzing and interpreting
Time required to conduct it
Moment — the performance of quantitative research is recommended at later stages of the design process because, with its help, the performance of a website or mobile application is tested

The reason behind doing quantitative research at later stages of the design is also its cost.

Did You Know...

Usually, to ensure statistical significance in quantitative research, you need at least several dozen respondents, while for qualitative research, you only need a few (at least 5).

It's also worth remembering, as Raluca Budiu, the author associated with NN Group, states in the article "Quantitative vs. Qualitative Usability Testing," that qualitative research results usually allow you to:

Gain insight into the strong and weak sides of the design
Prioritize problems.

And simultaneously, they:

Depend on the knowledge, experience, and reliability of a researcher
Are susceptible to the evaluation effect — various research will identify different sets (sometimes overlapping) of problems with usability.

Raluca Budiu also writes that both types of research must provide the following:

External validity — respondents need to be a representative group, and the course of the research should be as close to the natural conditions in which the tasks are performed as possible.
Internal validity — research should be conducted in precisely the same conditions, circumstances, and contexts to avoid the influence of unknown and uncontrollable variables.

How to conduct quantitative research on user experience?

The choice of the research method is naturally crucial.

Nevertheless, if you want to find out how frequently a given problem occurs, you need to define what problem you have in mind.

Focusing the study on a specific research goal enables you to obtain much more reliable results.

When you know what you want to study, you can proceed to the next step, in which you have to match the adequate metrics to the research problem that will be able to capture the scale of the issue.

Standard indicators, such as the average time needed to finish a task or the number of mistakes, can be combined with more study-specific ones.

Determining indicators can be difficult, especially for researchers with little experience.

A tool, namely Google's framework — Heart — recommended in the article "Google's HEART Framework for Measuring UX" written by Interaction Design Foundation may prove helpful.

Quantitative research, as well as qualitative research, can be performed stationary or remotely or be moderated or unmoderated.

The choice of a particular configuration depends on the following:

Research problem — its complexity and difficulty
Budget
Time scheduled for performing a task
Experience and competencies of researchers
Available research tools

In the next step, it's necessary to create a research scenario, a detailed plan, in which you should define tasks, goals, tools, and contexts.

With a ready UX research scenario, you can proceed to the phase of recruiting respondents according to the adopted criteria.

In particular, you should consider the representativeness of participants and the size of the sample—usually, it shouldn't be smaller than 30 respondents.

The last phases include performing the research and preparing results in a statistical sense.

Remember that recruitment in quantitative research is one of the biggest challenges UX researchers face due to the sample size.

Did You Know...

UX research requires thorough preparation. The choice of a research method is the core of the researcher's work, but the most challenging task is the selection of the sample. It's problematic regarding representativeness, gratification, and availability of respondents within the scheduled period.

In any research project, you should bet on optimal solutions and match the research method to the research problem, but you shouldn't exclude different methods if you have a limited budget and time.

Quantitative research vs. qualitative research UX

It's also important to remember that while conducting quantitative research on UX, the most important, according to the article "Usability Testing," factors that condition the size of the budget include the following:

The size of a research team
Number of respondents
Number of days necessary to study all of the participants
Costs of recruitment (e.g., costs of the specialized companies that recruit respondents according to imposed criteria)
Remuneration for participants
Costs of a laboratory and/or dedicated access to research platforms

How to recruit participants for quantitative research?

Since quantitative UX research requires a large sample, this type of research method is challenging to organize. To maintain the reliability and accuracy of results, you can't just pick random respondents. So, how to choose a suitable sample?

First, you need to choose a large, general "population," such as users of mobile applications. Then, you need to pick a group called a sample from among them that will match your research criteria. For example, one such criterion can be "users of mobile applications between the ages of 18-25."

Remember that your requirements for research participants should be adequate to the research goals you want to achieve. Additionally, your sampling method should generate a random sample to obtain relevant results.