Quantitative Research answers the questions: how much, how many, and allows you to establish in a specific and measurable manner the size, extent, scope, frequency, and intensity of a given phenomenon.
Quantitative and Qualitative Research (e.g., focus groups, expert audits, and heuristic analysis) should be a permanent element of every digital product design.
The use of the potential of both types of research allows you to learn about the needs of end users (e.g., customers in an online store) in a more profound, multidimensional, and reliable way.
It allows you to create solutions that are adequate to the needs of end users.
It enables you to adjust the design of the website or mobile application to their specific and discovered, through research, preferences.
In practice, Quantitative and Qualitative Research should be treated as complementary and not opposing approaches.
What is the basic difference between Quantitative and Qualitative Research? How do these research methods differ?
Qualitative Research provides answers to the following questions: Why? In what way? When? Where?
In turn, Quantitative Research provides answers to How frequently? How much?
It may seem that for User Experience problems, Qualitative Research is more appropriate.
The quality of experiences and problems with usability refers to the impressions, opinions, emotions, attitudes, biases, and behavioral issues that are much more easily captured by this method.
So is Quantitative Research even useful regarding UX/UI design and offering a positive user experience?
What value can Quantitative Research bring to research, design, and optimization processes? How can you use Quantitative methods in UX?
Today we'll introduce you to the issue of Quantitative Research in UX. It's worth familiarizing yourself with it because it's an essential supplement to Qualitative Research.
It allows you to obtain a more complex, multidimensional, and complete picture of problems with usability. It gives you insight into the scope of these problems and their intensity.
As always, we cordially invite you to read the article!
Do you want to perform UX Research?
Quantitative Research — what is it?
Quantitative Research admittedly doesn't answer why a given phenomenon occurs, its causes, or what conditions it, but it shows for how many users a given problem is real.
Therefore it allows you to estimate the scale of the phenomenon.
What is Qualitative Research? For example, Qualitative Research can explain why a contact form achieves a low conversion rate.
The reason might be too many fields or incomprehensible labels.
In turn, Quantitative Research determines how many people consider the number of form fields as problematic.
It also enables you to identify the change that occurs during the optimization process when the number of fields is reduced.
By repeating Quantitative Research regarding this issue, you will discover if the number of users facing this problem increased or decreased.
Objective results will replace assumptions and unrealistic results.
Only the combined results of Qualitative and Quantitative Research (which, unfortunately, are not achievable in the short term) will enable you to make more rational and justified decisions regarding optimization.
The most important types of Quantitative Research in UX include the following:
- Benchmarking
- Web analytics
- A/B tests in UX
- Card sorting
- Surveys
- Eye-tracking.
Every UX research method is characterized by its own specificity in the following aspects:
- Scope of application
- Cost
- Difficulty in acquiring data and recruiting respondents
- Problems involved in analytical and interpretative work
- Context and conditions in which the study is performed
- Research utility and ability to compile results with other methods, types of research
- Research goals
- Time required to perform the study
- Exploratory value.
For example, card sorting is a method that gives you insight into the mental models of the website's users.
In particular, it allows you to see how they categorize and name phenomena.
Card sorting allows you to learn the terminology used by users to describe and group phenomena and determine what a product is.
This method determines the most optimal wording and forms of labels and information and architecture structures used in digital products.
It is very affordable, and at the same time, the data collection is uncomplicated. Card sorting doesn't cause many problems regarding interpretation.
Currently, it's possible to conduct this type of research on users in stationary conditions and remotely through a dedicated tool.
Card Sorting is sometimes called an indirect method because it gives you both quantitative and qualitative results.
It allows you to learn the names, terminologies, and categories and will enable you to determine the frequency of their use.
It's extremely useful in designing information architecture and, above all, creating categories and names in menus that will better suit users' mental models.
Major advantages of Quantitative Research
Naturally, the advantages of Quantitative Research are much broader and diverse.
By performing Quantitative Research, you gain insight into a given phenomenon in the following ways:
- Objective, through tools and tasks that are standardized
- Possible to compare results over time
- Statistically relevant, hence the credibility of this research is high.
In UX Research, Quantitative Research usually means performing Quantitative Usability Tests. Quantitative Usability Testing involves performing tasks defined by researchers on a website.
Adopted by the researcher UX Metrics allow them to express the efficiency of task performance in numerical form. A metric or UX indicator can be time or amount.
Usually, respondents are asked to do various tasks on the website (e.g., to find a product card or use a contact form).
The UX metrics that allow you to indirectly evaluate usability most often include the following:
- Time of performing a task
- Number of completed tasks
- Number of made mistakes during a task
- Quantified user satisfaction ratings for websites or mobile apps.
Admittedly, metrics won't indicate the causes. Still, they help you check or confront usability with competitive solutions (or previous designs) and thus determine the increase or decrease of the attractiveness for users.
As we've already mentioned, a limitation of Quantitative Research is the inability to discover the causes of quantitative results.
The conclusion you will draw will only relate to the frequency of the behavior.
Quantitative Research results don't determine or define the problems users encounter. They don't suggest solutions.
This kind of insight and knowledge can be gained from Qualitative Research.
Furthermore, Quantitative Research requires more respondents, which of course, raises the cost, increases the time needed to perform it, and raises organizational issues.
It also poses challenges for UX researchers to statistically process and interpret the results.
Nevertheless, an advantage that can't be underestimated is the far greater objectivity of this kind of research.
The randomness and low credibility of results are reduced by well-proven techniques for performing research and a whole background of mathematical and statistical knowledge that helps guarantee the expected credibility.
It's also worth remembering remarks made by Kate Moran in the article "Quantitative User-Research Methodologies: An Overview."
Researcher associated with Nielsen Norman Group observes, very aptly, that Quantitative Research should be an inseparable element of the corpus of tools of every UX researcher.
The justification of this belief, approach sounds very compelling.
According to Kate Moran, Quantitative Research enables you to:
- Obtain data that are significant in the decision-making process and make it possible to convince various stakeholders in a more concrete way
- Compare, estimate, evaluate, and rank
- Calculate and estimate the ROI level, which is especially important in optimization work.
Quantitative Research also measures the following:
- Effectiveness and efficiency of the digital product — most often, an indicator of high efficiency is the short time required to complete a task because time correlates with ease and intuitiveness of use.
- User satisfaction during the product use — it's measured just after performing a task, e.g., through a Subjective Mental Effort Questionnaire (SMEQ).
- Effectiveness (accuracy and completeness) — indicated by success rate and the number of mistakes made during a task.
Other differences between Qualitative and Quantitative Research
In the most general sense, the main difference between Qualitative and Quantitative Research is its scope and depth.
Quantitative Research, compared to Qualitative Research, provides the following:
- Larger scope
- But lower depth.
Moreover, both types of research pose other challenges to researchers in terms of the following:
- Size of the sample
- Recruitment
- Organization or performance of research
- Research tools
- Methods of analyzing and interpreting
- Time required to conduct it
- Moment — the performance of Quantitative Research is recommended at later stages of the design process because, with their help, the performance of a website or mobile application is tested.
The reasoning behind doing Quantitative Research at later stages of the design is also its cost.
Usually, for performing Quantitative Research, you need at least several dozen respondents, while for conducting Qualitative Research, you only need a few (at least 5).
It's also worth remembering, as Raluca Budiu, the author associated with NN Group, states in the article "Quantitative vs. Qualitative Usability Testing," that Qualitative Research results usually allow you to:
- Gain insight into the strong and weak sides of the design
- Prioritize problems.
And simultaneously, they:
- Depend on the knowledge, experience, and reliability of a researcher
- Are susceptible to the evaluation effect — various research will identify different sets (sometimes overlapping) of problems with usability.
Raluca Budiu also writes that both types of research must provide the following:
- External validity — respondents need to be a representative group, and the course of the research should be as close to the natural conditions in which the tasks are performed as possible.
- Internal validity — research should be conducted in precisely the same conditions, circumstances, and contexts to avoid the influence of unknown and uncontrollable variables.
How to conduct Quantitative Research on User Experience?
The choice of the research method is naturally crucial.
Nevertheless, if you want to find out how frequently a given problem occurs, you need to, above all, define what problem you have in mind.
Focusing the study on a specific research goal enables you to obtain much more reliable results.
When you know what you want to study, you can proceed to the next step, in which you have to match the adequate metrics to the research problem that will be able to capture the scale of the issue.
Standard indicators, such as the average time needed to finish a task or the number of mistakes, can be combined with more study-specific ones.
Determining indicators can be difficult, especially for researchers with little experience.
A tool, namely Google's framework — Heart — recommended in the article "Google's HEART Framework for Measuring UX" written by Interaction Design Foundation may prove helpful.
Quantitative Research, as well as Qualitative Research, can be performed stationary or remotely or be moderated or unmoderated.
The choice of a particular configuration depends on the following:
- Research problem — its complexity and difficulty
- Budget
- Time scheduled for performing a task
- Experience and competencies of researchers
- Available research tools.
In the next step, it's necessary to create a research scenario, a detailed plan, in which you should define tasks, goals, tools, and contexts.
With a ready UX research scenario, you can proceed to the phase of recruiting respondents according to the adopted criteria.
In particular, you should take care of the representativeness of participants and the size of the sample — usually, it shouldn't be smaller than 30 respondents.
The last phases include: performing the research and preparing results in a statistical sense.
Remember that recruitment in Quantitative Research is one of the biggest challenges UX researchers face due to the sample size.
UX Research requires thorough preparation. The choice of the research method is the clue of the researcher's work, but the most challenging task is the selection of the sample. It's problematic regarding representativeness, gratification, and availability of respondents within the scheduled period.
In any research project, you should bet on optimal solutions and match the research method to the research problem, but you shouldn't exclude different methods if you have a limited budget and time.
It's also important to remember that while conducting Quantitative Research on UX, the most important, according to the article "Usability Testing," factors that condition the size of the budget include the following:
- Size of the research team
- Number of respondents
- Number of days necessary to study all of the participants
- Costs of recruitment (e.g., costs of the specialized companies that recruit respondents according to imposed criteria)
- Remuneration for participants
- Costs of a laboratory and/or dedicated access to research platforms.
Quantitative Research in UX. Summary
- UX Research, Quantitative and Qualitative Research should be performed simultaneously during the design process or creating websites or mobile applications.
- UX Research (Qualitative and Quantitative methods) should be treated as complementary approaches.
- UX Research, especially Quantitative Research, allows you to determine the phenomenon's size, scope, frequency, and intensity.
- The combined results of Qualitative and Quantitative Research provide the most reliable, credible, and exhaustive approach to making rational design decisions.
- The essential types of Quantitative Research include: benchmarking, A/B tests, card sorting, surveys, and eye-tracking.
- Performing Quantitative Research gives you insight into a phenomenon in an objective, comparable, statistically relevant way.
- Quantitative Research most often involves conducting Quantitative Usability Tests in practice.
- User Experience Research, Quantitative Usability Tests consists of performing defined by researchers tasks on a website or in a mobile application.
- Quantitative Research results don't determine or define the problems users encounter.
- Using mathematical and statistical knowledge reduces randomness and low credibility of results.
- Quantitative methods allow you to compare, estimate, evaluate, rank, and calculate the level of ROI.
- Quantitative Research, compared to Qualitative Research, provides a larger scope but lower depth.
