The validity, reliability, and credibility of qualitative research is an issue that is rarely raised in UX research discussions.
We presumed that since something is studied, then at the same time, the results of such research are indisputable, trustworthy, and non-negotiable.
The attitude toward research results can sometimes be too uncritical. Many people take an attitude towards them (qualitative and quantitative research) that can be summarized in a simple sentence: Data isn't debatable; data is obtained and used.
But there are two sides to every coin. The mere execution of research shouldn't justify its questionable quality. It's worth keeping in mind the popular memento, according to which…
The only thing worse than a lack of research is unreliable, non-credible research, whose methodology, method of execution, and thus results arouse justified skepticism.
User Experience, UX research, and usability tests make sense if and only if they're performed with respect to academic methodological standards and when they're conducted, analyzed, and reported according to research standards.
In the social sciences, a lot of attention is paid to methodological correctness and the quality of results.
Methodologists and researchers put a lot of effort into improving research methods.
It's important to remember that every UX research and every research method (especially qualitative research) is rooted in academically developed tools, methods, and standards. Therefore, it's worth taking advantage of this potential.
So what are the internal and external Validity in UX Research?
How should research be prepared and conducted to have high confidence in the obtained results?
How is quality defined and achieved in qualitative research?
If you are ready for a portion of practical methodological knowledge, then we invite you to read!
Do you want to perform UX Research?
What is quality in qualitative research?
The Cambridge Dictionary defines quality as a characteristic or feature of something that distinguishes it from others.
In the context of qualitative research, Uwe Flick very accurately defined quality in his book "Designing Qualitative Research."
By the way, Uwe Flick's book is a very valuable compendium of knowledge that every UX researcher should use.
According to his approach, the quality of research is one of the main, or perhaps the most important, aspects of the project design stage.
Quality is strongly linked to the standardization and design of the research situation and the factors that influence it.
In his argument, Flick very aptly notes that if researchers can control and remove factors that interfere with the study – external and internal – they will obtain much more accurate, reliable, and objective results.
That will be useful in the design process and the development of a digital product.
We can define quality in qualitative UX research as the attention researchers pay to designing the study, conducting it, analyzing the data, reporting, and communicating the results.
Naturally, such a focus on quality, on ensuring that the study produces the most reliable results possible, also directs attention to the researcher, in particular:
- to his place, role, and limitations in the research process
- to his ability to see their own mistakes and limitations of their own perspective
- to his role in testing validity and validating quality in the research plan in each of its stages
- giving credibility validation equal status and priority as research goals, respondent selection criteria, research methods, and organizational and time framework.
How to achieve high quality at the design stage of UX research?
Flick has no doubts about this quality at this stage is ensured by a well-thought-out, justified, and adequate choice of research method. And this means, in practice, that the researcher can maintain the utmost non-arbitrariness in his choices.
In other words, the choice of research method cannot be dictated by the following:
- convenience
- personal preferences, likes, or dislikes
- a desire to speed up research, to make it simpler or less expensive.
In addition, the researcher needs to consider the chosen method's adequacy carefully. He needs to be sure that the method selected will be the best one to capture the fragment of reality that is going to be studied.
The adequacy is achievable, of course, through knowledge, and acquired experience, but also through the approach of the researcher, who should also, according to Flick, remain open to diversity.
The idea is to look for experiences and ways to study them that differentiate respondents, not just those shared by them.
At the stage of research implementation, quality will manifest itself primarily in the:
- rigor – the application of a given method in accordance with the art of research and consistently
- creativity – searching for new answers, observations, and inspiration rather than confirming established assumptions
- consistency – e.g., interviews are easier to compare and generalize if they've been done consistently
- flexibility – the ability to balance consistency, rigor, and creativity.
What are Reliability and Validity in UX?
Here we come to the problem of reliability, which Pallabi Roy Singh, in an article titled "Reliability and validity: ensuring a Foolproof UX research plan," defines as the need to ensure the repeatability and reproducibility of research.
In other words, the idea is that other researchers replicating the research by conducting it in the same way, under the same conditions, with respondents sharing the same characteristics that included them in the study, can get the same results.
In turn, Pallabi Roy Singh defines Validity as the degree to which the chosen research method measures what the researcher wants it to measure. The choice of research method is crucial for the high accuracy of research results.
The reliability of research and its accuracy is essential for the overall success of a research project.
And as Raluca Budiu, a researcher associated with the Norman Nielsen Group, teaches us in an article titled "Internal vs. External Validity of UX Research," whenever we prepare UX research (quantitative, qualitative), there is a high risk that the results won't reflect reality.
All because of a flawed research design.
We can also look at the Validity and reliability in qualitative research from a different perspective, which John W. Creswell proposed in his book "Research Design. Qualitative, Quantitative and Mixed Methods Approaches".
According to Creswell, Validity in qualitative research has different connotations than in quantitative research.
It's not directly related to reliability or generalizability.
The Validity of qualitative research means that the researcher has checked the reliability of the results using specific procedures.
In turn, Creswell believes that reliability should be understood as the consistency of capturing one's own current and historical research, as well as one's own and that done by other researchers.
Hence, it's essential in qualitative research to document procedures and create detailed protocols and research databases.
Pallabi Roy Singh, quoted above, distinguishes three methods that allow research results to be considered reliable:
- Test-Retest Reliability
- Parallel Forms Reliability
- Inter-Rater Reliability
The reliability in the Test-Retest formula means that similar results are obtained in both tests. Test-Retest works best as a reliability estimator when surveys or questionnaires are used in research.
The Parallel Forms Reliability formula compares two equivalent test forms that measure the same attribute.
Inter-Rater reliability is a measure of reliability used to assess the degree to which different researchers agree on their assessment decisions.
This reliability estimator is recommended for quality validation of observational, field, and contextual studies.
What are internal and external validity errors?
As we've already observed, Validity and Reliability are two different things that shouldn't be confused with each other.
Reliable research means that its results will be similar if we repeat it. Thanks to this, we will be sure that the result isn't coincidental.
However, it's worth remembering that research with high reliability and low validity is research that perfectly measures, unfortunately, not what we would like it to measure.
How to deal with this problem? The answer is to pay attention to internal and external validity.
The internal validity of research means that neither the researcher nor the procedure favors any possible results (e.g., reactions, behavior).
To put it a little differently, internal validity is the certainty that we cannot explain the observed, captured in the course of the research, cause-and-effect relationship in another way with the help of other variables or factors.
We should mention the remark made by Arlin Cuncic in the article "Understanding Internal and External Validity. How These Concepts Are Applied in Research".
Research can be considered internally valid if we can eliminate alternative explanations for the results.
We can establish cause-and-effect if the research meets the following three criteria:
- cause precedes effect in terms of time
- cause and effect vary
- there are no other likely explanations for this relationship.
For example, if we want to research whether a screen's diagonal influences the speed at which a user performs a task, we divide the respondents into two groups, A and B. Then we conduct tests with them at extremely different times of the day (e.g., early morning and late evening). That will cause low internal validity of the results.
The differences in performance, and the speed of execution of the task, may have their cause in the size of the screen diagonal, but also in another variable, such as the time of day, which can affect the result.
Research conducted in this way will have low internal validity because we can't establish the cause-and-effect relationship between task completion speed and a screen's diagonal size without a doubt.
The relationship between these variables may or may not occur, or it may occur and be further conditioned by yet another variable (or variables) that distorts the result.
The external validity of the research means that the participants and the research conditions are representatives, reflecting a fragment of reality.
Sticking with our example:
The research will have low external validity if it's conducted in a laboratory under extremely different conditions from typical user experience and behavior.
For instance, the distance from the user's monitor will be unnaturally long, the light level may be far from usual, and the device itself and its method of operation may be unknown to the respondent.
The difference may also stem from the sheer influence of researchers on a respondent (e.g., intrusive, distracting verbal, and behavioral interference).
Factors that can disrupt external validity most often include:
- research conditions that deviate far from natural conditions
- measuring tools (e.g., Head-Stabilized Eye Tracking).
The high internal validity of the qualitative research can be achieved by:
- using a random order of tasks
- controlling the configuration of the study from one session to another
- by detecting and neutralizing variables that distort the results.
The high external validity of the qualitative research can be achieved by:
- defining precisely the criteria for inclusion/exclusion of respondents
- recruiting respondents that are representative of the target group
- ensuring that the conditions, the context, and the course of the research correspond as much as possible to the natural and real processes, conditions, and context.
Internal and external Validity in UX Research. Summary
- A popular myth and approach to UX Research is the belief that data isn't debatable; data is acquired and used.
- It is worth remembering that the only thing worse than a lack of research is unreliable and non-credible research. The reliability of qualitative research, or rather the validity and reliability of qualitative research, is essential for making good business decisions.
- Every research method used in UX Research (e.g., UX testing) is rooted in tools, methods, techniques, and standards developed by scientists.
- The quality of research is one of the main aspects to be considered at every stage of the research project.
- Quality is strongly linked to the standardization and design of the research situation and the factors that influence it. Qualitative research is susceptible in this aspect.
- Researchers who can control and remove factors interfering with the research – external and internal – can obtain much more relevant, reliable, and objective results.
- Quality is defined as the attention researchers pay to designing the research, conducting it, analyzing the data, reporting, and communicating results.
- The quality, reliability, credibility, and validity of the results are also influenced by the researcher, who must be aware of his influence on the research results.
- Reliability is defined as the need to ensure the repeatability and reproducibility of research.
- Validity is the degree to which the chosen research method measures what the researcher wants it to measure.
- The choice of research method is crucial for the high accuracy of research results.
- By paying attention to quality in qualitative research, the risk that the results obtained in the study don't reflect reality is reduced.
- Validity in qualitative research isn't directly related to reliability or generalizability.
- The validity of qualitative research means that the researcher has checked the reliability of the results using specific procedures.
- The reliability in the Test-Retest formula means that similar results are obtained in both tests. Test-Retest works best as a reliability estimator when surveys or questionnaires are used in research.
- The Parallel Forms Reliability formula compares two equivalent test forms that measure the same attribute.
- Inter-Rater reliability is a measure of reliability used to assess the degree to which different researchers agree on their assessment decisions. It's recommended for quality validation of observational, field, and contextual research.
- Research with high reliability and low validity is research that perfectly measures, unfortunately, not what we would like it to measure.
- The internal validity of research means that neither the researcher nor the procedure favors any possible results.
- Internal validity is the certainty that we cannot explain the observed, captured in the course of the research, cause-and-effect relationship in another way with the help of other variables or factors.
- We can establish cause-and-effect if the research meets the following three criteria: cause precedes effect in terms of time, cause-and-effect vary, and there are no other likely explanations for this relationship.
- The external validity of the research means that the participants and the study's conditions are representatives, reflecting a fragment of reality.
- We can achieve the high internal validity of qualitative research by using a random order of tasks, controlling the configuration of the study from one session to another, and detecting and neutralizing variables that distort the results.
