Since the number of new mobile platforms and models is increasing yearly, shouldn't we adapt our research methods to these changes? Since usage patterns also change, shouldn't we consider them in the research process?
And can user experience research of mobile and desktop applications (e.g., usability testing) be conducted using the same methods? Are there any differences between them?
So today, we'll discuss UX research. Unsurprisingly, this topic comes up in the context of user experience. Without research, there is no good and positive user experience.
Without research, there are no good mobile or web apps. However, for research to lead to good solutions, it must be relevant to the subject of the study. How to make them that way? Soon you will find out everything.
I invite you to read the article.
We Audit. We Research. We Design.
What does user experience research involve?
Let's start with a quick refresher. User experience research (especially on usability) aims to match the mobile or web application to its users' expectations, requirements and needs in the best possible way. On the one hand, it helps understand users. On the other hand, it makes it possible to understand the impact of a project (e.g., a prototype) on a user.
User experience research rationalizes the design process and provides data, conclusions, analysis, and insights.
It equally uses qualitative and quantitative data. These provide a way to find answers to fundamental questions from a user experience perspective.
Important user experience questions:
- What expectations, habits, and experiences do users have?
- What needs do they want to fulfill with the app?
- What problems do they face?
- How do they use the application?
- What elements hinder, discourage, or prevent conversion?
- What fears, uncertainties, or doubts does the application raise (FUD — Fear, Uncertainty, Doubt)?
- How is the interface perceived and rated?
- What is the user's previous experience with similar applications?
Of course, these aren't all the questions that research can answer, but only the most common ones.
UX research also makes it possible to identify, pinpoint, and reveal the following:
- Mistakes made by users
- Errors occurring in the application
- Measures of success (e.g., the correctness of tasks performed, finalization of tasks)
- Cognitive load (e.g., speed of task execution, ease of use of the interface)
- Cognitive friction
The most common methods include:
- Observations
- IDI (Individual In-Depth Interview)
- Card sorting
- Focus groups
- Expert analysis/heuristic analysis
- Surveys
- A/B tests
- Eye tracking
- Click tracking
- Cognitive walk-through
Research—particularly of mobile applications—will become increasingly important, primarily due to the rise of mobile devices. All signs indicate that mobile phones will soon be our primary devices for entertainment and work.
The current pandemic situation reinforces this trend.
Another argument favoring dedicated UX research is the increasing complexity of functions, contexts of use, and usage purposes. And this creates significant problems related to usability, experiences, and satisfaction.
We should also remember the growing number of applications. Nowadays, competitiveness is more about ensuring good UX than creating innovative solutions or new functionality.
Offering a better experience is a matter of research, optimization, and continuous development.
It won't be an exaggeration to say that the development of mobile and web applications is "a work in progress," and UX research is an integral part of it.
The peculiarities of using mobile and desktop devices
From the user's perspective, mobile and desktop devices provide different experiences we must consider in the design and research process. We use smartphones in different contexts, for different purposes, and in different ways. So, our relationship with these devices differs significantly.
We use apps installed on our phones in a wide variety of contexts. Private, public, personal, and professional.
The changing nature of the contexts and their diversity are significant challenges for designers and researchers (I'll write more about the problem of simulating conditions in mobile application research).
When we use smartphones, we change our location. Therefore, our capabilities change (e.g., related to Internet access), and the way we use the phone changes (e.g., with only the thumb of one hand or with both hands).
A large amount of stimuli makes our attention highly dispersed. An excess of information burdens the working memory.
Consequently, interactions with the application interface are shorter, simpler, more spontaneous, impulsive, more shallow in terms of engagement, and more stressful.
Mobile devices are also treated much more personally and store private, intimate information and resources. Applications raise concerns about data protection and access to data related to health, finances, or plans.
Our phones also differ significantly regarding physical and technological parameters (e.g., diagonals, screen resolutions, operating systems). Diverse capabilities also produce complex, similar, but not identical experiences (e.g., related to tactility).
As noted by the authors of the article "Badanie użyteczności oraz dostępności interfejsu w aplikacjach mobilnych" (Usability and accessibility testing of mobile application interfaces), communication between the mobile device and the user currently takes place using elements such as voice, bum-ping, vibration, jostling, and gestures, including one-touch and multi-touch interactions.
Compared to the experience produced by desktop computers, the mobile experience means a smaller workspace, smaller interface elements, and operation usually with a thumb that is less precise than a cursor or a mouse.
It also involves experiencing interruptions, discontinuities, distractors, and chaotic processes.
That is why mobile UX design and research on mobile applications must focus on functionality and the determinants of the interaction between a user and the application interface.
UX research of mobile and web applications
Exploratory and test research are the two most important types of UX research. The former is mainly used to observe behavior and problems through conversation, while the latter is primarily used to observe application usage patterns. The former enables the discovery of needs, while the latter reveals issues.
Web and mobile application research is based on the same principles. The quantitative and qualitative methods used (e.g., usability testing) complement each other.
The significant difference stems from these devices' vastly different contexts and uses. This should be reflected in the choice of adequate testing methods for individual applications.
The most significant differences include the following:
- Context and external environment in which the device is used
- Diversity of mobile devices
- Variety of operating systems
- Wide selection of screen sizes and resolutions
- Diversity of sensors and peripherals in devices
- Ways in which a device interacts with a user
The differences mentioned above concerning the operation, functionality, and, above all, the context of use cause many problems, mainly when the research aims to improve the usability of mobile applications. Among the significant issues are:
- The dilemma of context — conducting field research under natural or laboratory conditions
- The problem of session recording — external (e.g., facial reactions) and internal — recording the actions performed on the phone
- The problem of the diversity of prototypes — resulting from the wide variety of devices and platforms and the requirements they place on applications
The most important problem from the perspective of reliability and validity of research is the dilemma of context. The research situation and typical usage of desktop devices differ little.
This difference is fundamental for mobile devices. Is it possible to conduct usability testing of mobile applications in a laboratory setting?
The problem of laboratory simulation of context in mobile application research
According to the authors of the article "Mobile Usability Research—The Important Differences from the Desktop," we can conduct mobile application research in a laboratory setting. Most importantly, the accuracy of the results obtained isn't a concern.
The comparative research mentioned in the article revealed little difference. For now, no convincing evidence undermines the quality of the results obtained under controlled variation of the context of mobile app usage.
Of course, observing users' behavior, reactions, and actions as they perform tasks in applications is extremely difficult under changing conditions (especially if the change is due to a change in location or movement). This is primarily due to organizational, technical, and methodological factors.
Reconstructing context in the laboratory is nothing new in the social and natural sciences. In her article "Mobile user research methods," Lorraine Patterson points out several ways to recreate natural conditions for user interaction with a mobile app interface.
Factors that help recreate natural conditions for user testing:
- Audiovisual distractors — music or video played in the laboratory
- Social distractors — research assistants who distract via their behavior or questions, etc.
- Motion and obstacle simulators — treadmills, walking routes with obstacles, simulating the variable movement speed
- Stop/start techniques — enable the simulation of involuntary task interruptions
The main difference we should keep in mind is addressing context variability. Research on desktop and mobile applications differs not so much in methods as in the conditions under which observations, measurements, and documentation are made.
Research of mobile applications conducted in a neutral context that doesn't provide significant changes in the situation, conditions, and strong stimuli will be incorrect.
There are several significant factors in favor of conducting tests under laboratory conditions:
- Reliability and validity of results
- Ease of organization and execution
- More control over variables
- Lower costs
- Greater potential for the use of recording equipment
The use of Nielsen's heuristics in mobile app research
One of the more popular methods for testing mobile applications is heuristic analysis. This method is well-known and often used in web application research. The heuristic analysis makes it possible to diagnose usability problems more accurately, especially regarding interfaces' usability.
It has become popular because of its speed of execution and low cost. The accuracy of the results is obtained with the involvement of a small amount of time and specialists.
Of course, Nielsen heuristics are the most popular for desktop devices. They help detect usability problems in 10 dimensions, including visibility, customization, control, consistency, and error prevention.
Nielsen's heuristics were initially developed for desktop devices, so they must be modified slightly for mobile application research.
Two authors, Rebecca Baker and Xiaoning Sun, adapted popular heuristics. In the article "Empirical Development of Heuristics for Touch Interfaces," they presented some interesting modifications worth using in the heuristic analysis of mobile applications.
The authors propose to supplement the classic heuristics with three additional ones.
The first is Inconsistent Interaction Design, which occurs when users cannot use existing knowledge gained from using similar applications. They expect results because of analogy and experience, but these don't appear in the expected form. This situation causes confusion and frustration and is a source of task abandonment.
The second problem is Inaccurate Selection, figuratively referred to by the authors as the Fat Fingers problem. Although tactility is, in a manual sense, more straightforward than operating a mouse, it isn't as precise as the method of interaction known from desktop computers.
Therefore, it's crucial to study the size of the interface elements, their mutual distances, the errors that may occur due to their mismatch with the sensitivity of the device, and the size of a typical thumb.
The lack of help and navigational aids (e.g., search box, home page icon) is a significant inconvenience when using mobile applications. Like the previous two, the third problem is also confusing and frustrating.
UX research apps: UX research tools
Here, we present (in no particular order) UX research tools that can facilitate the UX research process and user testing. They all provide invaluable features that can help us conduct usability testing and improve the user experience of web and mobile applications.
Userlytics
Userlytics is one of the most popular UX research tools. It's a comprehensive user-testing platform that allows us to gather user insights and feedback.
Features
Notable features include the ability to conduct moderated and unmoderated user research. Userlytics also allows us to perform remote user research and recruit participants from a panel of 2 million research participants.
The platform makes it easy to set up and organize research thanks to its study builder, among other things. With it, we can choose the type of study we want to perform (quantitative/qualitative research) and the appropriate research method (e.g., card sorting, net promoter score, tree testing).
Moreover, it offers access to such features as transcriptions, video clips, unlimited data storage, compliance with ISO27001 and GDPR, and more.
Userlytics offers the following pricing plans:
- Custom (from $99 per session)
- Enterprise (from $34 per session and with volume discounts)
- Limitless (custom quote)
- DYI plans ($499 and $999 per month and paid annually)
Maze
Maze is a research platform offering tools for obtaining user feedback and insights. It also provides the option to conduct remote usability testing. Its study builder supports functionalities such as drag-and-drop and AI tools.
Features
Key features include prototype testing, website testing, interview studies, card sorting, and surveys. Similar to Userlytics, it offers the option to recruit study participants from their existing base, which allows us to quickly and effectively send e-mail campaigns.
Maze offers multiple integrations with one of the best UX research tools, such as Figma, Sketch, Amplitude, or Figjam.
Additionally, it provides access to design metrics, heatmaps, video recordings, path analysis, research templates, GDPR compliance, and more.
Maze offers the following pricing plans:
- Free
- Starter ($99 per month or $1,188 per year)
- Organization (custom pricing)
User Interviews
User Interviews is a UX research tool that simplifies participant recruitment. Thanks to the scoring system that assesses the researchers' feedback and reviews participants, we can be sure that we'll be matched with the best ones.
Features
Notable features include access to the participant database of over 4 million panelists, a fast recruitment process, smart targeting, and support from a project coordinator. This enables us to recruit research participants for moderated research, usability tests, and screener surveys.
Moreover, we can set up calendars so research participants can easily schedule sessions based on our availability, send Zoom links, distribute incentives such as gift cards, send reminders, e-mails, and texts, and rate participants. It also provides access to tools such as Google Calendar, Google Meet, SurveyMonkey, or Microsoft Outlook.
User Interviews offers the following pricing plans:
- Pay as you go ($49 per session)
- Essential ($36 or $41 per session paid annually)
- Custom (custom pricing)
Hotjar
This user research software allows us to track website user behavior through heatmaps. Heatmaps let us see where users interact with a site, meaning where they click, move, and scroll.
Features
Key features include recording user sessions, setting up a suggestion box to gather user feedback, conducting surveys and interviews, accessing the pool of over 200,000 research participants, and using AI support.
Furthermore, Hotjar provides multi-device support (desktop and mobile), smart visitor targeting, unlimited team members, and GDPR and CCPA compliance.
Additionally, it offers the following integrations and user research tools: Slack, Microsoft Teams, HubSpot, Google Analytics, AB Tasty, Optimizely, Amplitude, and more.
Hotjar offers the following pricing plans:
- Basic (free)
- Plus (€39 per month or €32 per month paid annually)
- Business (€99 per month or €80 per month paid annually)
- Scale (€213 per month or €171 per month paid annually)
- Custom package (custom pricing)
SurveyMonkey
SurveyMonkey is a platform for gathering actionable insights through surveys and forms. The software is AI-supported and enables us to choose from over 250 questionnaire templates that can be customized to meet our needs.
Features
Notable features include reaching people via e-mail, web links, texts, and kiosks, analyzing results with a customizable dashboard, and sharing reports.
We can utilize SurveyMonkey for such use cases as obtaining customer and employee feedback, creating forms and quizzes, managing events, performing market research, and developing products.
Moreover, it offers the following integrations: Salesforce, HubSpot, Zendesk, Mailchimp, Microsoft Teams, Zoom, Tableau, and Zapier.
SurveyMonkey offers the following pricing plans:
Team plans:
- Team Advantage ($25 per user per month and paid annually)
- Team Premiere ($75 per user per month and paid annually)
- Enterprise (custom pricing)
Individual plans:
- Premier Annual ($1,428 paid annually)
- Advantage Annual ($468 paid annually)
- Standard Monthly ($99 per month)
Loop11
Lopp11 is a user/usability testing tool designed for marketers, designers, and product managers. Its user-friendly test builder allows us to conduct usability testing (moderated and unmoderated). Like its competitors, it also provides access to participants, making recruitment much more accessible.
Features
Key features include remote usability testing, A/B testing, prototype testing, mobile and tablet UX testing, information architecture testing, true intent studies, UX benchmarking, and search engine findability.
Additionally, Loop11 uses AI to analyze results and provide reports. It also offers clickstream analytics, heatmap analysis, and user session recording to understand user behavior better.
On top of all these features, it also provides integrations with User Interviews and Slack.
Loop11 offers the following pricing plans:
- Rapid Insights ($199 per month or $179 per month and paid annually)
- Pro ($399 per month or $358 per month paid annually)
- Enterprise ($599 per month or $533 per month paid annually)
Lookback
Lookback enables us to conduct usability testing and interviews. It facilitates the UX research process for all involved, whether they're researchers, product teams, or stakeholders.
Features
Notable features include the ability to run qualitative research, import existing research results, transcribe sessions, generate links to sessions or recordings, create reels and clips, capture screen touches, and more.
Additionally, Lookback offers Eureka, an AI-powered assistant that speeds up the analysis of UX research results.
Furthermore, the platform supports integrations with User Interviews and Respondent.
Lookback offers the following pricing plans:
- Freelance ($25 per month paid annually)
- Team ($149 per month paid annually)
- Insights Hub ($344 per month paid annually)
- Enterprise (custom pricing)
Ethnio
Ethnio is a customer relation management software specialized in user research management. It's a powerful UX research tool that allows us to track the actions of test participants. It enables testing on multiple platforms, including desktop, Android, and iOS.
Features
Key features include access to the pool of participants, multiple incentive options, whether monetary or in the form of gift cards, generating and sending screener links, setting up intercepts, and using diverse scheduling options. Furthermore, Ethnio is DGPR and CCPA-compliant.
Moreover, the software offers the following integrations: Slack, Zoom, Microsoft Teams, Google Meet, Lookback, Google, Outlook and macOS calendars, Excel, Google Sheets, and more.
Ethnio offers the following pricing plans:
- Starter ($15 per month or $12 per month paid annually)
- Plus ($49 per month or $40 per month paid annually)
- Pro ($99 per month or $82 per month paid annually)
- Enterprise (custom pricing)
Optimal Workshop
Optimal Workshop is a user research platform that allows us to improve the user experience on our websites and applications via diverse UX research methods. The platform offers access to specialized UX research tools such as OptimalSort, Treejack, Chalkmark, Questions, and Reframer.
Features
Notable features include the ability to perform quantitative and qualitative research, card sorting, tree testing, online surveys, first-click testing, user interviews, and participant recruitment (239 million participants). Moreover, it provides automated analysis and various visualization options.
Optimal Workshop offers the following pricing plans:
- Pro ($249 per month or $208 per month paid annually)
- Team ($249 per month or $191 per month paid annually)
- Enterprise (custom pricing)
Optimizely
Optimizely is a comprehensive platform offering diverse tools for marketing and product optimization. It offers products such as a content management system, feature experimentation, web experimentation, digital asset management, and more. UX research is just one aspect of it.
Features
Key features include A/B testing, multivariate testing, click and conversation tracking, audience targeting, and AI support. It also enables the performance of user research on both desktop and mobile and testing of individual features.
Furthermore, Optimizely offers the following integrations: Baymard Institute, Nansen, Fullstory, Microsoft Clarity, Slack, Usercentrics, UserTesting, and many more for diverse use cases.
Optimizely offers the following pricing plans:
- For detailed pricing, customers need to contact Optimizely directly.
UXtweak
UXtweak is a platform offering diverse UX research tools that help us improve our product's user experience and usability.
Features
Notable features of this UX research platform include card sorting, tree testing, mobile testing, first click testing, prototype testing, website testing, preference tests, five-second tests, online surveys, session recording, freeform interviews, and study interviews.
Moreover, it offers access to a user panel with participants from more than 130 countries, onsite recruiting that converts website visitors into testers, and the ability to implement our own database with participants.
Furthermore, it's GDPR compliant.
UXtweak offers the following pricing plans:
- Starter (free)
- Plus (€99 per month or €49 per month paid annually)
- Business (€179 per month or €153 per month paid annually)
- Enterprise (custom pricing)
Userfeel
Userfeel is a user testing tool for conducting UX research without the need to pay subscriptions. With Userfeel, we can quickly obtain user feedback and gather insights into the needs of our target audience.
Features
Key features of this tool include moderated and unmoderated tests, testing in multiple languages (over 40), a tester panel with high-quality participants (over 1 million), screener questions, video clips, notes, cross-platform testing, system usability scale, and voice transcriptions.
Userfeel offers the following pricing plans:
- 60 US dollars per credit (20-minute video recording)
- Pay as you go
UserTesting
UserTesting is a user research platform that allows us to remove the guesswork from our usability tests and tangibly improve our products.
Features
Notable features include a participant database with the option to recruit more from partner networks, a feedback engine, AI-powered analytics and visualizations, reels, clips, mobile testing, agile design testing, click testing, tree testing, accessibility testing, A/B testing, and more.
Furthermore, UseTesting offers the following integrations: FigJam, Jira, Canva, Miro, Fullstory, Qualtrics, Quantum Metrics, Slack, Google Calendar, Microsoft Teams, Trello, and more.
UserTesting offers the following pricing plans:
- For detailed pricing, customers need to contact UserTesting directly.
Lyssna
Lyssna (formerly known as UsabilityHub) is a UX research platform that gathers user insights and feedback. It's an all-in-one usability testing platform that allows us to make user-centered, data-driven decisions.
Features
Key features include the ability to perform card sorting, first-click testing, five-second testing, interviews, preference testing, prototype testing, online surveys, and tree testing.
Moreover, Lyssna provides access to a panel of over 530,000 research participants, precise targeting with screeners, result filtering, navigation analysis, click and heatmaps, flow analysis, and more.
Furthermore, the remote user research platform offers the following integrations: Zoom, Microsoft Outlook, Google Calendar, Figma, and Microsoft Teams.
Lyssna offers the following pricing plans:
- Free
- Basic ($89 per month or $75 per month paid annually)
- Pro ($199 per month or $175 per month paid annually)
- Enterprise (custom pricing)
How to select the right UX research tool?
At the end of the day, the choice of the best UX research tool depends on the research methods we plan to use and the type of research we want to conduct. The set budget for our UX research also plays a significant role. If we're a start-up, then it's obvious that we won't go for the top-shelf user research software; we are small, so we should start small.
Naturally, this doesn't mean that we should save money on UX research since, in the long run, the test results and their implementation will only increase our profits.
Fortunately, the market offers an abundance of research platforms and tools and allows us to choose the most suitable one.
UX research of web and mobile applications. Summary
- UX research aims to tailor a mobile or web application to users' expectations, requirements, and needs as closely as possible.
- It rationalizes the design process and provides data, conclusions, analysis, and insights.
- UX research makes equal use of qualitative and quantitative methods.
- The development of mobile and web applications is an example of a typical "work-in-progress" in which UX research is integral.
- Mobile and desktop devices provide different experiences we must consider in the design and UX research process.
- The mobile experience means a smaller workspace, smaller interface elements, and operation with a thumb that is less precise than a mouse. It also involves experiencing interruptions, discontinuities, distractors, and chaotic processes.
- In summary, web and mobile application research is based on the same principles.
- The significant difference stems from these devices' vastly different contexts and uses.
- The most important problem from the perspective of reliability and validity of research is the dilemma of context.
- There is no convincing evidence that undermines the quality of the results obtained under controlled variation of the context of mobile app usage.
- Research on desktop and mobile applications differs not so much in methods but in the conditions under which observations, measurements, and documentation are made.
- Mobile application research conducted in a neutral context that doesn't provide significant changes in the situation, conditions, and strong stimuli will be incorrect.
- There is a variety of UX research tools that can help us conduct usability tests and improve the user experience of mobile and web applications.
