How unmoderated testing could be harming your designs
Unmoderated testing is a popular way of getting fast results and feedback from users but the nature of this testing can be harming your design outcomes.
In this post I look at 5 reasons why this common testing technique can reduce the quality of your product and what to do instead.
- Participants that don’t represent your users
- No opportunity for clarification
- Treating everything as usability
- A bias towards validation
- False quantification
1. Participants that don’t represent your users
Participant selection is an important step in any form of research and needs to be carefully considered given the study type, what you want to learn and who you want to learn that information from. Unmoderated testing platforms rely on panels of paid participants that tend to be more familiar with technology, testing tools and prototypes.
Why it matters
Even with screening criteria, these participants aren’t representative of everyday users especially when considering digital literacy, specific needs or accessibility requirements. This means you may be optimising your designs for people who aren’t your intended users.
What to do instead
Recruit from places that give you real representation, e.g:
- Existing customer base
- Community groups
- Disability advocacy groups
- Recruitment agencies that do in-depth screening
2. No opportunity for clarification
Once a study is live, there is no opportunity to make any changes. You won’t be able to clarify any misunderstood tasks or questions, explore interesting responses or ask why someone hesitated.
Why it matters
If a participant misunderstands a task or question, results reflect the quality of the study rather than the design or prototype. You may not even know a misunderstanding exists because you are unable to observe the participant and they may not verbalise their confusion.
What to do instead
- Perform a moderated test with 2 - 3 participants first to test your study and make any necessary adjustments.
- Keep tasks short, making sure they are clear, concise, unambiguous and free from jargon.
- Ask open questions to encourage a thoughtful response and avoid leading questions or questions that can be answered with a simple yes or no.
- Consider moderated follow ups to unmoderated sessions.
3. Treating everything as usability
Not everything from prototype testing should be treated as usability. Usability is an important part of design and while information from performing activities such as click, impression, comprehension and preference tests contribute to good usability, they are not usability tests in themselves.
Why it matters
Labelling everything as usability can detract from what you are trying to learn. For example, labelling a comprehension test as usability shifts the focus and you may reach the wrong conclusions. A participant can still complete a task in spite of poor comprehension and task success doesn’t mean the content is well understood.
What to do instead
- Be explicit about the goal of your testing, what you want to learn and the most appropriate activity to gain that information.
- In reporting, label activities accurately and explain how they contribute towards good usability.
4. A bias towards validation
Because unmoderated testing is a quick and relatively easy way of putting a prototype in front of someone and getting feedback, it’s often used as a way to validate designs or concepts. This shifts the goal from learning to validating.
Why it matters
This framing can create a narrow focus and important, insightful information can be ignored. By looking to validate something, we look for things that tell us we are right and our confirmation bias comes into how we design the study and how we interpret the results.
What to do instead
- Frame objectives as ‘What do we want to learn?’ rather than ‘How do we validate?’
- Have someone with no vested interest in the project independently review your study design looking for examples of confirmation based tasks and language e.g. leading questions.
5. False quantification
Survey questions in unmoderated testing like ‘How easy was this to use?’, ‘Which design did you prefer?’ or ‘How likely would you be to use this feature?’ are often used to quantify why a design provides a good user experience. The problem is that these questions are:
- Attitudinal - making them subjective
- Lacking context - participants may not represent real users
- For small sample sizes - having no statistical significance
Why it matters
It’s easy to mislead yourself, your team and stakeholders with quantitative sounding data that doesn’t provide any real insight. If 3 people preferred concept A and 2 people preferred concept B, that data alone doesn’t tell anyone why there was a preference either way. An average score of 4.1 might sound positive but ignores the insights from the person who really struggled and gave an individual score of a 2.
What to do instead
- Focus on qualitative insights in unmoderated testing by asking open questions about their experience. Someone saying that they found something difficult to use because they couldn’t understand a label is much more actionable than a 3 when asked ‘How easy was this to use?’
- Pair existing large scale data that informed the designs with behavioural observations from testing to explain changes and impact.
- Study participant responses over time. By having the same participants in each study, if a 2 changes to a 3 or 4 in response to ‘How easy was this to use?’ when the design is changed, real improvement can be shown.
Conclusion
Unmoderated testing can be a useful research tool when it’s used with care. At the core of meaningful research are the decisions we make to use the right methods and tools, in the right way, to gather relevant information. By understanding the limitations of unmoderated testing, we can design more suitable, useful and insightful studies rather than wasting time on activities that generate flawed data and misguided conclusions.
When unmoderated testing is used as a quick fix or as a replacement for more suitable research methods, business knowledge and the user’s experience both suffer. Knowing when it’s the right choice and when it isn’t is what leads to meaningful insights and better design decisions.
Need Help?
If you’re looking for expertise in experience strategy, user research, service design, usability and UX we’re here to support you. Whether you're looking for advice, a helping hand or a full independent review, get in touch.
