Why Your Usability Testing Results Are Confusing (and How to Fix Them)

‍

Do you feel like you’re putting a lot of effort into usability testing and still not getting the answers you need? 

Chances are you’ve accidentally stumbled into one of these 5 common pitfalls: 

‍

1. Losing focus on usability
2. Attitudinal questions framed as tasks
3. Testing the user, not the design
4. Confusing brand for usability
5. Forgetting the details

‍

Let’s take a look at each one, how to avoid them and improve your usability testing

‍

1. Losing focus on usability

When creating tasks for a usability test it can be tempting to want to learn as much as you can from your users. After all, the more the better, right? 

Usability testing is a specific activity. Adding broader or unrelated questions and tasks can dilute your results and pull the participants' focus away from what they’re doing.  

This can happen in both moderated research, when a moderator or facilitator is present and unmoderated research where a participant completes tasks on their own.

‍

‍In Moderated Research

During a usability test in moderated research, it’s important to observe. Once your product goes live, you won’t be with your users to give them guidance, ask or answer any questions. This should be avoided while a participant is completing a task as it can negate your results and produce false-positives. 

‍

Solutions

• Split your moderated research into two parts: Usability testing and Interview. 
• Communicate that your role is observational during a task to avoid priming the participant and influencing the results. 
• Take the opportunity to ask additional probing questions once your usability test has concluded.

‍

In Unmoderated Research

Platforms and tools that allow unmoderated research, while popular and easy to use, can inadvertently skew your results. The quality of the prototype, framing of the task and when and how the questions are asked will all impact your results. 

There is no opportunity to respond to the participant in the context of what you’ve just observed.

These tools also allow you to conduct different types of tests that are marketed as usability e.g. click test, concept test or preference test to name a few, but do not evaluate usability.  

‍

Solutions

• Acknowledge the limitations of unmoderated research and plan accordingly. 
• Consider moderated research if you still have a lot of unanswered questions. 
• Use a high fidelity, bug free prototype
• Eliminate questions that rely on a context you won’t be able to observe.
• Be explicit about the type of test e.g if it’s a concept test or preference test, call it that rather than usability.
• Identify how the results will contribute to your understanding and what decisions you can make from the information.

‍

2. Attitudinal questions framed as tasks

“What did you like about this screen?”

“Do you think you would enjoy using this?”

“How easy was this to use?”

‍

These are common examples of questions that can be both leading and add confusion to your results. Typically, the responses to these questions come from small data sets which isn't representative of an entire user base.

• What do you do if 3 out of 5 participants liked X on the screen and 2 out of 5 liked Y?
• Do you replace X with Y just because 1 more person liked it? 
• What happens if we never get 5 out of 5 people liking it?

‍

Hopefully these questions sound absurd when you think about how we design software. Letting users dictate design decisions based on the results of attitudinal questions leads to ineffective design. 

‍

Solutions

• Focus on constructing well written tasks to uncover usability issues
• Use large scale surveys for attitudinal questions as this lends itself better to identifying groups and trends.
• Attitudes are unique to individuals, consider what they are saying in the context of what they are doing or have just done.

‍

3. Testing the user, not the design

A clear signal that your users are being tested is the phrase: ‘X out of Y users’. Usability testing is about identifying pain points and issues in the design. If a user struggles, the key question is ‘Why?’ 

When exploring different design concepts, this framing can also influence us to focus on a ‘winner’ rather than evaluating the components of good usability. It might look something like this: 

‍

• Concept A - 4 out of 5 users completed the task successfully
• Concept B - 3 out of 5 users completed the task successfully
• Therefore, Concept A is better

‍

In reality, the difference in task success is 1 person and that could have nothing to do with the design. This type of metric doesn’t tell us why some succeeded and some failed. 

‍

Solution

Focus on what you observed and bring it back to the elements of the design. 

Example: 

• We observed that some participants clicked on transfer and some participants clicked on payment. 
• Those that clicked on transfer were able to successfully complete the task the first time, whereas those that clicked on payment had to backtrack. 
• This means we need to re-look at the language we’re using and the flows of both the transfer and payment journeys. 

‍

‍

4. Confusing Brand for Usability

Branding is important to a company. It’s unique and contains many design elements and skilled hours of work to communicate a look, feel and a tone of voice to potential customers. 

Knowing your audience and how your brand resonates is useful. 

Asking users questions about what should be design decisions is like letting an unqualified stranger come into your house and redesign the interior without any consideration for your own style.

‍

Solution

Remove questions relating to how users feel about your brand from your usability testing. Instead, focus on the different elements of the design that go into great usability:

• Is the copy in your tone of voice clear and well understood?
• Do the images you’ve chosen support the written content and aid in understanding?
• Are the typography choices drawing attention to the most important information?

‍

‍

5. Forgetting the details

If you notice participants struggling with a certain part of the prototype that contains odd or unusual data that is not realistic, this can create noise. 

There may not be a problem with the usability. It could be a detail that is causing a participant to pause and make sense of something they wouldn’t have to if they were using it in production.  

‍

Solutions

• Use realistic content in your prototypes
• Check default text, numbers to ensure they are accurate and consistent
• Do a run through with another member of your team to help you pick up any errors or mistakes you may have missed
• Define your success criteria before testing

‍

‍

Conclusion

Usability testing is a powerful activity in designing products that are easy to use. When done well, it can provide clarity about what’s working well and areas for improvement. When done poorly it can cause confusion and poor direction.

I hope the above guidance helps you to identify and correct any of these 5 common pitfalls. 

Have you noticed any of these in your organisation? 

‍

‍

Need Help?

If you’re new to usability evaluations or want expert input, we’d love to support you. Whether you're looking for advice or a full independent review, get in touch - we’re here to help make your product better.