Bias in Human Factors Interviews
Moderating interviews is crucial to conducting human factors or user experience (UX) research. A usability test is the moment of truth to get the data we need. Obtaining the best data possible starts with the moderator. Human beings are socially proficient creatures, and participants can pick up on inadvertent biases if the moderator is anything less than objective. So that we may harvest pure, high-quality data, it is essential that we reduce these biases. For Human Factors Engineers, remaining objective is a skill that must be mastered along the way to moderating greatness. Below, we describe our most helpful tips and strategies to mitigate experimenter bias:
1. Identify your biases
The first step is to simply identify any biases you might already have. It is much easier to be objective if you are aware of the threats that seek to take it from you.
After a comprehensive review of the product, talk with your team and try to identify some anticipated results. Any expectations of a particular outcome can be a bias. And while it’s okay to have these biases, swaying the participant, intentional or not, is not ideal.
Be aware of your expectations and biases. Awareness is the first step to keeping these tendencies in check.
2. Be aware of your language
It is important to understand the effect of your words on the participant. For example, if you say “Great” as a response to someone completing a task, two things have occurred: First, by saying anything at all, you’ve confirmed to the participant that they have completed the task. He or she may have been unsure! Second, saying “Great” tells the participant they completed the task successfully, regardless of whether they actually did or not. Not only does this give the participant unrealistic information for that task, it can also unnaturally affect anything that includes that task in the future.
Remain neutral and objective in your word choice. Saying “Okay” or “Do you believe you have completed the task?” after a long pause is usually good practice. It may sound repetitive or a bit automated, but that’s okay; it ensures that each participant receives the same conditions of testing.
3. Be aware of your body language
For the same reasons as above, be careful to keep your body language neutral during user experience interviews. Usability testing evaluates the product itself, not the participant. Despite that, many participants feel like they are being assessed. Even simple gestures like a deep sigh or a head nod give participants an idea of how they are doing. These seemingly insignificant micro-expressions are extremely common, and they can alter participant behavior.
If the participant does everything correctly, great, but keep it on the down-low. No nods, no smiles; nothing that could affirm to the participant that he/she is doing something “correctly.” On the flip-side, deep sighs express frustration. Rubbing your eyes or leaning back may convey fatigue or inattention. If the participant just barely missed the “happy path,” that’s okay. Don’t display frustration or negative energy. You’ll get great data by observing their search and remaining even-keeled.
4. If you’re a stakeholder, keep out
If you were part of the design team or otherwise helped create the product/system, you should not be moderating. In other words, if you have a vested interest in the success of the product, ask someone else to moderate.
While not impossible, it is difficult for stakeholders to remain objective when discussing a product they’re invested in. Sometimes moderators know how the product is supposed to work and guide the participant along that path unintentionally. Equally common, moderators really, really want the product to perform well in testing and inadvertently cause that to happen.
It is not impossible. There are many people who perform their own usability testing and do it successfully. However, as a general rule, it’s best to have someone from an objective third-party as the moderator.
5. Discussion guides are great; write one that keeps you on track
Not all of the safeguards against experimenter bias occur during the interview itself. Discussion guides can be an excellent tool for this.
Construct a discussion guide that has already considered the various threats of bias. The highest risk of subjectivity is introduced when moderators work under flexible scripts and are allowed to probe as they see fit. This is a very common strategy, but untrained moderators can easily introduce bias in these situations.
If you have specific research questions, write the discussion guide to address those questions objectively. By doing so, even if the moderator is harboring some kind of bias, the script will help keep them on track. Additionally, if you catch yourself leading the participant, just lean on your script. If you and your team did it correctly, the discussion guide ought to be solid. You should always be able to follow it if you don’t feel confident in your ability to be objective.
6. Know the product well and know what to expect
As discussed above, it can be dangerous to expect certain results. However, if you can keep your moderating objective, knowing the product well can be very helpful.
Not knowing much about the product (a sort of double-blind research strategy, essentially) means the moderator learns about the test product alongside the participant. The threat of bias lies in the corresponding moderator reactions to a good or bad design. Reactions are often unintentional and subconscious, and they can be noticed by the participant. Much like micro-expressions, these can influence the participant’s perceptions of the product.
If you know the product and know what to expect from participants, good or bad interactions won’t be nearly as surprising. This will enable you to remain neutral in your actions and your words.
Conclusion
The threat of moderator bias is constant in usability research. Fortunately, there are ways to meet the threat, head on. It just takes a bit of practice.
It all starts with self-awareness: identifying sources of bias and anticipating how they might show through during a research session. Neutral language, controlled body language, and careful planning can make all the difference when trying to remain objective and gather meaningful, high-quality data.
One of our favorite usability testing books for Human Factors Engineers is Jeff Rubin and Dana Chisnell’s Handbook of Usability Testing. You might also be interested in our book entitled Humanizing Healthcare: Human Factors for Medical Device Design.
For helpful resources on Medical Human Factors Engineering and User Experience, check out our blog. For helpful Medical Human Factors Engineering and User Experience check out our YouTube channel.
