Usability Testing
Actually fielding a usability study is the moment of truth. Staying true to project goals and completing a task analysis accomplishes a ton for the seeker of usable results, but those strategies quickly become moot if researchers can’t collect quality data when it’s showtime.
Again, creating usable results isn’t about writing them better; it’s about being proactive enough to gather usable data in the first place.
There are helpful strategies at various points in the usability study process: Before the study takes place, in the middle of the study, and after it has finished. Below are some strategies for generating and harvesting top-of-the-line data:
Pre-Session
Establish clear research questions
It’s worth mentioning again; take the time to establish clear research questions and stick to them. It’s beyond silly to review a study’s data and find that you’ve answered questions you just didn’t need the answers to. Even worse: that you haven’t answered questions you really needed answered. Use kick-off meetings to communicate with stakeholders; find out what questions they want answered, and if the team just doesn’t know that yet, define those questions as soon as possible. Usability testing is a journey and strong research questions are the compass.
Write a strong discussion guide
Discussion guides can be incredible tools for conducting high-quality research. They add structure to the session. Specifically, they really help to keep moderators on track and objective, as a strong discussion guide is an excellent mitigator of experimenter bias.
Most importantly, however, discussion guides keep things consistent. Especially in heavily qualitative studies, moderators have the tendency to ask the same questions in slightly different ways. Use a discussion guide to ensure you’re asking –and answering– the right questions once everything is in motion.
Create a strong notetaker’s guide
If you make an organized and comprehensive note guide beforehand, it’s a simple matter of filling it in as you go. Notetakers may simply write down everything they see and hear, but lack of structure can a) make it incredibly difficult to keep up, and b) traumatize whoever analyzes the raw data.
Note guides enhance the note taker’s efficiency and productivity. When conversations get rolling or participants unwittingly find themselves down a rabbit hole, note takers are doing everything they possibly can to keep up. The last thing they need is the added task of locating the appropriate place to put those notes; that part should be the most obvious thing in the world. So when it comes to fielding preparation, finding ways to make the note taker’s life a little bit easier allows them to focus on their job: taking good notes.
Run a dang pilot
Run. A. Pilot. I don’t care how prepared you think your team is; something can and will go wrong during your first run through the script. Some of us get lucky occasionally and hit it out of the park on P1, but you just can’t count on that.
Running a subpar session for any reason leads to poorer quality data. Work out the kinks before the sessions count so that you can get good feedback from the start.
Mid-Session
Manage session length
Keeping the session length somewhere between too-short and too-long accomplishes a few things. First, it helps your participants stay fresh. Second, it helps you stay fresh. Neither scenario is good. If the participant is tired, anxious, or just fed-up with your questions, they might not provide good feedback. They’ll give up quicker, their qualitative responses will be shorter, and they’ll just want to leave. Burnt-out moderators will experience similar symptoms: sharpness declines and qualitative probing won’t be optimal.
It’s crucial to keep sessions on time as well. Even if you get great stuff from one participant, other participants will get less time. Plus, you really don’t want to make your participants wait any longer than they need to.
Truly know the product you’re testing
Moderators should really know their stuff during a usability test. First of all, you will look silly if you cannot field participants’ questions successfully. Second, unanticipated errors or system glitches are almost impossible to avoid, so researchers must be able to manage those mid-session. Third, understanding the product allows the moderator to probe for extra detail when needed. Only when the moderator fully understands the product can he or she explore specific use-cases with the participant.
Don’t trust your memory
It’s very easy to think that your vague or incomplete notes will suffice during a session. This is foolish! It doesn’t matter how much you think you’ll remember; details will rapidly decay in your memory. If you don’t write down that small detail during the session (or at least directly after), it won’t end up in the report.
The moderator may have tapped into a rich source of qualitative feedback, but none of that matters unless someone actually documents all that valuable data. It’s your evidence. It’s part of your deliverable. Even when you think you’ll remember, don’t skimp on taking strong, detailed notes.
Having said that, things do happen. Notetakers get tired. Audio cuts out. Other folks in the observation room get chatty. For these reasons, always record the sessions. Maybe you won’t need to refer back to those recordings, but when you need them, you really need them.
Post-Session
Debrief after each session
It depends on the client and it depends on the project, but there should be some kind of debrief at the conclusion of each session. Which parts were confusing for the participant? What did the team learn? What needs to be discussed further?
At a minimum, the moderator and notetaker should be doing this. Usually, these two people will be responsible for the deliverable, so being on the same page is essential. It will also be very helpful to have someone from the development team involved in these debriefs. They know the product best and they know the research questions most intimately. They’ll be able to tell you which parts of the session were of particular importance to them, which is great because you can then ensure those details are discussed in the final report. Remember, always know your audience!
Organize notes at the end of each session
Immediately following a study session, the notetaker should take a moment to reflect and review what they’ve just written. Especially in fast-paced studies, it’s easy to misplace certain details or even leave some out. I don’t care how good you think your memory is, details go down the drain with every passing minute. Go back to make sure everything you want to be there is properly documented.
Once the moderator and notetaker have debriefed, the notetaker should go back into the notes and add detail to whatever needs to be expanded on. At the time, it will undoubtedly feel like a pain, but those extra five minutes spent on boosting the note quality are well worth the headache.
Strong usability studies lead to usable results
Providing usable and actionable results relies heavily on robust data. You can practice writing usability reports all you want, but the capacity for tremendous work is inhibited by poor data; there’s just no way around it. Fortunately, proactive measures before, during, and after fielding pay dividends in the long run.
To view the other parts of this blog mini-series, please follow the links below!
Part 1: Formative vs. Validation
What are formative and validation studies? How do they differ from one another? What are their goals? How do those topics impact how the results are reported?
Part 2: Using Task Analysis
What is a human factors task analysis? What are its contents? How do you define critical tasks and task success criteria? How does its quality impact how usable results will be?
Part 3: Usability Studies
How do you collect high-quality data? How will that impact how usable results will be later in the research process.
