Picture this… You’ve successfully completed your usability test. You open your notes to begin analyzing the data and they look like gibberish. Now you’re forced to go back to watch study videos to make sense of what happened during the sessions. As a researcher, this is my nightmare. It has probably happened to many of us (to varying degrees). But not to worry! This nightmare is avoidable if you take the right steps before, during, and after your study. Here are some tips for successful Human Factors note-taking.
Note-taking – Before the usability study
First, let’s discuss what you can do before the usability study begins. Effective notes begin with a well thought out note guide (commonly known as “note sheet”, “data sheet” or “data log”). Using a spreadsheet is a common and effective way to organize your note guide. Consider what questions your study is trying to answer and the data you will need to answer them. Will you need task completion scores, qualitative notes, or something else entirely? Then, think about how to most effectively organize the data so it is easy to interpret during analysis.
For example, when conducting a usability study, it is wise to document a task’s score and the objective and subjective evidence you will collect separately. This can be done by creating separate rows or columns for notes about a task and its score. This method will also allow you to easily count successes, use errors, close calls, and difficulties using the built-in functions of the spreadsheet while the qualitative notes can be analyzed separately.
Planning with the moderator
It is common practice in human factors research to use one moderator and one note-taker team to conduct study sessions. When this is the case, it is important that the moderator and note-taker agree on what information is important and how it should be recorded. Describing each task’s success criteria in the note guide is often a good idea. Additionally, different researchers will likely have different styles of note-taking.
For example, when I take notes, I prefer to denote the moderator’s questions in [brackets], while others prefer to do so by denoting “M:” or “Mod:” before their question. There is no one right way to take notes, but understanding the note-taker’s style can be helpful, especially when someone other than the note-taker is responsible for writing the report. Someone who did not observe the session should be able to read the notes and understand what happened.
Note-taking during the usability study
Now, let’s talk about some best-practices for note-taking during the study. The best note-takers know what information is important and what is not. Adding too much information to the notes makes it harder to find what you’re looking for during analysis. Not every piece of information a participant shares is relevant to answer our study questions. If it’s not pertinent to the study questions, there’s usually no need to record it even if it’s tempting to transcribe the whole session.
Similarly, understanding the level of detail needed to answer a question equally is important. For example, let’s imagine we are evaluating whether a participant can remove the cap from a pen-injector. We observed the participant successfully remove the cap in 1 second by pinching it with their left thumb and forefinger and pulling it straight off.
To determine what details are relevant to include in the notes, let’s refer back to our study questions. If we’re simply interested in whether or not they successfully removed the cap, it may be unnecessary to include all the details about how they removed it or how long it took them. Alternatively, if we’re concerned about whether participants twist or pull the cap to remove it, we should include that information in our notes.
The moderator/note-taker team
The moderator and note-taker should have a means of communicating during the session. A live note guide to which both researchers have editing access during the session is an effective way to do this. When using this method, both researchers can take notes in the note-guide and see in-real time what each has recorded, ensuring all necessary information is captured. It is not the responsibility of just one researcher, moderator or note-taker, to ensure all questions are answered and all necessary data is collected.
Rather, the researchers should work as a team to do this. For example, if a use error occurred during a task, the note-taker should help the moderator ensure no follow ups are overlooked and a comprehensive debrief occurs. The note-taker can do this by 1) flagging the use error in the datasheet so the moderator sees it when it comes time to debrief, 2) formulating questions to help the moderator with debriefing, and 3) letting the moderator know if they think the debrief has not gone deep enough to identify a solid root cause of the use error.
It’s never a good feeling to realize that something has been overlooked after the participant has already left a session. Therefore, it is the job of both researchers to review the notes at the end of the session to confirm there are no further questions to be asked before the participant is dismissed.
Note-taking – After the usability study
After each session, the moderator and note-taker should review the notes together to ensure they make sense and have captured the necessary information for analysis and reporting. During this time, it is also helpful to denote a root cause in the note guide while the session is fresh in both researchers’ minds. It doesn’t need to be “report-ready” yet; a brief note is just fine.
Resolve any confusion about the note-taker’s style or which data is to be collected early on in the study. If video review does become necessary, it is smart to review it as soon as possible and not to leave it until the end of the study.
If the researchers have followed the advice outlined above, there’s not much left to do but get started on data analysis! In any case, it is good practice to review the full dataset once more for completeness. Once the study has concluded, raw notes should be preserved. Any analysis completed on the dataset should be documented separately from the raw notes, such as in a new spreadsheet or tab.
Conclusion
Note-taking is a learned skill that may take some time to master. With practice and collaboration between researchers, and a few tips and tricks, effective note-taking can become second nature.
What best practices do you use while note-taking? We’d love to hear from you!
Contact us to learn more about conducting effective usability studies and how we can assist your human factors needs.
