Collecting High-Quality Interview Data
User Experience Researchers, ideally, run a ton of studies. Whether it’s qualitative or quantitative, formative or summative, many things can harm the quality of the data you collect. The following is a list of 10 tips to help you prevent that from happening:
1. Run a pilot
No matter how much preparation you and your team have done, you will encounter problems once the participants walk in the door. A good-ole-fashioned trial run to work out the kinks can’t be beat. It will not only give the moderator a chance to practice but the notetakers as well, and it’ll help to ensure that you have all the necessary supplies for a full session. Take it seriously and treat it like the real thing if you can, especially if you’re rusty.
2. Manage the length of sessions
It’s important to keep you and (more importantly) your participants sharp for the duration of each session, so you have to walk the line between too long and too short. Obviously, the whole reason you’re conducting the study is to collect data, so a 15-minute session isn’t going to work. Start by creating a session that gets what you need, then go from there. If the session turns into a 3-hour marathon, you may need to add breaks or trim where possible.
There are three factors to keep in mind:
- The participant’s comfort – Go too long, and the participant will get fatigued and will hate you for keeping them for so long. Regardless of if you’re paying them, they’ll lose sight of that and they won’t be sharp. Fatigued participants = Bad data.
- Your own comfort/sanity – If the participant is fatigued, I bet you are, too. More on this later.
- Keeping up with the schedule – Whatever length you’ve scheduled the sessions for, do your best to stick with that. Going long isn’t a huge deal if you’re getting good data, but it can disturb the quality of data down the line and things are rushed at the end of the day.
3. Know when to shut up
Some participants will give short replies to your questions. As moderators, it’s our job to probe those participants who are less-than-talkative but be careful. A lot of the time, it’s best to simply remain quiet and observant as the participant may have a thought that just hasn’t been verbalized yet. Socially, remaining silent feels slightly awkward, but your participant is likely to elaborate given extra time and some silence. The material that follows is the sweet-spot of rich data you’re after. Give it time, grasshopper.
4. Know when to speak up
Being quiet is a good rule of thumb… most of the time. As you interact and develop a connection with your participant, you’ll start to develop a sense of when the participant is searching for the next move, and when he or she is truly stuck. Asking what the participant expects to see or what they are thinking at that moment can be very useful. Once you’ve gathered that information, it might be necessary to provide gentle prods in the right direction.
There’s also the chance that the participant just doesn’t know what to say or what to talk about. A few carefully selected, open-ended questions can send the discussion in a helpful direction. It’s useful to have a team refresher discussion before the study begins to have a small bank of go-to open-ended questions you can rely on when the going gets tough.
5. Know when to move on
Again, sometimes the participant is just stuck and probing isn’t the best strategy. For the sake of time, it can be best to just move on. In an ideal world, you take your time and let the participant continue to work. There is always knowledge to be gained, but circumstances may dictate that you carry on.
You may also run into a different problem: the participant has plenty to say about the product. Especially for first-time usability participants, it’s very easy to become fixated on some obscure detail because they are trying to be helpful. The fact that they have so much feedback is great, but you don’t want to spend too much time on any one thing. Find an opening and redirect to details you’re more interested in getting their thoughts on.
6. Know the product you’re testing
Sometimes you never know how an interview might progress. It’s very possible the system you’re testing encounters a problem or the participant inadvertently changes a setting. Stay sharp and pay attention to these possible scenarios; because if it happens, the moderator is responsible for ensuring the system runs smoothly and providing consistent conditions for each participant.
7. Establish a line of communication between moderator and notetaker
If you’re like us and typically work as a team, listen up. Whenever possible, figure out a way for notetakers, stakeholders, clients, and other team members behind the scenes to communicate with the moderator. People observing from behind the two-way glass may notice something the moderator missed; they must have the ability to communicate that message to the moderator discreetly.
In order to keep communication subtle, we have found that chat systems work well. To keep the participant from thinking we’re rudely texting or typing on their time, we usually go ahead and let them know we are using the chat for study purposes. Alternatively, Google documents/sheets enable not only chat but allow you to write messages over the GoogleDrive cloud. With this tool, you can leave notes in specific places (along with the script or moderator worksheet, for instance) for the moderator to find.
8. Backup your note taker (Record the session)
Even the best notetakers miss the occasional detail, whether it’s typing, handwritten, or audio notes. Audio and video record your studies so that you can look back and grab missed details on the back-end.
It’s also great for presentation purposes later on. If you’re expected to deliver a report of some kind, screenshots and specific examples of your findings will be super helpful.
9. Check your external validity
It’s often beyond our means to test something out in the field. Whether it’s due to budget, time, or safety, many studies are run in a laboratory setting. Simulations like these are great because they save you resources, but the research is worthless unless the simulation adequately represents the real world. For example, if the system you’re testing will typically be used under conditions of high stress and pressure, don’t test the participants in a calm environment! They may be able to complete the tasks with ease now, but once situational pressure is in the picture, performance may suffer. Make your study as realistic as possible, given the circumstances.
10. Take care of yourself
Just as you want to keep your participants happy and fresh, you’ve got to take care of yourself. If you’re someone who needs a 10-minute break after a couple hours, take a 10-minute break. The data collected will absolutely suffer if you’re tired and just want to get out of there. Ensure that you’re fresh and rested so that you’re able to shut up, speak up, and move on when you need to, and with a positive attitude.
