How many participants do you need?
You’re running a study. You’ve got everything you need in place. The study plan, the moderator scripts, your data collection sheets are all ready. But how many participants do you need to do it right?
Well, we could jump on into G-Power and calculate your n based on the estimated effect size, but that might not be necessary. We are in the world of industry, where small, iterative test sessions glean the information we need at the speed we need it. Conversely, we don’t want to perform a “Mom Test”[1], relying on the results of one person. What is the answer?
As a rule of thumb, we can use between 5 and 15 people.
Background
In 1993, Jakob Nielsen and Thomas Landaurer [2] described how many participants are necessary to uncover usability issues. Loosely stated, for each participant tested, a certain number of total usability issues in a system will be found. Of course, there is a limit. The model they created is essentially a curve that exhibits diminishing returns after a certain number of participants are tested. For good measure, a paper published by Laura Falkner in 2003 [3] verified this model.
Still, why between 5 and 15? Based on Falkner’s research, if you were to test 5 participants, you would catch approximately 80% of the usability issues. Similarly, 15 participants are able to identify an average of approximately 97% of the issues. Notably, the FDA requires medical device validation usability studies to run at least 15 participants [4]. Coincidence? Definitely not.
Conclusion
Remember that these numbers only work when a study is qualitative in nature. When we are talking about quantitative studies that allow us to report meaningful statistics, we’ll want to abide by the traditional sampling rules taught in statistics courses. With that said, we can gain a lot of insight from a small group of people, and we can do that quickly, on a budget, and more frequently.
For more Medical Human Factors and User Experience resources, check out our blog and YouTube channel.
References
[1] Fitzpatrick, R., & Fitzpatrick, R. (2014). The Mom Test: How to Talk to Customers and Learn If Your Business is a Good Idea when Everyone is Lying to You. CreateSpace Independent Publishing Platform.
[2] Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems (pp. 206-213). ACM.
[3] Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379-383.
[4] US Food and Drug Administration. (2016). Applying human factors and usability engineering to medical devices. Guidance for industry and Food and Drug Administration staff.
