When Role Playing Doesn’t Work: Seven Guidelines for Grounding Usability Testing in Participants’ Real Lives
“Assume you live in San Francisco and you are looking for a hotel for a friend who is visiting you. You type hotel in San Francisco into a search box and get these results. Have a look at the results and select three hotels.”
Usability testing makes use of a lot of role-playing scenarios like this one, and many findings and design recommendations result from participants’ responses to these scenarios. But an over-reliance on role playing when testing a product and making design recommendations can have major downsides and risks, including the following:
* identifying false usability issues and user needs that lead design iterations in the wrong direction and result in poor user experiences—potentially leading to users’ abandoning a product and revenue losses for a company
* overlooking serious usability issues
* losing opportunities to gain important insights
* calling the reliability and quality of study findings into question
This article presents some common limitations and downsides of role-playing in usability testing and provides guidelines for avoiding them by grounding usability testing in participants’ real lives. All of these guidelines come from my experience in user research—mainly from testing Web sites—but these guidelines extend to all types of product usability testing. What makes the difference is the participants’ level of engagement with the product being tested, not the product itself.
These guidelines are especially applicable to qualitative studies, where the goal is to understand whether participants are able to achieve a goal or perform a task they have in mind, see what problems they encounter, and identify the mental processes behind their behaviors—why they do what they do. Some of these guidelines might be less applicable if you need to collect strict performance data and statistical measures of usability—for example, success rates or times to complete a task—or if you need to resolve very simple usability issues. These guidelines are also better adapted to formative studies than summative ones.
However, whatever type of usability testing you are doing or type of product you are testing, it is always important to consider whether the artificiality of the testing conditions might bias your findings and to what extent you should ground usability testing in participants’ real lives.
1. Recruit passionate users.
By recruiting passionate users, I mean recruiting participants who
* use the type of product or content you are testing in their real lives
* are truly interested in the product or content
* have an emotional connection to it
Recruiting participants who typically use the type of product or content you are testing in their real lives may seem obvious, but it is easy to overlook this when you are absorbed in the recruiting process and confronted with external deadlines and constraints on recruiting. This mistake is especially easy to make when testing a product or Web site that is somewhat specialized, but pretty easy for anyone to understand—for example, a dating, movie, or travel Web site.
Example: Testing a Dating Web Site
When testing a dating Web site, one might think: Oh, pretty much everybody knows about dating Web sites or has used one at some point, so let’s just ask the participants we’ve recruited for this other Web site to pretend they’re looking for a mate. If you’re testing common usability issues like the discoverability of a data refinement menu, this approach can work. The personal engagement and interest of participants in the activity of dating online would have very little impact on the discoverability of the menu.
However, if you are testing user perceptions of the subscription process—does it take too long to subscribe and create a profile—involving participants who are not interested in dating could be misleading. The emotional involvement a participant has in the process of finding a mate online can have a big influence on a participant’s perception of the process. If finding a mate is crucial to a participant, he might be ready to invest a lot of time and energy in it. His perception of the length of the process might be very different from that of someone who is not emotionally involved in the process. What one person might find unendurably long, the emotionally involved participant might find okay—if the process is so important to him, he’s ready to go through it.
It is simply impossible to put yourself in the shoes of somebody who is emotionally involved in the process of looking for a mate if you are not. You can’t predict how a person who is emotionally involved would react. Participants don’t need merely to understand the content; they need to be emotionally involved in the process." (Continued via UXmatters. Isabelle Peyrichoux) [Usability Resources]