Monday, October 15, 2007

Multiple-User Simultaneous Testing: MUST

Usability testing with large numbers of users ...

"Testing 5-10 users at once lets you conduct large-scale usability testing and still meet your deadlines.

Sometimes you need to test a large number of users. One option, of course, is to apply the standard user-testing methodology, and just do more of it. Keep testing until you're blue in the face. Unfortunately, this often gets you into serious trouble with project deadlines.

Alternatively, you can use multiple-user simultaneous testing, or MUST. As the name indicates, with MUST, you test multiple users at the same time so you get done sooner. In most MUST studies, we test 5-10 users at once (but, as I describe below, you can also set up labs with many more test stations). Theoretically, there's no upper limit to the number of users you can test in each session.

When to Use MUST
Most usability studies should be simple and small-scale, but in some scenarios, it's useful to conduct MUST:

* For quantitative studies and benchmarking, you typically need to test at least 20 users per condition in order to get statistical significance.

* For long-duration tasks, you need to test each user for days or weeks to observe valid behaviors. Examples here include:

o Developer tools. You can't test a system to support professional programmers by having users develop and debug a 20-line "hello world" program; users must work through an industrial-scale problem. The same is true for other high-end problem-solving applications, such as CAD.

o E-learning. You can't test lesson 39 unless the learners have first made it through lessons 1-38. For a sufficiently advanced e-course, each test could take a week or more.

* Usability focus groups. To alleviate the problems with traditional focus groups, each participant should start with a one-on-one session testing the live user interface. Following the test sessions, participants can then congregate to discuss the experience and how it relates to their everyday needs. This method definitely requires MUST because all participants should test the interface just before the focus group meets.

* Games design. I describe this case in detail below.

How to Run Many Users Simultaneously
When testing many users, you usually need many test facilitators. The main exception is for the long-duration tests, where a few facilitators can circulate among users and/or review video recordings of critical incidents.

If you're one of the lucky few companies with many usability specialists, they can facilitate MUST sessions. This is expensive, but efficient: All you need to do is turn the experts loose, since they already know how to run a study.

Most companies, however, don't have enough usability pros to assign one to each test user. Happily, non-usability staff can run user test sessions, especially if a seasoned usability expert has prepared the test plan and written the tasks.

For our latest MUST study, we hired cognitive science students from Don Norman's former department at the University of California, San Diego; they were excellent facilitators. In other studies, we've used developers and marketers from the project team. Being responsible for a MUST session is a great way for such team members to get intense exposure to customers."    (Continued via Jakob Nielsen's Alertbox)    [Usability Resources]


Post a Comment

<< Home

<< Home