Monday, March 03, 2008

Measuring satisfaction: Beyond the usability questionnaire

Measuring satisfaction ...

"Measuring user satisfaction

A common mistake made by novice usability test moderators is to think that the aim of a usability test is to elicit a participant's reactions to a user interface. Experienced test moderators realise that a participant's reaction is just one measure of usability. To get the complete usability picture, we also need to consider effectiveness (can people complete their tasks?) and efficiency (how long do people take?).

These dimensions of usability come from the International Standard, ISO 9241-11, which defines usability as:

"Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."

The ISO definition of usability makes it clear that user satisfaction is just one important dimension of usability. People may be well disposed to a system but fail to complete business-critical tasks with it, or do so in a roundabout way. The three measures of usability — effectiveness, efficiency and satisfaction — are independent (PDF document) and you need to measure all three to get a rounded measure of usability.
Importance of collecting satisfaction measures

A second mistake made by people new to the field of usability is to measure satisfaction by using a questionnaire only (either at the end of the session or on completion of each task). There are many issues to consider when designing a good questionnaire, and few usability questionnaires are up to scratch.

For example, we've known for over 60 years that you need to avoid the "acquiescence bias": the fact that people are more likely to agree with a statement than disagree with it (Cronbach, 1946). This means that you need to balance positively-phrased statements (such as "I found this interface easy to use") with negative ones (such as "I found this interface difficult to navigate"). So it's surprising that two commonly used questionnaires in the field of usability — the Usefulness, Satisfaction, and Ease of use (USE) questionnaire and the Computer System Usability Questionnaire (CSUQ) — suffer from just this problem: every question in both of these questionnaires is positively phrased, which means the results from them are biased towards positive responding.

Questionnaires that avoid this source of bias often suffer from other sources of bias. For example, few undergo tests of reliability. This means that the same questionnaire may yield different results at different times (this can be checked by measuring the questionnaire's test-retest reliability). Even fewer usability questionnaires are assessed for validity. This means that there is no guarantee that the questionnaire actually measures user satisfaction.
Problems with measuring satisfaction

In our studies, we notice that participants tend to rate an interface highly on a post-test questionnaire even when they fail to complete many of the tasks. I've spoken to enough of my colleagues at conferences and meetings to know that this problem is commonplace. Is this because we are about to give the participant £75 for taking part in a test session or is there something else at work? For example, one group of researchers makes this point:

"In studies such as this one, we have found subjects reluctant to be critical of designs when they are asked to assign a rating to the design. In our usability tests, we see the same phenomenon even when we encourage subjects to be critical. We speculate that the test subjects feel that giving a low rating to a product gives the impression that they are "negative" people, that the ratings reflect negatively on their ability to use computer-based technology, that some of the blame for a product's poor performance falls on them, or that they don't want to hurt the feelings of the person conducting the test." - Wiklund et al (1992).

Once you ask participants to assign a number to their experience, their experience suddenly becomes better than it actually was. We need some way of controlling this tendency."    (Continued via Info Design, UserFocus)    [Usability Resources]


Post a Comment

<< Home

<< Home