Tuesday, March 20, 2007

User Research Doesn’t Prove Anything

Using scientific methodology in user research ...

"Recently, I was reading through a sample chapter of a soon-to-be-published book. The book and author shall remain nameless, as shall the book’s topic. However, I was disappointed to read, in what otherwise appeared at first glance to be an interesting publication, a very general, sweeping statement to the effect that qualitative research doesn’t prove anything and, if you want proof, you should perform quantitative research. The author’s basic assumption was that qualitative research can’t prove anything, as it is based on small sample sizes, but quantitative research, using large sample sizes, does provide proof.

This may come as a shock to everyone, but quantitative research does not provide proof of anything either.

Here, I’m using the word proof in the mathematical sense, because that is the context within which the author made those statements. In mathematics, a proof is a demonstration that, given certain axioms, some statement of interest is necessarily true. The important distinction here is the use of the word necessarily. In user research, as with all avenues of statistical inquiry, we’re able to demonstrate only that a hypothesis is probably true—or untrue—with some specific degree of certainty.

Granted, I’m being pedantic; and you might think this just an interesting exercise in semantics. But let me take you through a brief survey of this topic, then perhaps you’ll appreciate the importance of this distinction.

Statistical Sampling
In general, our user research activities involve working with a small subset of our overall audience of users, to

• gather information about a particular topic
• test users’ response to some feature of our design solution
• measure an increase or decrease in the efficiency of performing a certain task
• or some other similar goal

The size of the entire audience prohibits us from involving all of our users in our research activities.

Our first step is to select our sample from the total population of users. If we’ve done that successfully, our sample should reflect, as closely as possible, the composition of the full user population in the user characteristics that matter for our research."    (Continued via UXmatters)    [Usability Resources]


Post a Comment

<< Home

<< Home