Usability Evaluation Considered Harmful?

commented on
Brian Salato
Lei Gu


Source
"Usability Evaluation Considered Harmful (Some of the Time)" by Saul Greenberg and Bill Buxton CHI 2008.

The main point this article was trying to make is that usability evaluations, when performed without thinking it through, can be detrimental to creative design or innovative technology. They believe that the HCI community is negatively affected by this and proposed several ways to fix it.

First, usability tests is not the only method when performing user centered design and should be used when the problem and stage of user interface development warrants it.

Second, there are some times when usability testing is would not produce useful information, such as at the very beginning of design, when usefulness outweighs usability, and where the unpredictable culture affects how an innovative system is used.

Next, there are other ways other than using usability evaluations to validate the work, such as rationalizing the design, expected usage, case studies, and participant critiques. One of his complaints with CHI is that the conference specifies using a usability study, and in particular, a quantitative study. They believe that submissions should be judged based on the question, the system or situation described, and whether the inventors used a reasonable method to argue their points.

Fourth, the method of testing usablity tends to result in weak science. For novel innovations, they should use an existence proof, where they show one case where it could be use. For variations of already existing products, risky hypothesis testing is appropriate. Finally, there should be more help for others to replicate the results.

Lastly, they argue that HCI should look to other fields to determine worthiness. Design teams need to be able to differentiate between sketches, early designs, which are ideas that can quickly be changed, and prototypes, which are more finalized and thoughtout.

Commentary
I agree on some of the points. There are some things that are extremely useful but are designed poorly. I don't think they should be discarded. To paraphrase what I read in another paper, if the designers explain sufficiently why they designed something the way they did, it would be acceptable by the public.