Undo and Erase Events as Indicators of Usability Problems

Source:
"Undo and Erase Events as Indicators of Usability Problems" by David Akers, Matthew Simpson, Robin Jeffries, Terry Winograd. CHI 2009

Summary
The authors of this paper wanted to study negative critical incidents which will show usability problems with Google SketchUp. They differentiated event-based reporting and self reporting. In this experiment, they used undo and erase operations for the event-based reporting and compared it with self reporting by the user. In the formal experiment, they had 35 participants from a variety of backgrounds. The experiment was 90 minutes and was separated into training to use SketchUp and identifying critical events, practice using the program, performing the modeling assignment, and retrospective commentary. For the assignment, the users were given specific instructions and told not to deviate from the instructions, and the commentary was moved to after the assignment was over because previous work noted that immediate commentary was disruptive to the process. The commentary portion included a series of questions about the critical events detected automatically (undo and erase operations) and/or self reported. The questions asked what was happening immediately before the event occured, was the problem surprising, what did the user do to get around the problem, and also if they reported it as a problem and if not, why. The commentary session paired off users as a speaker and listener. The listener would ask the question and judge when the question had been answered, and the speaker would talk about the event.

Results
The events were assigned severity based on how many operations detected it. Mild severity events were detected due to self reporting. Undo operations indicated medium severity. If all three methods were used, then the event would be catagorized as severe. Also they catagorized events based on frequency of the event and the impact of the event (whether or not the event would completely stop all work). After analyzing the data, the authors determined that undo and erase events detected about 90% of all severe usability problems. Similar numbers for the other catagories existed. Figure 1 shows a venn diagram of the results. Of particular note is the events detected only by undo or erase. When asked why the user didn't report the issue, many users mentioned that they blamed themselves and not the program, which is similar to what Don Norman said in "The Design of Everyday Things."


Figure 1
Analysis
I believe that this experiment showed some interesting results. I knew that undo and erase operations can show simple mistakes, but I never associated that they would use those operations to deal with usability problems. In hind sight, it made sense. I think that this experiment should be repeated with other programs as part of a usability test to see how to better design the interface to reduce the number of problems.