Analyzing and Reporting Usability Data

The Just-In-Time (JIT) method of data analysis has the virtue of immediacy, rapid turn-around, and team involvement; however there are several disadvantages. First, this type of analysis is problem-focused, rather than goal-focused. Long lists of problems are generated, but there is no clear relation to specific usability goals. Second, developers may not be able to fix things immediately so the context of the problem may be lost when it is time to fix the problem. Third, the JIT analysis requires that the entire development team observe the testing sessions since problems may occur that are the responsibility of different developers.

Wilson, Chauncey E. Usability Interface (1997). Articles>Usability>Testing>Reports

Better Reports: How to Communicate the Results of Usability Testing  

You've spent several days setting up a usability test, recruiting the participants and running it. Then you've pored over the data. What next? If you are doing usability testing as part of user-centred design within a business setting, then there are many ways that you can communicate the results. This paper looks at reports and then considers presentation and observation as alternatives to reports.

Jarrett, Caroline. STC Proceedings (2004). Articles>Usability>Testing>Reports

Communicating the Results of Field Studies to Support Usable Design  

When you have completed the study, analyzed the data, and organized the interpretations and conclusions along with supporting data, you have to communicate the results to the people who need to know about them. How you communicate the results depends upon who the intended audience is, content needs of the audience, and the scope of the content. Increasing the odds of this information being used in the design process requires an understanding of the company's culture and the barriers limiting its use in the development process. Various strategies such as computer-slide presentations, reference notebooks, bound reports, and memos have been shown to be very effective in various circumstances.

Carlevato, Denise, Judith A. Ramey and Erin Leanne Schulz. STC Proceedings (1996). Articles>Usability>TC>Reports

Formal Usability Reports vs. Quick Findings

Formal reports are the most common way of documenting usability studies, but informal reports are faster to produce and are often a better choice.

Nielsen, Jakob. Alertbox (2005). Articles>Usability>Reports

How to Write a Report Without Getting Lynched

You put forth your best effort to explain to the stupid sods exactly how and where they screwed up, then they have the temerity to not appreciate your fine efforts. Here's how to write a report that will cause change, instead of uproar.

Tognazzini, Bruce. Nielsen Norman Group (2001). Articles>Usability>Reports>Technical Writing

Industry Usability Reporting and the Common Industry Format (ANSI-NCITS 354-2001)

The Common Industry Format (CIF) is a standard method for reporting usability test findings. The format is primarily for reporting results of formal usability tests in which quantitative measurements were collected and is particularly appropriate for summative/comparative testing. The CIF targets two audiences: usability professionals and stakeholders in an organization. Stakeholders can use the usability data to help make informed decisions concerning the release of software products or the procurement of such products. While the CIF is formally aimed for software products, it can be used for hardware usability as well.

Parush, Avi and Emile Morse. Usability Professionals Association (2003). Articles>Usability>Standards>Reports

Making Usability Recommendations Useful and Usable    

This paper evaluates the quality of recommendations for improving a user interface resulting from a usability evaluation. The study compares usability comments written by different authors, but describing similar usability issues. The usability comments were provided by 17 professional teams who independently evaluated the usability of the website for the Hotel Pennsylvania in New York. The study finds that only 14 of the 84 studied comments (17%) addressing six usability problems contained recommendations that were both useful and usable. Fourteen recommendations were not useful at all. Sixteen recommendations were not usable at all. Quality problems include recommendations that are vague or not actionable, and ones that may not improve the overall usability of the application. The paper suggests characteristics for "useful and usable recommendations," that is, recommendations for solving usability problems that lead to changes that efficiently improve the usability of a product.

Molich, Rolf, Robin Jeffries and Joseph Dumas. Journal of Usability Studies (2007). Articles>Usability>Assessment>Reports

Standards Update: Usability Test Reporting

It’s a truism that even a bad usability test will help improve your software. But the findings from different usability tests are notoriously difficult to compare. This makes it difficult to track usability improvements or to see how you compare against an earlier product. An emerging international standard looks set to solve this problem.

Userfocus (2003). Articles>Usability>Reports

A Structured Process for Transforming Usability Data into Usability Information  

Much research has been devoted to developing usability evaluation methods that are used in evaluating interaction designs. More recently, however, research has shifted away from evaluation methods and comparisons of evaluation methods to issues of how to use the raw usability data generated by these methods. Associated with this focus is the assumption that the transformation of the raw usability data into usability information is relatively straightforward. We would argue that this assumption is incorrect, especially for novice usability practitioners. In this article, we present a structured process for transforming raw usability data into usability information that is based on a new way of thinking about usability problem data. The results of a study of this structured process indicate that it helps improve the effectiveness of novice usability practitioners.

Howarth, Jonathan, Terence S. Andre and Rex Hartson. Journal of Usability Studies (2007). Articles>Usability>Reports>Technical Writing

Towards the Design of Effective Formative Test Reports    

Many usability practitioners conduct most of their usability evaluations to improve a product during its design and development. We call these 'formative' evaluations to distinguish them from 'summative' (validation) usability tests at the end of development. A standard for reporting summative usability test results has been adopted by international standards organizations. But that standard is not intended for the broader range of techniques and business contexts in formative work. This paper reports on a new industry project to identify best practices in reports of formative usability evaluations. The initial work focused on gathering examples of reports used in a variety of business contexts. We define elements in these reports and present some early guidelines on making design decisions for a formative report. These guidelines are based on considerations of the business context, the relationship between author and audience, the questions that the evaluation is trying to answer, and the techniques used in the evaluation. Future work will continue to investigate industry practice and conduct evaluations of proposed guidelines or templates.

Theofanos, Mary Frances and Whitney Quesenbery. Journal of Usability Studies (2006). Articles>Usability>Testing>Reports

Usability Report Tips

Have you ever wondered about reports of usability tests? How much time does it take to write one? What should you keep in mind when designing and writing the report? Here are some rules of thumb that I use.

Wilson, Chauncey E. Usability Interface (1997). Articles>Usability>Reports