Usability Techniques: Analyzing and Reporting Usability Data

There is an ongoing discussion in usability circles about the importance of formal reports that document the results of usability testing. I think that each usability evaluation should have a formal report that provides some context for the problems. Not all problems can be addressed immediately and memories fade. Usability reports are also important for showing what a usability specialist has done. They can also be used to determine some metrics, such as the number of problems addressed by development or the number of problems that occurred during successive prototypes or versions of a product.

Wilson, Chauncey E. STC Usability SIG (1997). Articles>Usability>Testing>Methods

Usability Test Data

People often throw around the terms 'objective' and 'subjective' when talking about the results of a usability test. These terms are frequently equated with the statistical terms 'quantitative' and 'qualitative'. The analogy is false, and this misunderstanding can have consequences for the interpretations and conclusions of usability tests.

Hodgson, Philip. Userfocus (2003). Articles>Usability>Testing>Methods

Usability Testing Best Practices: An Interview with Rolf Molich

If you’ve done any usability testing, design evaluations, or heuristic inspections, then you’ve been affected by Rolf Molich's pioneering work.

Perfetti, Christine. User Interface Engineering (2003). Articles>Usability>Methods>Testing

Usability Testing: 8 Quick Tips for Designing Tests

This document is intended to help beginners design questions to help them conduct a good usability testing session. If you already have a prototype you want to test, you've already drafted a few questions, and you're eager to learn how to make the most of your opportunity to learn from your users, then this document is for you.

Jerz, Dennis G. Seton Hill University (2002). Articles>Usability>Methods>Testing

Usability Testing: Assess Your Site's Navigation and Structure  

Usability is literally the 'ease of use' or understanding it takes to make something work. In this case, Web Site usability is the understanding of how an individual user navigates, finds information and interacts with your Web Site. Unlike online surveys or focus groups, usability testing is a oneon- one process in a 'watch and learn' approach. The results of the sessions are used to improve your user’s experience. Having the development team watch the testing and witness the results helps resolve most internal issues in an undisputed manner. You can’t fight the reality of usability testing.

Goto, Kelly. GotoMedia (2000). Articles>Usability>Methods>Testing

Usability Testing: Don't Let the Myths Put You Off  

Jarrett dispels several myths about usability testing that may dissuade technical communicators from applying valuable usability techniques.

Jarrett, Caroline. Intercom (2003). Articles>Usability>Methods>Testing

Usability Testing: Getting It Right The First Time  

User-centered product design is a design approach that focuses on the users’ job tasks, skills, and abilities. Usability testing has emerged as a critical component in the user-centered design process to assure that a product meets the needs of the user. Implemented correctly, usability testing can increase customer satisfaction and acceptance, improve product image, and reduce development costs. A variety of information is available to help you get started in developing a usability testing process. This workshop will provide sample questionnaires, checklists, scenarios, scripts, etc. However, the main focus of the workshop will be to train participants in the following two areas: (1) writing measurable usability test goals; and (2) collecting and interpreting the test data. These activities are critical because they relate directly to the usefulness of the test results.

Cope, Michael and Kevin Uliano. STC Proceedings (1994). Articles>Usability>Methods>Testing

Usability Testing: What is it?

Better-written technical documents enable people to work with greater speed, recall, accuracy, and comfort. These qualities, when taken together, make up the usability factor. This document suggests some ways you might conduct tests to measure the usability of your technical documents. It covers the kind of data you should collect, how many test subjects you need, and how you should treat those subjects.

Jerz, Dennis G. Seton Hill University (2000). Articles>Usability>Methods

The Use and Misuse of Focus Groups

Focus groups often bring out users' spontaneous reactions and ideas and let you observe some group dynamics and organizational issues. You can also ask people to discuss how they perform activities that span many days or weeks: something that is expensive to observe directly. However, they can only assess what customers say they do and not the way customers actually operate the product. Since there are often major differences between what people say and what they do, direct observation of one user at a time always needs to be done to supplement focus groups.

Nielsen, Jakob. Useit (1997). Articles>Usability>Methods>Focus Groups

User Interviews: A Basic Introduction

A basic introduction to user interviews, a great way to build research on your users and help improve the usability of your site.

Gray, Alistair. Webcredible (2008). Articles>Usability>Interviewing>Methods

User Testing vs. Expert Evaluation

It is something of a given in the usability industry that testing with real users is the most effective way of identifying interface issues that may cause usability problems. Whilst it is true that testing is the best way to get real feedback on user experience with a site or application, there is an argument that evaluation by an expert is an equally efficient way to identify these problems. For some reason, the process of expert evaluation in usability circles has acquired the name 'heuristic evaluation', a term that is largely inaccurate in this context and certainly confusing for those unfamiliar with the jargon. However, that is a different story. Here we are more concerned with the pros and cons of expert evaluation as compared with user testing.

Gaine, Frank. Frontend Infocentre (2001). Articles>Usability>Methods

User-Testing Techniques - Site Reviews

Accurately measuring reader-friendliness is a complex task for any Website. Usability testing techniques have been applied with some success to Web pages. But formal usability tests are difficult to set up, and very expensive to conduct (often prohibitively so). And the diversity of the audience of Internet sites probably limits the utility of formal usability testing of many Web pages. Happily, inspection-based usability evaluation is generally as effective as formal, experimental testing, and it is much easier (and less costly) to conduct. This series focuses on inspection-based usability assessment. A site review is basically an unstructured inspection by a third party, typically focusing on the site's usability and aesthetics. It's a difficult but worthwhile effort for serious-minded authors to seek out reviewers who are willing to invest the time to undertake a serious review of your site, offering constructive suggestions for improvement.

Sullivan, Terry. All Things Web (1996). Articles>Usability>Methods>Web Design

Using a Survey to Help Plan and Develop Your Document  

Surveys can be valuable tools for developing your document. We used our survey to answer many of our user and task analysis questions before planning content for different service manuals. When creating and using a survey, identify what information you need to know about your users, consider your options for mailing the surveys, plan a smooth processor gathering, tallying, and analyzing the data, and then remember to report back to your respondents. Ultimately, your findings can be incorporated into a recommendation report for your clients, as well as help you plan the content and layout for your document.

Lester, Susan M.J. and Lauren Y. Hansen. STC Proceedings (1996). Articles>Usability>Methods>Writing

Using an Ethnographic Method to Gather Usability Data from the Field  

Observation is a way to gather rich information about how users work with software products that also provides a clearer understanding of the users' work. The method consists of watching users performing their normal work routine where they work. Observers can be usability professionals or trained individuals from the company. The richness and type of data collected can be used to identify design opportunities for the next release, define usability goals for all products, and create realistic customer scenarios.

Carlevato, Denise. STC Proceedings (1996). Articles>Usability>Ethnographies>Methods

Using Personas: Bringing Users Alive

How do we communicate what we know about the people who use our products in an engaging, efficient way? How do we get beyond statistics to a portrait of users that helps us use this information to make decisions?

Quesenbery, Whitney. Usability Interface (2003). Articles>Usability>Methods>Personas

Using Site Evaluations to Communicate with Clients

How do you prove your worth to clients in today's difficult economy? Performed as part of a sales proposal or the discovery phase of a project, a site assessment can uncover opportunities for improvement and help you speak knowledgeably about solutions to your potential client's problems.

Rabinowitz, Dorelle. Boxes and Arrows (2003). Articles>Usability>Methods

Using the 5Es to Understand Users

One of the exercises I find helpful is to look at usability requirements for different aspects of the user experience. For each of the five dimensions of usability (the 5Es), we think about how it is reflected in requirements for each of the user groups.

Quesenbery, Whitney. WQusability. Articles>Usability>Methods

Utiliser les 5 E pour Comprendre les Utilisateurs

En matière d'amélioration de votre site web, produit ou logiciel, comment passer de la simple volonté à l'action? Que dites-vous de ceci pour commencer: si la réponse c'est l'utilisabilité, quelle peut bien être la question?

Quesenbery, Whitney. WQusability. (French) Articles>Usability>Methods

Voodoo Usability

The good news is that usability has been recognized as an important element of Internet success: the average speaker at industry conferences now promotes good user experience in preference to 'cool sites.' The bad news is that most sites employ horribly misguided methodologies that do not assess real usability. Sometimes the methods are simply worthless; other times they are directly misleading.

Nielsen, Jakob. Alertbox (1999). Articles>Usability>Methods

Want Free Beer?

Whether you are testing the User Interface for a new technology or just re-branding your service, chances are that you could benefit from some sound market feedback. The good news is that you don't have to spend weeks on research or thousands of dollars to get it. Café testing - quick, low-cost, informal market testing at a café - can help you get the feedback you need fast. This article tells you everything you need to know to get started.

Burns, Erik. GotoMedia (2004). Articles>Usability>Methods>Focus Groups

What's Going On? The Importance Of User Feedback

It is a basic assumption on the web that download times should be kept to a minimum. After all, the longer people are left waiting, the more likely they are to back out (quite literally) of a transaction and look elsewhere for the same service. But despite the general accuracy of this point of view, there is perhaps slightly more to it than this. What frustrates users about slow download times is not so much the wait as the uncertainty. The online environment remains somewhat unreliable - certainly unreliable enough for a user to be unsure that every page request is likely to be successful. In this situation, the lack of ability of the browser to accurately reflect progress is a serious issue in terms of web effectiveness and user satisfaction.

Farrell, Tom. Frontend Infocentre (2000). Articles>Usability>Methods

When 100% Really Isn't 100%: Improving the Accuracy of Small-Sample Estimates of Completion Rates    

Small sample sizes are a fact of life for most usability practitioners. This can lead to serious measurement problems, especially when making binary measurements such as successful task completion rates (p). The computation of confidence intervals helps by establishing the likely boundaries of measurement, but there is still a question of how to compute the best point estimate, especially for extreme outcomes. In this paper, we report the results of investigations of the accuracy of different estimation methods for two hypothetical distributions and one empirical distribution of p. If a practitioner has no expectation about the value of p, then the Laplace method ((x+1)/(n+2)) is the best estimator. If practitioners are reasonably sure that p will range between .5 and 1.0, then they should use the Wilson method if the observed value of p is less than .5, Laplace when p is greater than .9, and maximum likelihood (x/n) otherwise.

Lewis, James R. and Jeff Sauro. Journal of Usability Studies (2006). Articles>Usability>Testing>Methods

When Observing Users Is Not Enough: 10 Guidelines for Getting More Out of Users' Verbal Comments

One of the principles underlying usability testing is that observing a user perform a task provides more reliable information than simply asking the user how easy it would be to perform the task. By observing users, you can assess whether they are actually able to use a product. By asking them, you simply cannot.

Peyrichoux, Isabelle. UXmatters (2007). Articles>Usability>Testing>Methods

When the Field is Far Afield: Multiple-Country Observations of Complex System Use  

Describes the challenges of usability studies of complex systems that are used internationally, based on a case study of a multinational company's enterprise-wide call management system.

Anscheutz, Lori, Deborah Hinderer and Janice Anne Rohn. Tec-Ed, Inc. (1998). Articles>Usability>Methods>International

When To Introduce Usability

On the basis that user-centred design, user testing and other forms of usability engineering are a 'good thing', it is worth asking how they are best integrated into the development process. Specifically, the issue of timing is critical to the successful implementation of a usability strategy. At what stage of development should these techniques be introduced? Of course the answer to that question rather depends on the unique circumstances of each particular project, but the general principle seems to be 'as early as possible' - with the important caveat that any user involvement is better than none, even at a late stage in the day. But the accepted wisdom that usability engineering should be introduced as early as possible in the development process has many arguments in its favour.

Farrell, Tom. Frontend Infocentre (2000). Articles>Usability>Methods