Accentuate the Negative: Obtaining Effective Reviews Through Focused Questions    

How you ask a question strongly determines the type of answer that you will obtain. For effective documentation reviews, whether they are conducted internally or externally as part of usability testing, it's important to use precise questions that will provide concrete information on which to base revisions. This paper proposes an approach to obtaining useful feedback that emphasizes negative, 'what did we do wrong?' questions. This approach focuses limited resources on areas that need improvement rather than areas that already work well and that don't require immediate improvement.

Hart, Geoffrey J.S. Technical Communication Online (1997). Articles>Usability>Methods>Testing

Analyzing and Reporting Usability Data

The Just-In-Time (JIT) method of data analysis has the virtue of immediacy, rapid turn-around, and team involvement; however there are several disadvantages. First, this type of analysis is problem-focused, rather than goal-focused. Long lists of problems are generated, but there is no clear relation to specific usability goals. Second, developers may not be able to fix things immediately so the context of the problem may be lost when it is time to fix the problem. Third, the JIT analysis requires that the entire development team observe the testing sessions since problems may occur that are the responsibility of different developers.

Wilson, Chauncey E. Usability Interface (1997). Articles>Usability>Testing>Reports

Analyzing the Interaction Between Facilitator and Participants in Two Variants of the Think-Aloud Method    

This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods.

van den Haak, Maaike J. and Menno D.T. de Jong. IEEE PCS (2005). Articles>Usability>Testing>Methods

Authentic Behavior in User Testing

Despite being an artificial situation, user testing generates realistic findings because people engage strongly with the tasks and suspend their disbelief.

Nielsen, Jakob. Alertbox (2005). Articles>Usability>Testing

Better Reports: How to Communicate the Results of Usability Testing  

You've spent several days setting up a usability test, recruiting the participants and running it. Then you've pored over the data. What next? If you are doing usability testing as part of user-centred design within a business setting, then there are many ways that you can communicate the results. This paper looks at reports and then considers presentation and observation as alternatives to reports.

Jarrett, Caroline. STC Proceedings (2004). Articles>Usability>Testing>Reports

Beyond the Universal User: How to Design for the Universe of Users  

Current 'user-centered' design methods place great value on design for the user. In this, I examine how investigation methods for user-centered design like usability testing and field methods are often only used to design for the universal user and not the universe of users. I critically explore the universalizing of the user that occurs during these investigation methods. I then address the problems with designing for a universal user and finally present ways to design for the universe of users.

Bowie, Jennifer L. STC Proceedings (2003). Articles>Usability>Testing

Beyond Usability Testing

Usability testing is a powerful tool in identifying problems and issues that users may have with a website or software application. But for all its benefits, traditional testing does not necessarily give a complete picture at how effective a site or application is in terms of meeting business goals.

Farrell, Tom. Frontend Infocentre (2001). Articles>Usability>Testing>Methods

Capturing the Usability Chimera: Practical Applications of Usability Testing  

How do I effectively include usability as a part of developing online and printed documentation?

Holmes, Fredrick and Melissa Wallentine. STC Proceedings (1994). Articles>Usability>Testing>Documentation

Cleaning Up for the Housekeeper, or, Why it Makes Sense to do Both Expert Review and Usability Testing

Contrasts the unique aspects of expert reviews and usability testing. The usability goals they address are different. Know when to use which one, and when to use both.

Straub, Kathleen. Usability Professionals Association (2006). Articles>Usability>Methods>Testing

Clustering for Usability Participant Selection  

User satisfaction and usefulness are measured using usability studies that involve real customers. Given the nature of software development and delivery, having to conduct usability studies can become a costly expense in the overall budget. A major part of this expense is the participant costs. Under this condition, it is desirable to reduce the number of participants without sacrificing the quality of the experiment. If a company could use a smaller participant pool and get the same results as the entire pool; this would result in significant savings. Given a participant pool of size N, is there a subset of N that would yield the same results as the entire population? This research addresses this question using a data-mining clustering tool called Applications Quest.

Gilbert, Juan E., Andrea Williams, and Cheryl D. Seals. Journal of Usability Studies (2007). Articles>Usability>Testing>Methods

A Comparative Analysis of Heuristic and Usability Evaluation Methods  

Usability testing and heuristic evaluation are two methods for detecting usability problems, or bugs in software user interfaces. Usability testing identifies bugs that impair user per$ormance. It provides a realistic context for the product evaluation, whereas heuristic evaluation does not. Further, it provides an estimate of bug severity, while heuristic evaluation does not. To its detriment, usability testing is more expensive and time intensive than heuristic evaluation. It also tends to overlook bugs that may not affect user pelformance but may negatively impact the user’s perception of product quality. Recently, we have incorporated the bestfeatures of these two methods into one procedure which seems to work better than either method alone.

Simeral, Elizabeth J. and Russell J. Branaghan. STC Proceedings (1997). Articles>Usability>Testing

Comparing Assessment Techniques  

In just the last few years, we have begun to see research studies comparing usability testing to other techniques for assessing usability. In general, usability testing has found more of the most serious usability problems and fewer of the least serious problems than other methods. Heuristic evaluation--having people evaluate the interface either from their own expertise or from a set of guidelines--has achieved mixed results. Although usability testing seems expensive compared to other methods, it may be less expensive when considered on the basis of 'cost per problem that needs to be fixed.'

Redish, Janice C. 'Ginny'. STC Proceedings (1993). Articles>Usability>Testing>Assessment

A Comparison of Eye Tracking Tools in Usability Testing  

Eye tracking tools have recently attracted attention from usability professionals. Eye tracking offers usability researchers a new way to identify very fine-grained behaviors that indicate usability problems. This paper is a comparison of different types of eye tracking tools and their potential usefulness in usability testing. Specifically, the paper examines the cost of the systems, system types, sampling rate, and some system limitations. The paper aims to provide a basic introduction to technical communicators who are considering adding an eye- tracking system to their toolkit.

DeSantis, Rich, Quan Zhou and Judith A. Ramey. STC Proceedings (2005). Articles>Usability>Testing>Eye Tracking

Conducting Usability Studies at User Group Meetings  

In this day and age, getting closer to your user base is imperative for creating user-centric documentation. This paper discusses how the Technical Publications group at Mentor Graphics tapped into their annual User Group meeting (MUG) to conduct usability studies. We cover: Convincing management of the ROI of participating in the User Group meeting; establishing relationships with meeting organizers; defining proper "protocol" for interacting with users and other meeting attendees; planning for and dealing with equipment setup; recruiting users to the usability lab; considering and acquiring incentives for usability lab participants.

Yaspo, Catherine and Sarah E. Leritz-Higgins. STC Proceedings (2005). Articles>Usability>Testing>User Centered Design

Conducting Usability Studies with Users Who Are Elderly or Have Disabilities    

Many disabled or elderly users have embraced the Web as a way to get easier access to information, services, contacts with others, and entertainment. But very often, Web sites are creating barriers for them, at the same time as they are reducing barriers of access. We can safely assume that Web sites are made by designers who have no intention to exclude groups of users from using the site. Our studies, however, have proved that good intentions are not enough to create Web content that is accessible and also usable for people with various kinds of physical, sensory, or cognitive disabilities. Web designers need to familiarize themselves with accessibility guidelines and apply them properly, but that goal is seldom achieved. Even when the guidelines are applied, a site's ease of use for people with disabilities needs to be confirmed in a usability test with actual users who have disabilities. This article describes how to conduct user-focused tests with test participants who are elderly or who have disabilities.

van der Geest, Thea. Technical Communication Online (2006). Articles>Usability>Testing>Elderly

Creative Low-Budget Usability Testing Methods  

Usability testing doesn't come cheap. You can however, follow test models that will help you improve the quality of your products, including websites. Usability professionals agree that some testing is better than none, and traditional formal usability testing can be adapted to fit your needs and your budget. This paper discusses how all four of these methods: low-cost usability testing, heuristic evaluations, expert reviews, and checkpoints in the development process were used to analyze subsites and applications at a federally funded public health website.

Reilly, Maggie T. and Cathy N. Hogan. STC Proceedings (2005). Articles>Usability>Testing>Methods

Designing and Testing Customer Satisfaction Surveys on WWW Sites  

The types of surveys being conducted on the web fall into two categories: surveys that determine who is using the WWW and surveys that determine customer satisfaction with the product or service. To the survey guidelines described by GVU, we add five guidelines for designing web customer satisfaction surveys based on reviews of recent surveys and browsing the web: (1) Begin with a clear mission statement. (2) Classlfy current users of the site. (3) Report the results online. (4) Limit the length of the questionnaire to no more than 25 multiple-part questions. (5) Limit big graphics.

Feinberg, Susan G. and Peter Y. Johnson. STC Proceedings (1997). Articles>Usability>Testing>Web Design

Discount Usability: Time To Push Back the Clock?

Discount usability techniques are a great way to eradicate usability problems. But they can never answer the question, 'How usable is this system?' We blow the dust off some techniques commonly used in the early days of usability testing to see if they can provide an answer.

Userfocus (2003). Articles>Usability>Methods>Testing

Discovering User-Generated Metaphors Through Usability Testing

A post-hoc analysis of data collected from a usability test on the Fluke ScopeMeter 97â (a diagnostic instrument for analyzing electrical signals) revealed that subjects miss-hit certain interface keys significantly more than other keys. Comments that subjects made during the usability test through the thinking-aloud protocol led us to the hypothesis that miss-hits were caused by subjects' internal metaphors (user-generated) that were unsupported by the design of the keys. A review of literature revealed a lack of research in the area of user-generated metaphor and its effects on user performance. With further analysis we found aspects of the ScopeMeter keys and subject characteristics that provide a strong foundation for this hypothesis. We conclude that formal experiments should be set up to test this and other hypotheses that address the role user-generated metaphors play in building mental models and influencing user behavior.

Schulz, Erin Leanne, Judith A. Ramey, Maarten van Alphen and William Rasnake. City University of Hong Kong (1995). Articles>Usability>Testing>Tropes

Do Usability Expert Evaluation and Testing Provide Novel and Useful Data for Game Development?    

A case study was done to study whether usability expert evaluation and testing are suitable for game development. In the study, a computer game under development was first evaluated and then tested. Game developers were then asked to rate the findings and give other feedback about the methods used and the results gained. It was found that the usability expert evaluation and testing provided both novel and useful data for game development. Based on these and the other results it is argued that the usability expert evaluation and testing have considerable face validity in game development. In addition to the usefulness and face validity of the methods it was studied whether the usability experts participating in the game usability expert evaluation should be double experts. It was found that there was no significant difference in the number or the rated relevancy of the problem the gamer and non-gamer usability specialists found.

Laitinen, Sauli. Journal of Usability Studies (2006). Articles>Usability>Testing>Methods

Don't Test Users, Test Hypotheses

User testing typically consists of a sort of fishing trip. We lower a lure (the user) into the water (the application or site) and see what critters (defects) bite. This is a valuable and time-tested approach. But when we start fishing for defects, we are left with some tough questions. For instance: When are we finished? How many defects do we need to find before we have fully tested the site or application? If we find a defect, how do we know how severe it is, and by what measure? In iterative testing, how do we compare results from the test of the current version with results from testing earlier versions?

Soudack, Avi. Boxes and Arrows (2003). Articles>Usability>Methods>Testing

Examining the Relationship Between Quality Writing and Quality Reading    

This article introduces the ISTE approach as a way to measure the relationship between the design of a document and the way that users handle it. ISTE is an acronym for Information SubTypes and Effects. The article describes the classification of information into a general typology as well as into a specific one, to illustrate how the approach can be adapted to specific usability questions. ISTE also requires a typology for categorizing possible user reactions. This article first positions ISTE within the broader context of usability testing and then describes its basic approach and some recent applications. Unlike most formal research methods, ISTE does not require an inordinate amount of time and effort for analyzing high-fidelity data. Electronic versions such as the ERR instrument allow for the creation of a complete and detailed profile of the use of a document 10 minutes after task completion. These characteristics make ISTE a viable instrument for gathering usability metrics in industrial settings.

van der Meij, Hans. Technical Communication Online (2000). Articles>Usability>Testing>Assessment

Experience Equity and Universal Access: Designing Clinical Studies for Low Vision  

In this paper, I describe web page design for those interested in conducting clinical, low vision studies. Ideally, web pages should be accessible and usable for all readers; however, the web is a highly visual medium for communication and creates serious accessibility issues for specialized (diverse needs) those with vision needs. Therefore, I propose that researchers consider a paramount and concurrent user-centered design approach when creating stimulus materials for these specialized audiences. This paper introduces readers to this design approach for a low vision audience as described in the WebText Study.

Reece, Gloria A. STC Proceedings (2005). Articles>Usability>Testing>Visual