Measuring Online Experience: It's About More Than Time!

Users' specific online activities, their knowledge of computers and the Internet, and how they feel about being online could differ significantly. In order to obtain a more robust measure, researchers may want to consider how and what users think, feel, and do online when they assess level of experience. In this article we explore each of these areas and present a new measure of measuring online experience.

Rogers, Bonnie Lida. Usability News (2003). Articles>Usability>Testing>Online

A Methodology for Testing Voting Systems  

This paper compares the relative merit in realistic versus lab style experiments for testing voting technology. By analyzing three voting experiments, we describe the value of realistic settings in showing the enormous challenges for voting process control and consistent voting experiences. The methodology developed for this type of experiment will help other researchers to test polling place protocols and administration. Comparing the results from laboratory experiments with voter verification and realistic voting experiments further validates the procedure of testing equipment in laboratory settings. The methodology and protocol for testing voting systems can be applied to any voting technology. This protocol matches the real-world conditions of voting by replicating them for the experiment.

Selker, Ted, Elizabeth Rozenwieg and Anna Pandolfo. Journal of Usability Studies (2006). Articles>Usability>Testing>Civic

Methods for Successful 'Thinking Out Loud' Procedure

When you are screening the participants for a study, notice how they respond to your questions. Decide on a strategy for engaging the participant before they arrive for the usability study. Be careful of the social dynamics you set up with the participant.

Ramey, Judith A. STC Usability SIG (1998). Resources>Usability>Testing>Methods

Multiple-User Simultaneous Testing (MUST)

Testing 5-10 users at once lets you conduct large-scale usability testing and still meet your deadlines.

Nielsen, Jakob. Alertbox (2007). Articles>Usability>Testing>Methods

The Myth of the Genius Designer

Having a good designer doesn't eliminate the need for a systematic usability process. Risk reduction and quality improvement both require user testing and other usability methods.

Nielsen, Jakob. Alertbox (2007). Articles>Web Design>Usability>Testing

The Need for Usability Analysis  

An overview of methods for usability testing and analysis.

Xerox (1995). Presentations>Usability>Testing>Methods

Ninety Percent of All Usability Testing is Useless

Ninety percent of all usability testing performed on Web sites is useless. This is not to say that it doesn’t have a significant role to play in user experience design. When done right, usability testing will improve your Web site and your development process, but the current culture surrounding Web site usability testing is such that it rarely benefits the design. Worse, this misapplication can undermine the acceptance of this important technique throughout an organization.

Becker, Lane. Adaptive Path (2004). Articles>Usability>Testing

Outliers and Luck in User Performance

6% of task attempts are extremely slow and constitute outliers in measured user performance. These sad incidents are caused by bad luck that designers can -- and should -- eradicate.

Nielsen, Jakob. Alertbox (2006). Articles>Usability>Testing>Methods

Performance Testing

Performance testing is a rigorous usability evaluation of a working system under realistic conditions to identify usability problems and to compare measures such as success rate, task time and user satisfaction with requirements.

UsabilityNet. Resources>Usability>Testing

Planning Usability Tests For Maximum Impact  

Usability tests make products better. Those of us who have seen their results understand their value, but we sometimes have difficulty convincing managers or developers of their worth. The pay-off of a usability test depends on its context. Both the type of test you perform and the timing of that test influence what you can do with the results. At best, you can guide the design of a product or clearly demonstrate its usability. At worst, you can ruin your credibility and have no effect on the product whatsoever. This paper explores what you get from a usability test at each stage of the user interface design process. It also provides advice on selling testings to management.

McDaniel, Scott M. and Laura Snyder. STC Proceedings (2004). Articles>Usability>Testing

Protocol Considerations for Using Eye-Tracking in Website Usability Testing  

Eye-tracking systems can enrich a Website usability study by providing an additional method for observing users' behavior. While eye-trackers can provide valuable data, the pros and cons of adding eye-tracking to a usability study need to be considered before designing the study's protocol. This paper discusses the kinds of usability questions that benefit from eye-tracking data and considerations for designing and running the study. Our findings are based on work done in the Laboratory for Usability Testing and Evaluation (LUTE) at the University of Washington, which is equipped with the Eye-gaze Response Computer Aid (ERICA) system controlled by the Gaze Tracker software, both of which were developed by Eye Response Technologies, Inc.

Cuddihy, Elisabeth, Zhiwei Guan and Judith A. Ramey. STC Proceedings (2005). Articles>Usability>Testing>Eye Tracking

Quantitative Studies: How Many Users to Test?

We can define usability in terms of quality metrics, such as learning time, efficiency of use, memorability, user errors, and subjective satisfaction. Sadly, few projects collect such metrics because doing so is expensive: it requires four times as many users as simple user testing. Many users are required because of the substantial individual differences in user performance. When you measure people, you'll always get some who are really fast and some who are really slow. Given this, you need to average these measures across a fairly large number of observations to smooth over the variability.

Nielsen, Jakob. Alertbox (2006). Articles>Usability>Testing>Methods

Reading to Decide: Designing for Usability with a Needs, Users, and Learnings (NUL) Analysis  

Information products that help users make decisions are a challenge to design, develop, and test for usability. We can begin to formulate a strategy for producing such documents effectively by doing a comprehensive preassessment of the specific project and by adapting testing methodologies from studies of creative thinking processes and expert system design.

Knodel, Elinor L. STC Proceedings (1995). Design>Usability>Methods>Testing

Recording Screen Activity During Usability Testing

Recording what users do is a crucial aspect of usability testing. Fortunately, recording screen activity doesn’t necessarily cost much. Three Windows-based software programs range between $30 and $150 and offer excellent performance.

Fast, Karl. Boxes and Arrows (2002). Articles>Usability>Methods>Testing

Recruiting User Testing Participants

To meet your users’ needs, it is essential to know your audience and to design for them. A key way to do this is by identifying your Web site’s primary users and recruiting a sample for usability testing. Consider these four aspects.

Martin, Beth A. Usability.gov (2006). Articles>Usability>Testing>Methods

Remote Usability: Insight into New Tools

A review of software packages for moderated usability testing in a native environment via a telephone/web interface.

Madaio, Mike. Usability Professionals Association (2006). Articles>Usability>Testing>Online

The Return on Usability Testing

Usability testing is the qualitative measurement of user experience with a product or service, such as a website. Usability testing is for anyone who wants to make their website, software, device, etc. easier to use. Whether you employ thousands of people such as Microsoft or fall into the small business category, usability testing can play a significant role in giving you direct feedback from your customers.

Subramanian, Subha. GotoMedia (2005). Articles>Usability>Testing

Review: A Review of Morae 2.0 for Usability Testing

TechSmith's recent release, Morae 2.0, features a new graphing tool, integrated satisfaction survey, and embedded task definitions. The editable marker log in Observer and the improved timeline controls in the Manager improve operator efficiency. This article highlights these and other new features of the new 2.0.

Shrestha, Sav. Usability News (2007). Articles>Reviews>Usability>Testing

A Review of Morae for Usability Testing

TechSmith's Morae is a powerful tool that is useful for any usability study. Its simplicity, efficiency, cost effectiveness and portability has redefined the data collection process and allows researchers to focus on the user experience rather than the data collection tools. This article discusses some of the features of this tool and proposes some enhancements to make it even more powerful to usability practitioners.

Shrestha, Sav. Usability News (2007). Articles>Usability>Testing>Software

Rigor in Usability Testing    

Usability testing is an evaluation method used by technical communicators that can combine aspects of both quantitative and qualitative research methodologies. This article compares and contrasts the standards and techniques between these two methods of inquiry with particular emphasis on maintaining rigorous tests in regard to validity and reliability of the findings. Whether an evaluator relies on quantitative methods, qualitative methods, or both, should depend on the questions the research or evaluation seeks to answer. Both methods have well-established practices meant to ensure the validity and reliability of their findings. Not only should usability evaluators be competent within the method of inquiry they apply, they also need to help clients understand the legitimate application and limitations of their findings.

Hughes, Michael A. Technical Communication Online (1999). Articles>Usability>Testing>Methods

Role of the Highlights Video in Usability Testing: Rhetorical and Generic Expectations    

Despite the proliferation of books and articles dealing with usability testing as a sub-field of technical communication, there remains one artifact that is underdefined: the usability highlights video. Several sources suggest that usability testers should deliver a video to their clients, but there is no common description of the contents of this video or the rhetorical purpose it fulfills. In this article, we examine the use of the highlights video as described in the literature, but we go further to discover some ways practicing usability professionals understand the role of the highlights video. Through the use of generic conventions, rhetorical theory, and industrial practices, we attempt to draw conclusions that point to some common uses of the highlights video that can instruct both teachers in the usability classroom and practicing usability experts as they create videos for client projects.

Yeats, Dave and Locke Carter. Technical Communication Online (2005). Articles>Usability>Testing>Video

RSVP in Review: A Comparison of Programs for the PC and Handheld Devices

RSVP is a method of displaying text one word (or sentence) at a time in the same area of a screen. RSVP has been used as a method of teaching reading, speed reading, as well as assessing reading speed and comprehension in students and adults. At SURL, we have been interested in the use of RSVP with small-screen interfaces, such as those used on handheld devices.

Russell, Mark C. and Shannon Riley. Usability News (2003). Articles>Usability>Testing

Scheduling Hard-to-Find Users

Developers may hesitate to start usability testing because they worry that their product poses special problems in finding, scheduling, or compensating the right users. This shouldn’t stop them. We successfully find and test hundreds of users a year and about 10% of these require special tactics for scheduling.

User Interface Engineering (1998). Articles>Usability>Methods>Testing

Shaping Knowledge through Usability Testing Shaping Knowledge through Usability Testing  

Usability testing can make a difference in the product and and the documentation. Seeing is believing.

Barnum, Carol M. IEEE PCS (2003). Presentations>Usability>Methods>Testing