http://tc.eserver.org/dir/Articles/Usability/Methods/Testing A listing of the most recently indexed works about Articles and Usability and Methods and Testing in the field of technical communication. en-us Copyright (c) 2005-08 by the EServer. All rights reserved. tclib-editorial@eserver.org (TC Library Editorial Board) webmaster@eserver.org (Geoffrey Sauer) http://tc.eserver.org/images/newlogo.gif http://tc.eserver.org/dir/Articles/Usability/Methods/Testing http://tc.eserver.org/35599.html http://tc.eserver.org/35599.html User research focused on single experiences with a feature or workflow uncovers different problems and issues than longitudinal research. http://tc.eserver.org/35317.html http://tc.eserver.org/35317.html Usability tests are a core design tool and, when done well, they deliver tremendous insights to the team. However, when a usability test is done poorly, it can be a disaster for everyone involved. An important key to their success is the work of a great moderator. http://tc.eserver.org/35308.html http://tc.eserver.org/35308.html Simple user testing with 5 participants, paper prototyping, and heuristic evaluation offer a cheap, fast, and early focus on usability, as well as many rounds of iterative design. http://tc.eserver.org/35266.html http://tc.eserver.org/35266.html Preparing for usability testing requires a surprisingly large amount of planning. Here are the 6 key steps you should go through to get ready. http://tc.eserver.org/34938.html http://tc.eserver.org/34938.html Every day in offices around the world usability professionals ask and are asked this question: How many users do we need for our usability test? Its an important question. We want to find most of and the most severe problems. So, we need to test enough people. But usability testing is so expensive, and the cost of testing increases with each participant. So, we don't want to test too many, either. http://tc.eserver.org/34943.html http://tc.eserver.org/34943.html We present a study comparing usability testing of a single interface versus three functionally equivalent but stylistically distinct designs. We found that when presented with a single design, users give significantly higher ratings and were more reluctant to criticize than when presented with the same design in a group of three. Our results imply that by presenting users with alternative design solutions, subjective ratings are less prone to inflation and give rise to more and stronger criticisms when appropriate. Contrary to our expectations, our results also suggest that usability testing by itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions. http://tc.eserver.org/34875.html http://tc.eserver.org/34875.html The trade show booth on the exhibit floor of a conference is traditionally used for company representatives to sell their products and services. However, the trade booth environment also creates an opportunity, for it can give the development team easy access to many varied participants for usability testing. The question is can we adapt usability testing methods to work in such an environment? Extremely rapid usability testing (ERUT) does just this, where we deploy a combination of questionnaires, interviews, storyboarding, co-discovery, and usability testing in a trade show booth environment. We illustrate ERUT in actual use during a busy photographic trade show. It proved effective for actively gathering real-world user feedback in a rapid paced environment where time is of the essence. http://tc.eserver.org/34460.html http://tc.eserver.org/34460.html The benefits of user testing have long been established. It is still important however to try and maximise these benefits. One way in which this can be done is by viewing the user test yourself. http://tc.eserver.org/33666.html http://tc.eserver.org/33666.html The beauty of the whiteboard method is that your report becomes simply a summary of what you have already written on the whiteboard, including completion metrics, findings, and recommendations that have been vetted by key stakeholders. http://tc.eserver.org/32357.html http://tc.eserver.org/32357.html This article is a case study of a demonstration project intended to prove the value of usability testing to a large textbook publishing house. In working with a new client, however, the research team discovered that what our client thought were simple problems for their users were actually complex problems that required the users to evaluate potential solutions in a surprisingly complex context of use. As Redish (2007) predicted, traditional ease of use measures were "not sufficient" indicators and failed to reveal the complex nature of the tasks. Users reported high levels of satisfaction with products being tested and believed they had successfully completed tasks which they judged as easy to complete when, in fact, they unknowingly suffered failure rates as high as 100%. The study recommends that usability specialists expand our definition of traditional usability measures so that measures include external assessment by content experts of the completeness and correctness of users' performance. The study also found that it is strategically indispensable for new clients to comprehend the upper end of complexity in their products because doing so creates a new space for product innovation. In this case, improving our clients' understanding of complexity enabled them to perceive and to take advantage of a new market niche that had been unrealized for decades. http://tc.eserver.org/31652.html http://tc.eserver.org/31652.html This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods. http://tc.eserver.org/30436.html http://tc.eserver.org/30436.html User satisfaction and usefulness are measured using usability studies that involve real customers. Given the nature of software development and delivery, having to conduct usability studies can become a costly expense in the overall budget. A major part of this expense is the participant costs. Under this condition, it is desirable to reduce the number of participants without sacrificing the quality of the experiment. If a company could use a smaller participant pool and get the same results as the entire pool; this would result in significant savings. Given a participant pool of size N, is there a subset of N that would yield the same results as the entire population? This research addresses this question using a data-mining clustering tool called Applications Quest. http://tc.eserver.org/30198.html http://tc.eserver.org/30198.html Testing 5-10 users at once lets you conduct large-scale usability testing and still meet your deadlines. http://tc.eserver.org/29636.html http://tc.eserver.org/29636.html Usability testing doesn't come cheap. You can however, follow test models that will help you improve the quality of your products, including websites. Usability professionals agree that some testing is better than none, and traditional formal usability testing can be adapted to fit your needs and your budget. This paper discusses how all four of these methods: low-cost usability testing, heuristic evaluations, expert reviews, and checkpoints in the development process were used to analyze subsites and applications at a federally funded public health website. http://tc.eserver.org/28935.html http://tc.eserver.org/28935.html The topic of test subject compensation generates a lot of conversation...how do you motivate test participants? http://tc.eserver.org/28915.html http://tc.eserver.org/28915.html One of the principles underlying usability testing is that observing a user perform a task provides more reliable information than simply asking the user how easy it would be to perform the task. By observing users, you can assess whether they are actually able to use a product. By asking them, you simply cannot. http://tc.eserver.org/28909.html http://tc.eserver.org/28909.html Everyone knows by now that they should test the usability of their applications, but still hardly anybody actually does it. In this article I'll share some tips I've picked up for doing usability tests quickly and effectively. Relatively recent tools like Django and Ruby on Rails allow us to develop projects faster and to make significant changes later in the project timeline. Usability testing methods should now be adapted to fit this modern approach to development. http://tc.eserver.org/28700.html http://tc.eserver.org/28700.html The sooner you complete a usability study, the higher its impact on the design process. Slower methods should be deferred to an annual usability checkup. http://tc.eserver.org/28617.html http://tc.eserver.org/28617.html Usability Leistungsspektrum Die ausgefeiltesten digitalen Strategien scheitern oft am Einfachsten: der Usability. Doch in einer Zeit, in der Ihr Wettbewerber nur einen Mausklick weit entfernt ist, stellt Usability eine der größten Herausforderungen im Bereich der digitalen Kommunikation dar. http://tc.eserver.org/28378.html http://tc.eserver.org/28378.html 観察している人たちを最優先に考えた調査をすべきではない。たとえ観察していてつまらないタスクばかりだとしても、デザインを真に検証するテストを実施すべきだ。 http://tc.eserver.org/28102.html http://tc.eserver.org/28102.html Contrasts the unique aspects of expert reviews and usability testing. The usability goals they address are different. Know when to use which one, and when to use both. http://tc.eserver.org/28020.html http://tc.eserver.org/28020.html A case study was done to study whether usability expert evaluation and testing are suitable for game development. In the study, a computer game under development was first evaluated and then tested. Game developers were then asked to rate the findings and give other feedback about the methods used and the results gained. It was found that the usability expert evaluation and testing provided both novel and useful data for game development. Based on these and the other results it is argued that the usability expert evaluation and testing have considerable face validity in game development. In addition to the usefulness and face validity of the methods it was studied whether the usability experts participating in the game usability expert evaluation should be double experts. It was found that there was no significant difference in the number or the rated relevancy of the problem the gamer and non-gamer usability specialists found. http://tc.eserver.org/28023.html http://tc.eserver.org/28023.html This paper argues that in the field of usability, debates about number of users, the use of statistics, etc. in the abstract are pointless and even counter-productive. We propose that the answers depend on the research questions and business objectives of each project and thus cannot be discussed in absolute terms. Sometimes usability testing is done with an implicit or explicit hypothesis in mind. At other times the purpose of testing is to guide iterative design. These two approaches call for different study designs and treatment of data. We apply control systems theory to the topic of usability to highlight and frame the value of iterative usability testing in the design lifecycle. Within this new metaphor, iterative testing is a form of feedback which is most effective and resource-efficient if done as often as practically possible with project resources and timelines in mind. http://tc.eserver.org/28017.html http://tc.eserver.org/28017.html Small sample sizes are a fact of life for most usability practitioners. This can lead to serious measurement problems, especially when making binary measurements such as successful task completion rates (p). The computation of confidence intervals helps by establishing the likely boundaries of measurement, but there is still a question of how to compute the best point estimate, especially for extreme outcomes. In this paper, we report the results of investigations of the accuracy of different estimation methods for two hypothetical distributions and one empirical distribution of p. If a practitioner has no expectation about the value of p, then the Laplace method ((x+1)/(n+2)) is the best estimator. If practitioners are reasonably sure that p will range between .5 and 1.0, then they should use the Wilson method if the observed value of p is less than .5, Laplace when p is greater than .9, and maximum likelihood (x/n) otherwise. http://tc.eserver.org/27941.html http://tc.eserver.org/27941.html 6% of task attempts are extremely slow and constitute outliers in measured user performance. These sad incidents are caused by bad luck that designers can -- and should -- eradicate. http://tc.eserver.org/27897.html http://tc.eserver.org/27897.html We can define usability in terms of quality metrics, such as learning time, efficiency of use, memorability, user errors, and subjective satisfaction. Sadly, few projects collect such metrics because doing so is expensive: it requires four times as many users as simple user testing. Many users are required because of the substantial individual differences in user performance. When you measure people, you'll always get some who are really fast and some who are really slow. Given this, you need to average these measures across a fairly large number of observations to smooth over the variability. http://tc.eserver.org/27405.html http://tc.eserver.org/27405.html Doesn't one need to test with at least 100 or more users for statistical significance, accuracy and validity? http://tc.eserver.org/27413.html http://tc.eserver.org/27413.html Some people think that usability is very costly and complex and that user tests should be reserved for the rare web design project with a huge budget and a lavish time schedule. Not true. Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford. http://tc.eserver.org/27147.html http://tc.eserver.org/27147.html To meet your users’ needs, it is essential to know your audience and to design for them. A key way to do this is by identifying your Web site’s primary users and recruiting a sample for usability testing. Consider these four aspects. http://tc.eserver.org/21366.html http://tc.eserver.org/21366.html Recording what users do is a crucial aspect of usability testing. Fortunately, recording screen activity doesn’t necessarily cost much. Three Windows-based software programs range between $30 and $150 and offer excellent performance. http://tc.eserver.org/21153.html http://tc.eserver.org/21153.html Self-described as the absolute [sic] easiest introduction to usability testing you could possibly find anywhere. http://tc.eserver.org/21125.html http://tc.eserver.org/21125.html First, when you are conducting a usability test, it is important to understand exactly what data you should be collecting. You should not run a test without first deciding on what data is required to address your business challenges. Plan ahead! Second, in a usability test, you don't just watch users. You must collect data that reflects how customers actually use your products and services. This is easier said than done. http://tc.eserver.org/21030.html http://tc.eserver.org/21030.html Jarrett dispels several myths about usability testing that may dissuade technical communicators from applying valuable usability techniques. http://tc.eserver.org/20894.html http://tc.eserver.org/20894.html User testing typically consists of a sort of fishing trip. We lower a lure (the user) into the water (the application or site) and see what critters (defects) bite. This is a valuable and time-tested approach. But when we start fishing for defects, we are left with some tough questions. For instance: When are we finished? How many defects do we need to find before we have fully tested the site or application? If we find a defect, how do we know how severe it is, and by what measure? In iterative testing, how do we compare results from the test of the current version with results from testing earlier versions? http://tc.eserver.org/20855.html http://tc.eserver.org/20855.html These are the instructions we gave to the people at various Sun branch offices in Europe and Asia for their user testing of a new design for the company's web pages. In a few places, these instructions refer to web-specific issues, so they will have to be modified slightly for use in other projects. These instructions were sent by electronic mail to those local Sun reps who had volunteered to lead a test. http://tc.eserver.org/19750.html http://tc.eserver.org/19750.html If you’ve done any usability testing, design evaluations, or heuristic inspections, then you’ve been affected by Rolf Molich's pioneering work. http://tc.eserver.org/19626.html http://tc.eserver.org/19626.html This document is intended to help beginners design questions to help them conduct a good usability testing session. If you already have a prototype you want to test, you've already drafted a few questions, and you're eager to learn how to make the most of your opportunity to learn from your users, then this document is for you. http://tc.eserver.org/19606.html http://tc.eserver.org/19606.html Discount usability techniques are a great way to eradicate usability problems. But they can never answer the question, 'How usable is this system?' We blow the dust off some techniques commonly used in the early days of usability testing to see if they can provide an answer. http://tc.eserver.org/19604.html http://tc.eserver.org/19604.html People often throw around the terms 'objective' and 'subjective' when talking about the results of a usability test. These terms are frequently equated with the statistical terms 'quantitative' and 'qualitative'. The analogy is false, and this misunderstanding can have consequences for the interpretations and conclusions of usability tests. http://tc.eserver.org/19524.html http://tc.eserver.org/19524.html Usability is literally the 'ease of use' or understanding it takes to make something work. In this case, Web Site usability is the understanding of how an individual user navigates, finds information and interacts with your Web Site. Unlike online surveys or focus groups, usability testing is a oneon- one process in a 'watch and learn' approach. The results of the sessions are used to improve your user’s experience. Having the development team watch the testing and witness the results helps resolve most internal issues in an undisputed manner. You can’t fight the reality of usability testing. http://tc.eserver.org/19290.html http://tc.eserver.org/19290.html Usability testing is a powerful tool in identifying problems and issues that users may have with a website or software application. But for all its benefits, traditional testing does not necessarily give a complete picture at how effective a site or application is in terms of meeting business goals. http://tc.eserver.org/19040.html http://tc.eserver.org/19040.html Once an organisation decides to go ahead with a user testing programme, the questions really begin. Is it really necessary to undertake testing in a 'usability lab'? And what exactly should a fully functioning lab consist of anyway? As one might imagine, opinion is divided on these issues. We take a quick look at what a typical lab might consist of and the pros and cons of lab-based testing. http://tc.eserver.org/18808.html http://tc.eserver.org/18808.html User-centered product design is a design approach that focuses on the users’ job tasks, skills, and abilities. Usability testing has emerged as a critical component in the user-centered design process to assure that a product meets the needs of the user. Implemented correctly, usability testing can increase customer satisfaction and acceptance, improve product image, and reduce development costs. A variety of information is available to help you get started in developing a usability testing process. This workshop will provide sample questionnaires, checklists, scenarios, scripts, etc. However, the main focus of the workshop will be to train participants in the following two areas: (1) writing measurable usability test goals; and (2) collecting and interpreting the test data. These activities are critical because they relate directly to the usefulness of the test results. http://tc.eserver.org/18720.html http://tc.eserver.org/18720.html Not so very long ago, it was agreed that five to eight users was enough for a good usability test. Somehow, this idea achieved mythic status. We believed it. We preached it to everyone who would listen. It survived in areas where it had been disproved, and was introduced into new situations where it didn't even apply. What gives some ideas such staying power? What did the five-user myth accomplish? It reconciled test plans with testing budgets! If five to eight users are enough, then it's safe to act on the results of a test series with only five to eight users. http://tc.eserver.org/18622.html http://tc.eserver.org/18622.html Journaled sessions bridges usability inquiry, where you ask people about their experiences with a product, and usability testing, where you observe people experiencing the product's user interface. Journaled sessions are often used as a remote inquiry method for software user interface evaluation. A disk is distributed to a number of test subjects containing a prototype of the software product, as well as additional code to capture (or journalize) the subjects' actions when using the prototype. Users perform several tasks with the prototype, much as in formal usability tests, and their actions are captured with the journalizing software. Upon completion of the series of tasks, the users return the disks to you for you to evaluate. Because the journaling portion of the evaluation is largely automated, this approach to remote, hands-off inquiry is certainly more 'usable' then self-reporting logging, where users are requested to write down their observations and comments and send them back to you. http://tc.eserver.org/15003.html http://tc.eserver.org/15003.html Discusses how to choose different usability methods for iterative research. Slides only. http://tc.eserver.org/14996.html http://tc.eserver.org/14996.html Describes the strength and weaknesses of two usability assessment methods frequently applied to web sites to illustrate issues of special interest to designers. http://tc.eserver.org/11783.html http://tc.eserver.org/11783.html There is an ongoing discussion in usability circles about the importance of formal reports that document the results of usability testing. I think that each usability evaluation should have a formal report that provides some context for the problems. Not all problems can be addressed immediately and memories fade. Usability reports are also important for showing what a usability specialist has done. They can also be used to determine some metrics, such as the number of problems addressed by development or the number of problems that occurred during successive prototypes or versions of a product. http://tc.eserver.org/10578.html http://tc.eserver.org/10578.html Developers may hesitate to start usability testing because they worry that their product poses special problems in finding, scheduling, or compensating the right users. This shouldn’t stop them. We successfully find and test hundreds of users a year and about 10% of these require special tactics for scheduling. http://tc.eserver.org/10383.html http://tc.eserver.org/10383.html Usability testing is an evaluation method used by technical communicators that can combine aspects of both quantitative and qualitative research methodologies. This article compares and contrasts the standards and techniques between these two methods of inquiry with particular emphasis on maintaining rigorous tests in regard to validity and reliability of the findings. Whether an evaluator relies on quantitative methods, qualitative methods, or both, should depend on the questions the research or evaluation seeks to answer. Both methods have well-established practices meant to ensure the validity and reliability of their findings. Not only should usability evaluators be competent within the method of inquiry they apply, they also need to help clients understand the legitimate application and limitations of their findings. http://tc.eserver.org/10384.html http://tc.eserver.org/10384.html Know your audiences, comes the repeated message for technical communicators and in response, more and more technical communicators have turned to usability testing to learn more about their readers and to improve their communications. Technical communicators produce manuals, instructions, and warnings for hand tools, medical equipment, lawn mowers, tractors, pesticide sprayers, and thousands of different products. Most manuals, instructions, and products can benefit from usability testing. This case study provides guidance for technical communicators who are novices to usability testing. The lessons we learned can be of value to technical communicators beginning their first usability testing on a wide range of technical communications and products. http://tc.eserver.org/10318.html http://tc.eserver.org/10318.html How you ask a question strongly determines the type of answer that you will obtain. For effective documentation reviews, whether they are conducted internally or externally as part of usability testing, it's important to use precise questions that will provide concrete information on which to base revisions. This paper proposes an approach to obtaining useful feedback that emphasizes negative, 'what did we do wrong?' questions. This approach focuses limited resources on areas that need improvement rather than areas that already work well and that don't require immediate improvement.