http://tc.eserver.org/dir/Testing A listing of the most recently indexed works about Testing in the field of technical communication. en-us Copyright (c) 2005-08 by the EServer. All rights reserved. tclib-editorial@eserver.org (TC Library Editorial Board) webmaster@eserver.org (Geoffrey Sauer) http://tc.eserver.org/images/newlogo.gif http://tc.eserver.org/dir/Testing http://tc.eserver.org/35646.html http://tc.eserver.org/35646.html In this Ask UXmatters column—which is the second in a three-part series of columns focusing on usability—our experts discuss how to conduct usability testing with limited funding. http://tc.eserver.org/35651.html http://tc.eserver.org/35651.html In this Ask UXmatters column—which is the first in a series of three columns focusing on usability—our experts discuss the use of usability testing versus expert reviews. http://tc.eserver.org/35653.html http://tc.eserver.org/35653.html It is easy to get excited about eyetracking. Seeing where people look while using your Web site, Web application, or software product sounds like an opportunity to get amazing insights into their user experience. But eyetracking is expensive and requires extra effort and specialized knowledge. The heat maps and other visualizations certainly look impressive, but what can you really learn from them? After using eyetracking for the first time, many find that it is not easy to know how to analyze the visualizations and make conclusions from them. Does eyetracking really provide any additional insights you would not have discovered anyway through traditional usability testing? Does the value of eyetracking outweigh its limitations? This article will discuss and answer these questions. http://tc.eserver.org/35599.html http://tc.eserver.org/35599.html User research focused on single experiences with a feature or workflow uncovers different problems and issues than longitudinal research. http://tc.eserver.org/35602.html http://tc.eserver.org/35602.html A few years ago, some of my colleagues decided to run a first-experience study on one of our software packages. The purpose of such a study is to gain an understanding of what our users go through in their first hour of use. What do they experience? Where do they get stuck? How far can they get in the software? What are their learning strategies? As a side experiment, my colleagues asked several experts in the company for their expert opinion as to what problems users would run into, and compiled them into a list. (These experts included the software designers, domain experts, and the people who trained users on the software.) Then my colleagues ran their study, observing sixteen people using this software for the first time, and made a list of the problems that users actually ran into. The result? There was not one common item on the two lists. http://tc.eserver.org/35575.html http://tc.eserver.org/35575.html Basically, terrible testing yields terrible results, and since the goal of the comparative usability tests was to find best practices, some teams in those tests did not follow best practices and thus did not get good results. In other words, the fact that they did not get good results is not an inherent problem with usability tests; it’s a direct result of them doing a poor job. http://tc.eserver.org/35591.html http://tc.eserver.org/35591.html As a user researcher with a primarily qualitative background, I have to confess that when I was asked to conduct a usability benchmark study on AutoCAD, I was not exactly jumping out of my chair. Frankly, I was wary of the quantitative emphasis of the method and the proposal to reduce the whole user experience down to a single number. I was also more than slightly nervous about designing a benchmark study for a product as complex as AutoCAD. http://tc.eserver.org/35593.html http://tc.eserver.org/35593.html How can we designers get valid feedback from more design iterations in less time? One bottleneck in the design flow is finding a steady stream of usability testers. Between the extremes of the perfect (an actual user, on site) and the unacceptable (the developer who's coding the feature), lies the grey zone of user proxies. Can you use internal employees with relevant domain knowledge to usability test your products, and still get valid data? http://tc.eserver.org/35473.html http://tc.eserver.org/35473.html Here is a collection of some testing tools that we have compiled to aid your testing handily grouped into categories. Look out for our reviews of some of these tools coming soon. http://tc.eserver.org/35446.html http://tc.eserver.org/35446.html Eye tracking has been investigated and 'toyed with' for many years by researchers and commercial usability professionals. Many new techniques and therefore interesting and powerful results are now available. http://tc.eserver.org/35447.html http://tc.eserver.org/35447.html Incorporation of eye position recording into product usability evaluation can provide insights into human-computer interaction that are not available from traditional usability testing methods. We present here some thoughts on this topic which arose primarily from a CHI 99 workshop. This workshop brought together human-computer interaction designers, eye movement researchers and usability testing specialists for a discussion about how to extract information about product usability from users’ eye movements. http://tc.eserver.org/35404.html http://tc.eserver.org/35404.html Usability evaluations are good for many things, but determining a team's priorities is not one of them. The Molich experiment proves a single usability team can't discover all or even most major problems on a site. But usability testing does have value as a shock treatment, trust builder, and part of a triangulation process. Test for the right reasons and achieve a positive outcome. http://tc.eserver.org/35352.html http://tc.eserver.org/35352.html There seems to be this idea going around that usability testing is bad, or that the cool kids don’t do it. That it’s old skool. That designers don’t need to do it. What if I told you that usability testing is the hottest thing in experience design research? Every time a person has a great experience with a website, a web app, a gadget, or a service, it’s because a design team made excellent decisions about both design and implementation—decisions based on data about how people use designs. And how can you get that data? Usability testing. http://tc.eserver.org/35317.html http://tc.eserver.org/35317.html Usability tests are a core design tool and, when done well, they deliver tremendous insights to the team. However, when a usability test is done poorly, it can be a disaster for everyone involved. An important key to their success is the work of a great moderator. http://tc.eserver.org/35308.html http://tc.eserver.org/35308.html Simple user testing with 5 participants, paper prototyping, and heuristic evaluation offer a cheap, fast, and early focus on usability, as well as many rounds of iterative design. http://tc.eserver.org/35266.html http://tc.eserver.org/35266.html Preparing for usability testing requires a surprisingly large amount of planning. Here are the 6 key steps you should go through to get ready. http://tc.eserver.org/35237.html http://tc.eserver.org/35237.html We believe user research is too important to give up. So we have to run tests quickly and cheaply for our clients to accept the cost - and we have to clearly show how it brings value. Because of this, we’ve developed a toolbox of quick, cheap UX research techniques. In this article, we’ll talk about one technique known as fast prototyping, and how we effectively used it in a recent project for Vodafone Ireland. http://tc.eserver.org/34938.html http://tc.eserver.org/34938.html Every day in offices around the world usability professionals ask and are asked this question: How many users do we need for our usability test? Its an important question. We want to find most of and the most severe problems. So, we need to test enough people. But usability testing is so expensive, and the cost of testing increases with each participant. So, we don't want to test too many, either. http://tc.eserver.org/34943.html http://tc.eserver.org/34943.html We present a study comparing usability testing of a single interface versus three functionally equivalent but stylistically distinct designs. We found that when presented with a single design, users give significantly higher ratings and were more reluctant to criticize than when presented with the same design in a group of three. Our results imply that by presenting users with alternative design solutions, subjective ratings are less prone to inflation and give rise to more and stronger criticisms when appropriate. Contrary to our expectations, our results also suggest that usability testing by itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions. http://tc.eserver.org/34954.html http://tc.eserver.org/34954.html In a typical on-camera usability test, in the lab environment, we ask the subjects to perform certain tasks using the product. Such tests are time-bound and follow strict procedures in terms of user profiling, briefing, task descriptions, actual performance and think aloud exercise, screen capturing (in case of software), etc. You do the video recording of usability test to observe the user performance, their response and behavior. All this is perfectly fine. Coming back to certain observations from the Big Boss reality show. I found that the participants of Big Boss have evolved through distinct psychological stages, which are sequentially progressive. http://tc.eserver.org/34875.html http://tc.eserver.org/34875.html The trade show booth on the exhibit floor of a conference is traditionally used for company representatives to sell their products and services. However, the trade booth environment also creates an opportunity, for it can give the development team easy access to many varied participants for usability testing. The question is can we adapt usability testing methods to work in such an environment? Extremely rapid usability testing (ERUT) does just this, where we deploy a combination of questionnaires, interviews, storyboarding, co-discovery, and usability testing in a trade show booth environment. We illustrate ERUT in actual use during a busy photographic trade show. It proved effective for actively gathering real-world user feedback in a rapid paced environment where time is of the essence. http://tc.eserver.org/34685.html http://tc.eserver.org/34685.html Focus groups and usability testing are two very useful but very different user research disciplines. This article will look at the difference between focus groups and usability testing, the pros and cons of each and when in the development process you should use them. http://tc.eserver.org/34569.html http://tc.eserver.org/34569.html An Introduction to Usability Testing and Tips for Effective Usability Testing in India. Created and presented by Abhay Rautela at Management Development Institute, Gurgaon, India at Bar Camp Delhi 6 http://tc.eserver.org/34536.html http://tc.eserver.org/34536.html Many digital agencies are now talking about usability and including it in their offering, but few are incorporating into their everyday process. Here are some reasons why agencies should think seriously about integrating usability and usability testing into their offering. http://tc.eserver.org/34502.html http://tc.eserver.org/34502.html In case you are not using the services of a professional recruitment agency, or do not have an internal recruitment team that can help you acquire participants for usability tests, here are some tips to help you begin with finding those participants successfully. http://tc.eserver.org/34503.html http://tc.eserver.org/34503.html An introduction to usability testing and 15 tips for effective usability testing in India. Created and presented by Abhay Rautela at the Management Development Institute, Gurgaon, India during the second day of Bar Camp Delhi 6. http://tc.eserver.org/34460.html http://tc.eserver.org/34460.html The benefits of user testing have long been established. It is still important however to try and maximise these benefits. One way in which this can be done is by viewing the user test yourself. http://tc.eserver.org/33958.html http://tc.eserver.org/33958.html In today’s financial climate, organisations are trying to cut costs. This has led to lots of new and innovative cost-cutting usability techniques springing up. Some of them are ingenious, but not every cost saving measure is a good idea. One technique that is becoming popular with some is focus group usability testing. I recommend that you avoid this technique completely. I’ll try to explain why. http://tc.eserver.org/33666.html http://tc.eserver.org/33666.html The beauty of the whiteboard method is that your report becomes simply a summary of what you have already written on the whiteboard, including completion metrics, findings, and recommendations that have been vetted by key stakeholders. http://tc.eserver.org/33311.html http://tc.eserver.org/33311.html Sometimes we need to take a step back to ensure that our motivations are in the right place. It can be easy to forget that, when people participate in our studies, they are our partners. Dana Chisnell has taken the time to examine these attitudes and help us understand how to avoid falling into such traps. http://tc.eserver.org/33295.html http://tc.eserver.org/33295.html Discusses how usability testing can be useful at all stages of games development. http://tc.eserver.org/33244.html http://tc.eserver.org/33244.html This paper explores issues of professional, ethical conduct in usability testing centering around the concept of 'informed consent'. Previous work on informed consent has been in homogeneous geographic locations. With Internet sites being developed at a prodigious rate, these procedures need to be revisited for their applicability to heterogeneous locations, in terms of culture, business practice, language and legal requirements. Some previously valued principles might now be considered discretionary, that is their applicability has situational specificity. Other principles are mandatory. http://tc.eserver.org/33245.html http://tc.eserver.org/33245.html Videotape has become one of the CHI community's mostuseful technologies: it allows us to analyze users' interactions with computers,prototype new interfaces, and present the results of our research andtechnical innovations to others. But video is a double-edged sword. It isoften misused, however unintentionally. How can we use it well, without compromising our integrity? This paper presents actual examples of questionable videotaping practices. Next, it explains why we cannot simply borrow ethical guidelines from otherprofessions. It concludes with a proposal for developing usable ethical guidelines for the capture, analysis andpresentation of video. http://tc.eserver.org/33186.html http://tc.eserver.org/33186.html User testing is a valuable tool, but how does one conduct user tests internationally? This essay draws from my experience leading an international user testing project, and I hope you can learn from my mistakes and successes. http://tc.eserver.org/33163.html http://tc.eserver.org/33163.html Field studies, user observations, contextual analyses, and all procedures which aim at determining true human needs are still just as important as ever – but they should all be done outside of the product process. This is the information needed to determine what product to build, which projects to fund. Do not insist on doing them after the project has been initiated. Then it is too late, then you are holding everyone back. http://tc.eserver.org/33007.html http://tc.eserver.org/33007.html The proliferation of usability labs is a sign of success for the field of user-centered design. Whether it’s a low-rent lab comprised of a couple adjacent conference rooms, a video camera, and a television, or a fully decked-out space with remote-control cameras, two-way mirrors, an observation room, and bowls of M&Ms — more and more companies are investing in such set-ups. Conducting user tests in labs is probably the most common means of getting user input on projects. That’s a shame, because standard user testing practice is remarkably out of sync with reality. http://tc.eserver.org/32901.html http://tc.eserver.org/32901.html Although user-centered design is a well-supported concept in the literature on adult computer products, not until recently have publications begun to appear addressing the need to include the user in the design process of children’s computer products. http://tc.eserver.org/32592.html http://tc.eserver.org/32592.html Usability testing makes use of a lot of role-playing scenarios like this one, and many findings and design recommendations result from participants’ responses to these scenarios. But an over-reliance on role playing when testing a product and making design recommendations can have major downsides and risks http://tc.eserver.org/32546.html http://tc.eserver.org/32546.html Mary wants to compare the average registration times between the two versions to see if the second version is faster than the first. The method typically used when comparing averages is called a t-test of independent means. http://tc.eserver.org/32378.html http://tc.eserver.org/32378.html Completing usability testing quickly is a challenge anywhere but especially in consultancies, which have to overcome additional challenges, such as learning a new application. To assure success on these projects, I’ve developed a quick turnaround usability testing methodology (QTUT) that minimizes the time needed to complete testing. http://tc.eserver.org/32362.html http://tc.eserver.org/32362.html There is a need to develop new usability testing environments and methodologies for unconventional interactive systems. Pursuant to that need, we developed a low-cost test environment for a Head-Mounted Display (HMD)-based, virtual reality system called Osmose. Osmose was difficult to test for many reasons, one of which was its style of interaction. We began setting up the testing environment about two weeks before the start of the usability testing. We learned many lessons throughout the experience. This paper summarizes the study findings, both methodological - how to setup and conduct a usability lab for such an environment - as well as conceptual -the human experiences and behavioral patterns involved in using an immersive environment. http://tc.eserver.org/32357.html http://tc.eserver.org/32357.html This article is a case study of a demonstration project intended to prove the value of usability testing to a large textbook publishing house. In working with a new client, however, the research team discovered that what our client thought were simple problems for their users were actually complex problems that required the users to evaluate potential solutions in a surprisingly complex context of use. As Redish (2007) predicted, traditional ease of use measures were "not sufficient" indicators and failed to reveal the complex nature of the tasks. Users reported high levels of satisfaction with products being tested and believed they had successfully completed tasks which they judged as easy to complete when, in fact, they unknowingly suffered failure rates as high as 100%. The study recommends that usability specialists expand our definition of traditional usability measures so that measures include external assessment by content experts of the completeness and correctness of users' performance. The study also found that it is strategically indispensable for new clients to comprehend the upper end of complexity in their products because doing so creates a new space for product innovation. In this case, improving our clients' understanding of complexity enabled them to perceive and to take advantage of a new market niche that had been unrealized for decades. http://tc.eserver.org/32089.html http://tc.eserver.org/32089.html As part of the product, testing documentation seems like an obvious thing to do, but what does it really mean? I’ve fielded the question in a few different places now and it’s always interesting to delve deeper and understand the rationale behind the request. http://tc.eserver.org/31903.html http://tc.eserver.org/31903.html TiVo ran 12 user tests in 12 weeks while designing its new website. As TiVo's experience shows, frequent and regular testing keeps the design usability focused. http://tc.eserver.org/31652.html http://tc.eserver.org/31652.html This paper focuses on the interaction between test participants and test facilitator in two variants of the think-aloud method. In a first, explorative study, we analyzed think-aloud transcripts from two usability tests: a concurrent think-aloud test and a constructive interaction test. The results of our analysis show that while the participants in both studies never explicitly addressed the facilitator, the think-aloud participants showed more signs of awareness of the facilitator than the participants in the constructive interaction test. This finding may have practical implications for the validity of the two methods. http://tc.eserver.org/31625.html http://tc.eserver.org/31625.html All councils and public sector organisations have a clear remit to make their information available and accessible to their citizens, and increasingly they are applying usability testing to gather empirical evidence of the effectiveness of their sites. Whether this is done by internal teams or external, independent organisations and consultants, usability testing is now part of a process applied by many site managers. Indeed usability is considered by SOCITM in their annual UK council site reviews, and is a critical part to citizen fining services and transacting online. http://tc.eserver.org/31196.html http://tc.eserver.org/31196.html This article discusses users' visual scan paths of web pages containing text and/or pictures while conducting browsing and searching tasks. User performance on three usability tasks on an e-commerce website is described. Results show that users follow a fairly uniform scan path when browsing through pictures, and a more random path while specifically searching through them. Additionally, users appeared to follow Nielsen's 'F' pattern (2006) while both browsing and searching through text-based pages. http://tc.eserver.org/31194.html http://tc.eserver.org/31194.html TechSmith's recent release, Morae 2.0, features a new graphing tool, integrated satisfaction survey, and embedded task definitions. The editable marker log in Observer and the improved timeline controls in the Manager improve operator efficiency. This article highlights these and other new features of the new 2.0. http://tc.eserver.org/31199.html http://tc.eserver.org/31199.html TechSmith's Morae is a powerful tool that is useful for any usability study. Its simplicity, efficiency, cost effectiveness and portability has redefined the data collection process and allows researchers to focus on the user experience rather than the data collection tools. This article discusses some of the features of this tool and proposes some enhancements to make it even more powerful to usability practitioners. http://tc.eserver.org/30790.html http://tc.eserver.org/30790.html The three disciplines of usability, testing, and documentation are important to the success of hi-tech products. These three areas are often overlooked, but they have a profound affect on the end user's experience of a software product. As customers become more perceptive about IT, their expectations rise. Since customers now have more choice than ever, to be successful, a software product needs to stand out from the crowd. This article explains why the three disciplines are important to the overall success of a software solution. It concludes with recommendations for obtaining the best return on investment. http://tc.eserver.org/30762.html http://tc.eserver.org/30762.html Storytelling can make documentation more exciting for both writers and readers. Stories provide context and people tend to remember them. More all-∆around fun when stories are tests. http://tc.eserver.org/30655.html http://tc.eserver.org/30655.html 'Test on multiple browsers' has been a mantra ever since there have been multiple browsers to test on. Testing them all--especially these days--is impossible. But you can come a lot closer than you may think. In this article, learn a variety of techniques for cross-browser testing, from the very thorough to the quick and dirty. The choice you make will depend on your resources, but this is an issue you can't ignore. http://tc.eserver.org/30436.html http://tc.eserver.org/30436.html User satisfaction and usefulness are measured using usability studies that involve real customers. Given the nature of software development and delivery, having to conduct usability studies can become a costly expense in the overall budget. A major part of this expense is the participant costs. Under this condition, it is desirable to reduce the number of participants without sacrificing the quality of the experiment. If a company could use a smaller participant pool and get the same results as the entire pool; this would result in significant savings. Given a participant pool of size N, is there a subset of N that would yield the same results as the entire population? This research addresses this question using a data-mining clustering tool called Applications Quest. http://tc.eserver.org/30402.html http://tc.eserver.org/30402.html In just the last few years, we have begun to see research studies comparing usability testing to other techniques for assessing usability. In general, usability testing has found more of the most serious usability problems and fewer of the least serious problems than other methods. Heuristic evaluation--having people evaluate the interface either from their own expertise or from a set of guidelines--has achieved mixed results. Although usability testing seems expensive compared to other methods, it may be less expensive when considered on the basis of 'cost per problem that needs to be fixed.' http://tc.eserver.org/30328.html http://tc.eserver.org/30328.html Usability testing actually measures human factors as they apply to performed tasks based on the interaction between documentation and a product user. As technical communicators, we are becoming increasingly aware of the importance human factors have on our profession. http://tc.eserver.org/30327.html http://tc.eserver.org/30327.html The following is a brief description of how I tested a user manual for a word processing program to be used by low-level and intermediate-level users. http://tc.eserver.org/30313.html http://tc.eserver.org/30313.html Documentation usability testing has been with us at least since 1983. But, to all intents and purposes, it seems that there is still very little testing being done. Of all the documentation quality assurance activities that we can carry out, usability testing represents one of the best values in the process chain. Without devaluing any of the other quality assurance functions, it is safe to say that money spent on usability testing is extremely well spent. http://tc.eserver.org/30310.html http://tc.eserver.org/30310.html With just a modest amount of equipment and effort, you can uncover some serious product glitches and potentially earn your company thousands of dollars and a sterling reputation. The first step is to get a video camera, a product prototype, and a small sample of your target audience. http://tc.eserver.org/30198.html http://tc.eserver.org/30198.html Testing 5-10 users at once lets you conduct large-scale usability testing and still meet your deadlines. http://tc.eserver.org/30045.html http://tc.eserver.org/30045.html This paper compares the relative merit in realistic versus lab style experiments for testing voting technology. By analyzing three voting experiments, we describe the value of realistic settings in showing the enormous challenges for voting process control and consistent voting experiences. The methodology developed for this type of experiment will help other researchers to test polling place protocols and administration. Comparing the results from laboratory experiments with voter verification and realistic voting experiments further validates the procedure of testing equipment in laboratory settings. The methodology and protocol for testing voting systems can be applied to any voting technology. This protocol matches the real-world conditions of voting by replicating them for the experiment. http://tc.eserver.org/29887.html http://tc.eserver.org/29887.html This paper describes how undergraduate students at Mercer University designed and performed hands-on usability tests on seven different products from a wide range of outside clients. Though the projects were challenging and quite difficult for undergraduates, they resulted in significant learning for students and important usability data for the clients. The professor describes the course design and discusses the clients and projects, and three undergraduates report their experiences serving on project teams as project leader, usability specialist, and technical specialist. http://tc.eserver.org/29631.html http://tc.eserver.org/29631.html In this day and age, getting closer to your user base is imperative for creating user-centric documentation. This paper discusses how the Technical Publications group at Mentor Graphics tapped into their annual User Group meeting (MUG) to conduct usability studies. We cover: Convincing management of the ROI of participating in the User Group meeting; establishing relationships with meeting organizers; defining proper "protocol" for interacting with users and other meeting attendees; planning for and dealing with equipment setup; recruiting users to the usability lab; considering and acquiring incentives for usability lab participants. http://tc.eserver.org/29636.html http://tc.eserver.org/29636.html Usability testing doesn't come cheap. You can however, follow test models that will help you improve the quality of your products, including websites. Usability professionals agree that some testing is better than none, and traditional formal usability testing can be adapted to fit your needs and your budget. This paper discusses how all four of these methods: low-cost usability testing, heuristic evaluations, expert reviews, and checkpoints in the development process were used to analyze subsites and applications at a federally funded public health website. http://tc.eserver.org/29646.html http://tc.eserver.org/29646.html In this paper, I describe web page design for those interested in conducting clinical, low vision studies. Ideally, web pages should be accessible and usable for all readers; however, the web is a highly visual medium for communication and creates serious accessibility issues for specialized (diverse needs) those with vision needs. Therefore, I propose that researchers consider a paramount and concurrent user-centered design approach when creating stimulus materials for these specialized audiences. This paper introduces readers to this design approach for a low vision audience as described in the WebText Study. http://tc.eserver.org/29873.html http://tc.eserver.org/29873.html Usability tests make products better. Those of us who have seen their results understand their value, but we sometimes have difficulty convincing managers or developers of their worth. The pay-off of a usability test depends on its context. Both the type of test you perform and the timing of that test influence what you can do with the results. At best, you can guide the design of a product or clearly demonstrate its usability. At worst, you can ruin your credibility and have no effect on the product whatsoever. This paper explores what you get from a usability test at each stage of the user interface design process. It also provides advice on selling testings to management. http://tc.eserver.org/29678.html http://tc.eserver.org/29678.html Eye-tracking systems can enrich a Website usability study by providing an additional method for observing users' behavior. While eye-trackers can provide valuable data, the pros and cons of adding eye-tracking to a usability study need to be considered before designing the study's protocol. This paper discusses the kinds of usability questions that benefit from eye-tracking data and considerations for designing and running the study. Our findings are based on work done in the Laboratory for Usability Testing and Evaluation (LUTE) at the University of Washington, which is equipped with the Eye-gaze Response Computer Aid (ERICA) system controlled by the Gaze Tracker software, both of which were developed by Eye Response Technologies, Inc. http://tc.eserver.org/29691.html http://tc.eserver.org/29691.html Software products are tested. Hardware products are tested. But, how often and how thoroughly are the information deliverables tested? In the rush to bring products to market, the full and thorough testing of information often gets overlooked. Last minute changes are crammed in. Procedures are not thoroughly tested. Even something as simple as testing links can be ignored due to lack of time or resources. http://tc.eserver.org/29701.html http://tc.eserver.org/29701.html This paper reports on usability testing within five leading academic programs in technical communication. The authors give some background on usability testing at their respective institutions, describe their facilities, and briefly relate how their programs in technical communication incorporate usability testing into teaching, research, and consulting. http://tc.eserver.org/29702.html http://tc.eserver.org/29702.html What is current practice in usability testing? How has it changed? What is essential for a good test and what is optional? We compare typical usability testing practice in the past (10+ years ago) with what we find is typical today. Then we look forward to predict what may happen in the future. We predict trends towards testing as a purchasable commodity, more remote testing, as technology makes it easier to ‘observe’ users over the Internet and more ‘mass market’ testing as businesses like Amazon try out their design ideas by micro-launching variants of their web site to see which one plays best with their customers. http://tc.eserver.org/29563.html http://tc.eserver.org/29563.html Eye tracking tools have recently attracted attention from usability professionals. Eye tracking offers usability researchers a new way to identify very fine-grained behaviors that indicate usability problems. This paper is a comparison of different types of eye tracking tools and their potential usefulness in usability testing. Specifically, the paper examines the cost of the systems, system types, sampling rate, and some system limitations. The paper aims to provide a basic introduction to technical communicators who are considering adding an eye- tracking system to their toolkit. http://tc.eserver.org/29164.html http://tc.eserver.org/29164.html Teachers often test course materials by using them in class. Usability testing provides an alternative: teachers receive student feedback and revise materials before teaching a class. Case studies based on interviews and observations with two teaching assistants who usability tested materials before teaching introductory technical writing demonstrate how usability testing can make novice teachers more confident about and help them predict student experiences with their assignments. By helping to train teachers, usability testing can also help better serve students. http://tc.eserver.org/28952.html http://tc.eserver.org/28952.html Having a good designer doesn't eliminate the need for a systematic usability process. Risk reduction and quality improvement both require user testing and other usability methods. http://tc.eserver.org/28935.html http://tc.eserver.org/28935.html The topic of test subject compensation generates a lot of conversation...how do you motivate test participants? http://tc.eserver.org/28915.html http://tc.eserver.org/28915.html One of the principles underlying usability testing is that observing a user perform a task provides more reliable information than simply asking the user how easy it would be to perform the task. By observing users, you can assess whether they are actually able to use a product. By asking them, you simply cannot. http://tc.eserver.org/28909.html http://tc.eserver.org/28909.html Everyone knows by now that they should test the usability of their applications, but still hardly anybody actually does it. In this article I'll share some tips I've picked up for doing usability tests quickly and effectively. Relatively recent tools like Django and Ruby on Rails allow us to develop projects faster and to make significant changes later in the project timeline. Usability testing methods should now be adapted to fit this modern approach to development. http://tc.eserver.org/28894.html http://tc.eserver.org/28894.html The bottom line is how to ensure the customer that eye tracking provides additional value for their money. If we do numerical analysis in addition to video analysis, the need for extra time is remarkable and the analysis will become more expensive. To reduce analysis time we need automated special software and therefore we are currently developing scan path visualization software in which we include a new fixation recognition algorithm. http://tc.eserver.org/28824.html http://tc.eserver.org/28824.html How do you review a product for usability, but make that review user-centered? http://tc.eserver.org/28818.html http://tc.eserver.org/28818.html Once a site has been developed (or partially developed), you can test it to ensure that it is usable and spot any potential problems. http://tc.eserver.org/28761.html http://tc.eserver.org/28761.html What does 'bias' mean in usability testing? Is it bad? good? in between? What are sources of bias in usability testing? Which one(s) should we worry about most? How do we know our methods are any good? Especially when we all do things differently? http://tc.eserver.org/28700.html http://tc.eserver.org/28700.html The sooner you complete a usability study, the higher its impact on the design process. Slower methods should be deferred to an annual usability checkup. http://tc.eserver.org/28617.html http://tc.eserver.org/28617.html Usability Leistungsspektrum Die ausgefeiltesten digitalen Strategien scheitern oft am Einfachsten: der Usability. Doch in einer Zeit, in der Ihr Wettbewerber nur einen Mausklick weit entfernt ist, stellt Usability eine der größten Herausforderungen im Bereich der digitalen Kommunikation dar. http://tc.eserver.org/28378.html http://tc.eserver.org/28378.html 観察している人たちを最優先に考えた調査をすべきではない。たとえ観察していてつまらないタスクばかりだとしても、デザインを真に検証するテストを実施すべきだ。 http://tc.eserver.org/28365.html http://tc.eserver.org/28365.html A review of software packages for moderated usability testing in a native environment via a telephone/web interface. http://tc.eserver.org/28344.html http://tc.eserver.org/28344.html There are two major considerations when conducting usability testing. The first is to ensure that the best possible method for testing is used. Generally, the best method is to conduct a test where representative participants interact with representative scenarios. The tester collects data on the participant's success, speed of performance, and satisfaction. The findings, including both quantitative data and qualitative observations information, are provided to designers in a test report. Using 'inspection evaluations,' in place of well-controlled usability tests, must be done with caution. Inspection methods, such as heuristic evaluations or expert reviews, tend to generate large numbers of potential usability 'problems' that never turn out to be actual usability problems. http://tc.eserver.org/28294.html http://tc.eserver.org/28294.html The Graphical User Interface (GUI) of software usually consists of huge number of icons. Though the intention is to improve the usability of software, not all interface designers are able to test and evaluate the comprehensibility of icons. Increasing exposure to unevaluated icons causes cognitive fatigue to users and slows down the intuitive learning. Users from diverse geographic locations, cultures and religions are very likely to interpret and understand these icons differently. As software products are designed to address universal needs, testing and evaluation of GUI across the globe or at least, wherever the product is likely to be used becomes important. Creation of dedicated usability labs in various locations for usability testing is not a viable proposition. A software tool named 'UniFace' for remote usability testing of icons is designed capitalizing on far-reaching capability of Internet. UniFace extends the usability lab onto the desktop of every user. http://tc.eserver.org/28287.html http://tc.eserver.org/28287.html The web site for a Master's level course on Usability Testing in the Technical Communication department at the University of Washington. http://tc.eserver.org/28166.html http://tc.eserver.org/28166.html The more intuitive a product's interfaces and procedures become, the more usable it becomes. Thus, the less formal documentation it requires. To do our part, we can strive to reduce the number of words a customer needs to read. Focusing on clarity and concision, we can take pride that of the words that remain, every word will count. Working with usability experts, our fellow customer advocates, we can transmute unwieldy products into easily used ones. To me, that's a compelling reason to remain a technical communicator, regardless of why we became one. http://tc.eserver.org/28102.html http://tc.eserver.org/28102.html Contrasts the unique aspects of expert reviews and usability testing. The usability goals they address are different. Know when to use which one, and when to use both. http://tc.eserver.org/28092.html http://tc.eserver.org/28092.html Don't run your studies for the benefit of the people in the observation room. Test to discover the truth about the design, even when user tasks are boring to watch. http://tc.eserver.org/28020.html http://tc.eserver.org/28020.html A case study was done to study whether usability expert evaluation and testing are suitable for game development. In the study, a computer game under development was first evaluated and then tested. Game developers were then asked to rate the findings and give other feedback about the methods used and the results gained. It was found that the usability expert evaluation and testing provided both novel and useful data for game development. Based on these and the other results it is argued that the usability expert evaluation and testing have considerable face validity in game development. In addition to the usefulness and face validity of the methods it was studied whether the usability experts participating in the game usability expert evaluation should be double experts. It was found that there was no significant difference in the number or the rated relevancy of the problem the gamer and non-gamer usability specialists found. http://tc.eserver.org/28023.html http://tc.eserver.org/28023.html This paper argues that in the field of usability, debates about number of users, the use of statistics, etc. in the abstract are pointless and even counter-productive. We propose that the answers depend on the research questions and business objectives of each project and thus cannot be discussed in absolute terms. Sometimes usability testing is done with an implicit or explicit hypothesis in mind. At other times the purpose of testing is to guide iterative design. These two approaches call for different study designs and treatment of data. We apply control systems theory to the topic of usability to highlight and frame the value of iterative usability testing in the design lifecycle. Within this new metaphor, iterative testing is a form of feedback which is most effective and resource-efficient if done as often as practically possible with project resources and timelines in mind. http://tc.eserver.org/28024.html http://tc.eserver.org/28024.html Many usability practitioners conduct most of their usability evaluations to improve a product during its design and development. We call these 'formative' evaluations to distinguish them from 'summative' (validation) usability tests at the end of development. A standard for reporting summative usability test results has been adopted by international standards organizations. But that standard is not intended for the broader range of techniques and business contexts in formative work. This paper reports on a new industry project to identify best practices in reports of formative usability evaluations. The initial work focused on gathering examples of reports used in a variety of business contexts. We define elements in these reports and present some early guidelines on making design decisions for a formative report. These guidelines are based on considerations of the business context, the relationship between author and audience, the questions that the evaluation is trying to answer, and the techniques used in the evaluation. Future work will continue to investigate industry practice and conduct evaluations of proposed guidelines or templates. http://tc.eserver.org/28022.html http://tc.eserver.org/28022.html Usability testing a mobile application in the laboratory seems to be sufficient when studying user interface and navigation issues. The usability of a consumer application was tested in two environments: in a laboratory and in a field with a total of 40 test users. The same problems were found in both environments, differences occurred in the frequency of findings between the contexts. Results indicate that conducting a time-consuming field test may not be worthwhile when searching user interface flaws to improve user interaction. In spite of this, it is possible that field testing is worthwhile when combining usability tests with a field pilot or contextual study where user behavior is investigated in a natural context. http://tc.eserver.org/28017.html http://tc.eserver.org/28017.html Small sample sizes are a fact of life for most usability practitioners. This can lead to serious measurement problems, especially when making binary measurements such as successful task completion rates (p). The computation of confidence intervals helps by establishing the likely boundaries of measurement, but there is still a question of how to compute the best point estimate, especially for extreme outcomes. In this paper, we report the results of investigations of the accuracy of different estimation methods for two hypothetical distributions and one empirical distribution of p. If a practitioner has no expectation about the value of p, then the Laplace method ((x+1)/(n+2)) is the best estimator. If practitioners are reasonably sure that p will range between .5 and 1.0, then they should use the Wilson method if the observed value of p is less than .5, Laplace when p is greater than .9, and maximum likelihood (x/n) otherwise. http://tc.eserver.org/27941.html http://tc.eserver.org/27941.html 6% of task attempts are extremely slow and constitute outliers in measured user performance. These sad incidents are caused by bad luck that designers can -- and should -- eradicate. http://tc.eserver.org/27897.html http://tc.eserver.org/27897.html We can define usability in terms of quality metrics, such as learning time, efficiency of use, memorability, user errors, and subjective satisfaction. Sadly, few projects collect such metrics because doing so is expensive: it requires four times as many users as simple user testing. Many users are required because of the substantial individual differences in user performance. When you measure people, you'll always get some who are really fast and some who are really slow. Given this, you need to average these measures across a fairly large number of observations to smooth over the variability. http://tc.eserver.org/27701.html http://tc.eserver.org/27701.html When doing website tasks, the slowest 25% of users take 2.4 times as long as the fastest 25% of users. This difference is much higher than for other types of computer use; only programming shows a greater disparity. http://tc.eserver.org/27526.html http://tc.eserver.org/27526.html This article discusses how eye-tracking can be used to supplement traditional usability test measures. User performance on two usability tasks with three e-commerce websites is described. Results show that eye-tracking data can be used to better understand how users initiate a search for a targeted link or web object. Frequency, duration and order of visual attention to Areas of Interest (AOIs) in particular are informative as supplemental information to standard usability testing in understanding user expectations and making design recommendations. http://tc.eserver.org/27411.html http://tc.eserver.org/27411.html It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%. http://tc.eserver.org/27405.html http://tc.eserver.org/27405.html Doesn't one need to test with at least 100 or more users for statistical significance, accuracy and validity? http://tc.eserver.org/27413.html http://tc.eserver.org/27413.html Some people think that usability is very costly and complex and that user tests should be reserved for the rare web design project with a huge budget and a lavish time schedule. Not true. Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford. http://tc.eserver.org/27144.html http://tc.eserver.org/27144.html Good usability testing definitely provides an opportunity for clear-cut improvements in the usability of Web sites. In order to increase the chances of success with usability testing, it is important to measure effectiveness, efficiency, and satisfaction–they all measure different aspects of the usability of a Web site. If only one or two of these measures are used, it would provide an incomplete or partial picture of the possible human performance and user satisfaction results. http://tc.eserver.org/27147.html http://tc.eserver.org/27147.html To meet your users’ needs, it is essential to know your audience and to design for them. A key way to do this is by identifying your Web site’s primary users and recruiting a sample for usability testing. Consider these four aspects. http://tc.eserver.org/27146.html http://tc.eserver.org/27146.html Repeated (iterative) usability testing finds problems sooner, and usability engineering helps agencies save money. With those ideas in mind, it makes sense that agencies would want to submit their sites and other products to usability testing; doing so is an effective use of taxpayer money, especially with regard to performance measures. http://tc.eserver.org/27019.html http://tc.eserver.org/27019.html Over the coming months, I'll use eyetracking to evaluate a lot of world-renowned user interfaces--including Web sites like Amazon.com, Google News, and eBay; Rich Internet Applications (RIAs); and desktop applications--and analyze quantitative eyetracking data to provide best practices for designing user interface elements like navigation systems, menus, and forms, and for effective ad placement. http://tc.eserver.org/26946.html http://tc.eserver.org/26946.html A compilation of usability, usability testing, and user-centered design resources by Usernomics. http://tc.eserver.org/26871.html http://tc.eserver.org/26871.html This article discusses how eye-tracking can be used to supplement traditional usability test measures. User performance on two usability tasks with three e-commerce websites is described. Results show that eye-tracking data can be used to better understand how users initiate a search for a targeted link or web object. Frequency, duration and order of visual attention to Areas of Interest (AOIs) in particular are informative as supplemental information to standard usability testing in understanding user expectations and making design recommendations. http://tc.eserver.org/26851.html http://tc.eserver.org/26851.html How do software companies evaluate whether accessibility criteria mandated by law are met? Confirmation is often provided by filling out a checklist. However, the method used for determining compliance to the checklist is not specified. Typically the task of filling out the checklist is done by accessibility specialists, usability professionals, quality assurance testers, or, in one case we know of, the development team that wrote the software. We have conducted several types of accessibility evaluations, walkthroughs, and testing with scenarios by sighted test participants and testing by blind test participants. While testing with blind participants takes considerable preparation time, we have uncovered important findings that were not revealed with sighted participants. We consider accessibility testing by blind participants an important component of our evaluations. http://tc.eserver.org/26852.html http://tc.eserver.org/26852.html Many disabled or elderly users have embraced the Web as a way to get easier access to information, services, contacts with others, and entertainment. But very often, Web sites are creating barriers for them, at the same time as they are reducing barriers of access. We can safely assume that Web sites are made by designers who have no intention to exclude groups of users from using the site. Our studies, however, have proved that good intentions are not enough to create Web content that is accessible and also usable for people with various kinds of physical, sensory, or cognitive disabilities. Web designers need to familiarize themselves with accessibility guidelines and apply them properly, but that goal is seldom achieved. Even when the guidelines are applied, a site's ease of use for people with disabilities needs to be confirmed in a usability test with actual users who have disabilities. This article describes how to conduct user-focused tests with test participants who are elderly or who have disabilities. http://tc.eserver.org/26838.html http://tc.eserver.org/26838.html Performance testing is a rigorous usability evaluation of a working system under realistic conditions to identify usability problems and to compare measures such as success rate, task time and user satisfaction with requirements. http://tc.eserver.org/26537.html http://tc.eserver.org/26537.html You've spent several days setting up a usability test, recruiting the participants and running it. Then you've pored over the data. What next? If you are doing usability testing as part of user-centred design within a business setting, then there are many ways that you can communicate the results. This paper looks at reports and then considers presentation and observation as alternatives to reports. http://tc.eserver.org/26487.html http://tc.eserver.org/26487.html Eight essential guidelines for usability testing, helping you to plan, run and analyse usability tests. http://tc.eserver.org/26368.html http://tc.eserver.org/26368.html This article prioritizes areas of the Microsoft Windows XP interface that can be tested to ensure compatibility between assistive technologies and Windows XP. http://tc.eserver.org/26261.html http://tc.eserver.org/26261.html Usability testing is the qualitative measurement of user experience with a product or service, such as a website. Usability testing is for anyone who wants to make their website, software, device, etc. easier to use. Whether you employ thousands of people such as Microsoft or fall into the small business category, usability testing can play a significant role in giving you direct feedback from your customers. http://tc.eserver.org/26258.html http://tc.eserver.org/26258.html I've often observed that once wireframing begins, it's off to the races! In the rush to launch, we sometimes forget end-users. Is there a way to ensure that they get a voice during this always-hectic phase? http://tc.eserver.org/26260.html http://tc.eserver.org/26260.html This tutorial shows a very high-fidelity prototype—based upon the current gotomedia site—that might be created very late in the design process. http://tc.eserver.org/25746.html http://tc.eserver.org/25746.html Despite the proliferation of books and articles dealing with usability testing as a sub-field of technical communication, there remains one artifact that is underdefined: the usability highlights video. Several sources suggest that usability testers should deliver a video to their clients, but there is no common description of the contents of this video or the rhetorical purpose it fulfills. In this article, we examine the use of the highlights video as described in the literature, but we go further to discover some ways practicing usability professionals understand the role of the highlights video. Through the use of generic conventions, rhetorical theory, and industrial practices, we attempt to draw conclusions that point to some common uses of the highlights video that can instruct both teachers in the usability classroom and practicing usability experts as they create videos for client projects. http://tc.eserver.org/25733.html http://tc.eserver.org/25733.html As a cognitive framework for making meaning of the world, the narrative provides a powerful form for structuring information, and has been adopted as a useful design framework for many communicative forms, including interactive media. This paper reports on the use of visual narrative for user-testing an interactive museum show. The viewers’ perceived narratives of a sequence of graphics from a show on brain science were compared to the designers’ intended narrative. Mapping the audience’s reading of the visual arguments proved a useful testing structure in developing the show, with color and pattern tracking proving especially critical when viewers experienced novel or abstract information. http://tc.eserver.org/25695.html http://tc.eserver.org/25695.html Ninety percent of all usability testing performed on Web sites is useless. This is not to say that it doesn’t have a significant role to play in user experience design. When done right, usability testing will improve your Web site and your development process, but the current culture surrounding Web site usability testing is such that it rarely benefits the design. Worse, this misapplication can undermine the acceptance of this important technique throughout an organization. http://tc.eserver.org/25694.html http://tc.eserver.org/25694.html If you're interested in Web site usability then it's likely that you've come across the concept of professional usability testing. It's also likely that the practicalities of testing have remained a mystery, because very little has been published about it on the Web - until now. http://tc.eserver.org/25385.html http://tc.eserver.org/25385.html Success in usability testing is learning what you need to know. That includes finding out both what is working well and what is not working well. Focusing on formative tests—with an eye toward identifying problems and bringing the issues to the team—is the secret to successful usability testing. http://tc.eserver.org/25194.html http://tc.eserver.org/25194.html Despite being an artificial situation, user testing generates realistic findings because people engage strongly with the tasks and suspend their disbelief. http://tc.eserver.org/25190.html http://tc.eserver.org/25190.html Southwest Airlines' low-cost methods for formal usability testing. http://tc.eserver.org/24757.html http://tc.eserver.org/24757.html Most people use Microsoft Excel to analyse the results of usability tests, but did you know you can use it to collect the data too? This spreadsheet allows you to measure task completion rates, analyse questionnaire data, and summarise participant comments. It even includes a timer so you can measure time-on-task. http://tc.eserver.org/24758.html http://tc.eserver.org/24758.html For many usability engineers and human factors researchers, basic note-taking on a paper form or laptop computer represents the common data collection strategy. Unfortunately, this approach can be cumbersome and slow to compile the data following the completion of a study. http://tc.eserver.org/24699.html http://tc.eserver.org/24699.html Testing documents at each step of their development is one way to ensure that the final document is of high quality. It is not necessary to wait until a document is complete before we start testing; we can incorporate iterative testing into the information-development process so that we can build in quality each step of the way. http://tc.eserver.org/24695.html http://tc.eserver.org/24695.html The term 'usability testing' means different things to different people. In its 'high fidelity' form, it connotes two-way mirrors, fancy electronic equipment, sophisticated logging software, and tomes of test data. However, high fidelity options aren’t always required for learning about or performing substantive usability testing. Even in a 'low fidelity' environment, you can hit all of the usability high points and provide a valuable service to your software developers and users. http://tc.eserver.org/24613.html http://tc.eserver.org/24613.html Support, teamwork, and process are the keystones of our renewed efforts to reap the benefits of usability testing. Best practices from the literature led us to adopt Nielsen's suggestion to build reliance on task-based scenarios, heuristic evaluations, and protocol-aided revision. But integrating usability testing with documentation development processes is crucial. Our approach includes getting writers involved in planning and conducting informal and formal usability tests throughout the development cycle. Implementation is based on ISO-compliant work instructions and product information guidelines. Usability measurements and process metrics help demonstrate results and added value.