Animated Character Likeability Revisited: The Case of Interactive TV  

Animated characters have been a popular research theme, but the respective desktop applications have not been well-received by end-users. The objective of this study was to evaluate the use of an animated character for presenting information and navigating music videos within an interactive television (ITV) application. Information was displayed over music video clips with two alternative user interfaces: 1) semi-transparent information overlays, 2) an animated character. For this purpose, the differences between ITV and desktop computing motivated the adaptation of the traditional usability evaluation techniques. The evaluation revealed that users reported higher affective quality with the animated character user interface. Although the success of animated characters in desktop productivity applications has been limited, there is growing evidence that animated characters might be viable in a domestic environment for leisure activities, such as interactive TV.

Chorianopoulos, Konstantinos. Journal of Usability Studies (2006). Design>Multimedia>Interactive>Video

Can Collaboration Help Redefine Usability?    

A collaborative knowledge space would provide great value to the usability community. In particular it would: Help define the field and give it a presence that provides professionals and the public with a single source for theoretical, practical and speculative information about usability; encourage the integration of research and practice; invite colleagues in related fields to participate and share their perspectives; serve as a platform to advance our understanding of collaboration and knowledge management tools. Most of the tools needed to implement a collaborative knowledge space are already available and there are a number of related activities already underway that could feed into this project. It would be a great deal of work but I believe it would also yield a great deal of benefit.

Kreitzberg, Charles B. Journal of Usability Studies (2006). Articles>Usability>Collaboration

Case Study: Conducting Large-Scale Multi-User User Tests on the United Kingdom Air Defence Command and Control System    

IBM was contracted to provide a new Air Defence Command and Control (ADCC) system for the Royal Air Force. The IBM Human Factors (HF) team was responsible for the design of the operations room, workstations and the graphical user interfaces. Because the project was safety-related, IBM had to produce a safety case. One aspect of the safety case was a demonstration of the operational effectiveness of the new system. This paper is an in-depth case study of the user testing that was carried out to demonstrate the effectiveness of the system. Due to time constraints the HF team had to observe five participants working simultaneously. Further, to provide a realistic operational environment, up to twenty-eight operators were required for each test. The total effort for this activity was four person-years. The paper will detail the considerations, challenges and lessons learned in the creation and execution of these multi-user user tests.

Hey, Elliott. Journal of Usability Studies (2006). Articles>Usability>Web Design

Clustering for Usability Participant Selection  

User satisfaction and usefulness are measured using usability studies that involve real customers. Given the nature of software development and delivery, having to conduct usability studies can become a costly expense in the overall budget. A major part of this expense is the participant costs. Under this condition, it is desirable to reduce the number of participants without sacrificing the quality of the experiment. If a company could use a smaller participant pool and get the same results as the entire pool; this would result in significant savings. Given a participant pool of size N, is there a subset of N that would yield the same results as the entire population? This research addresses this question using a data-mining clustering tool called Applications Quest.

Gilbert, Juan E., Andrea Williams, and Cheryl D. Seals. Journal of Usability Studies (2007). Articles>Usability>Testing>Methods

Review: Comments on: Selker, Rosenzweig, and Pandolfo (2006). "A Methodology for Testing Voting Systems"    

In the article, 'A Methodology for Testing Voting Systems' (JUS, November 2006, pp7-21), Selker, Rosenzweig, and Pandolfo discuss their methodology for usability testing of voting systems. With so much at stake in the usability of our ballots and voting systems, we can only applaud any research in this field. There is little history of research in this area, so discussions of test protocols are especially valuable. Unfortunately, although this article sets out to compare 'the relative merit in realistic versus lab style experiments for testing voting technology,' it falls short of this goal. If their point is that real-world testing is important because real election environments add burdens that are not present in lab settings, this conclusion is not supported by any of the work described.

Quesenbery, Whitney, John Cugini, Dana Chisnell, Bill Killam and Janice C. 'Ginny' Redish. Journal of Usability Studies (2007). Articles>Reviews>Usability>Civic

Culture and Usability Evaluation: The Effects of Culture in Structured Interviews  

A major impediment in global user interface development is that there is inadequate empirical evidence for the effects of culture in the usability engineering methods used for developing these global user interfaces. This paper presents a controlled study investigating the effects of culture on the effectiveness of structured interviews in international usability evaluation. The experiment consisted of a usability evaluation of a website with two independent groups of Indian participants. Each group had a different interviewer; one belonging to the Indian culture and the other to the Anglo-American culture. The results show that participants found more usability problems and made more suggestions to an interviewer who was a member of the same (Indian) culture than to the foreign (Anglo-American) interviewer. The results of the study empirically establish that culture significantly affects the efficacy of structured interviews during international user testing. The implications of this work for usability engineering are discussed.

Vatrapu, Ravi and Manuel A. Pérez-Quiñones. Journal of Usability Studies (2006). Articles>Usability>Interviewing>Cultural Theory

Culture: Wanted? Alive or Dead?    

Is culture dead as a topic of interest to usability and user-interface usability and design professionals? One European anthropologist/ethnographer wrote recently that 'culture is dead' and only of interest to people in the USA (who seemingly have little or no understanding of other cultures around the world). On the other hand, another (USA) usability/design professional recently stated that she thought cross-cultural issues were one of the most important and potent trends in product/service development. Who is right?

Marcus, Aaron. Journal of Usability Studies (2006). Articles>Usability>Cultural Theory

Decision Models for Comparative Usability Evaluation of Mobile Phones Using the Mobile Phone Usability Questionnaire (MPUQ)  

A comparative usability evaluation was performed using various subjective evaluation methods, including Mobile Phone Usability Questionnaire (MPUQ). Further, decision-making models using Analytic Hierarchy Process (AHP) and multiple linear regression were developed and applied. Although the mean rankings of the four phones were not significantly different across the evaluation methods, there were variations across the methods in terms of the number of rank orderings, preference proportions, and methods to select their initial preference. Thus, this study provided a useful insight into how users make different decisions through different evaluation methods. Also, the result showed that answering a usability questionnaire affected a user's decision-making process for comparative evaluation.

Ryu, Young Sam, Kari Babski-Reeves, Tonya L. Smith-Jackson and Maury A. Nussbaum. Journal of Usability Studies (2007). Articles>Usability>Assessment>Surveys

Do Usability Expert Evaluation and Testing Provide Novel and Useful Data for Game Development?    

A case study was done to study whether usability expert evaluation and testing are suitable for game development. In the study, a computer game under development was first evaluated and then tested. Game developers were then asked to rate the findings and give other feedback about the methods used and the results gained. It was found that the usability expert evaluation and testing provided both novel and useful data for game development. Based on these and the other results it is argued that the usability expert evaluation and testing have considerable face validity in game development. In addition to the usefulness and face validity of the methods it was studied whether the usability experts participating in the game usability expert evaluation should be double experts. It was found that there was no significant difference in the number or the rated relevancy of the problem the gamer and non-gamer usability specialists found.

Laitinen, Sauli. Journal of Usability Studies (2006). Articles>Usability>Testing>Methods

Empirical Evaluation of a Popular Cellular Phone's Menu System: Theory Meets Practice    

A usability assessment entailing a paper prototype was conducted to examine menu selection theories on a small screen device by determining the effectiveness, efficiency, and user satisfaction of a popular cellular phone's menu system. Outcomes of this study suggest that users prefer a less extensive menu structure on a small screen device. The investigation also covered factors of category classification and item labeling influencing user performance in menu selection. Research findings suggest that proper modifications in these areas could significantly enhance the system's usability and demonstrate the validity of paper-prototyping which is capable of detecting significant differences in usability measures among various model designs.

Huang, Sheng-Cheng, I-Fan Chou and Randolph G. Bias. Journal of Usability Studies (2006). Articles>User Interface>Usability>User Centered Design

The Great Leap Forward: The Birth of the Usability Profession (1988-1993)    

In this editorial, I describe our birth and some personal experiences as I lived through those times. I present these observations, not as a historian, but as a usability professional viewing events of 15 years ago through my personal filter.

Dumas, Joe. Journal of Usability Studies (2007). Articles>Usability>History

Heuristic Evaluation Quality Score (HEQS): A Measure of Heuristic Evaluation Skills  

Heuristic Evaluation is a discount usability engineering method involving three or more evaluators who evaluate the compliance of an interface based on a set of heuristics. Because the quality of the evaluation is highly dependent on their skills, it is critical to measure these skills to ensure evaluations are of a certain standard. This study provides a framework to quantify heuristic evaluation skills. Quantification is based on the number of unique issues identified by the evaluators as well as the severity of each issue. Unique issues are categorized into eight user interface parameters and severity is categorized into three. A benchmark computed from the collated evaluations is used to compare skills across applications as well as within applications. The result of this skill measurement divides the evaluators into levels of expertise. Two case studies illustrate the process, as well as its applications. Further studies will help define an expert's profile.

Kirmani, Shazeeye and Shamugam Rajasekaran. Journal of Usability Studies (2007). Articles>Usability>Assessment>Heuristic Evaluation

Iterative Usability Testing as Continuous Feedback: A Control Systems Perspective    

This paper argues that in the field of usability, debates about number of users, the use of statistics, etc. in the abstract are pointless and even counter-productive. We propose that the answers depend on the research questions and business objectives of each project and thus cannot be discussed in absolute terms. Sometimes usability testing is done with an implicit or explicit hypothesis in mind. At other times the purpose of testing is to guide iterative design. These two approaches call for different study designs and treatment of data. We apply control systems theory to the topic of usability to highlight and frame the value of iterative usability testing in the design lifecycle. Within this new metaphor, iterative testing is a form of feedback which is most effective and resource-efficient if done as often as practically possible with project resources and timelines in mind.

Genov, Alex. Journal of Usability Studies (2006). Articles>Usability>Testing>Methods

Journal of Usability Studies  

This peer-reviewed publication is dedicated to promoting and enhancing the practice, research, and methods of usability engineering. Its aim is to provide usability practitioners and researchers with a forum to share usability research case studies, empirical findings, opinions and experiences in the practice and teaching of usability engineering, and good practices in usability engineering.

Journal of Usability Studies. Journals>Usability

Making Usability Recommendations Useful and Usable    

This paper evaluates the quality of recommendations for improving a user interface resulting from a usability evaluation. The study compares usability comments written by different authors, but describing similar usability issues. The usability comments were provided by 17 professional teams who independently evaluated the usability of the website for the Hotel Pennsylvania in New York. The study finds that only 14 of the 84 studied comments (17%) addressing six usability problems contained recommendations that were both useful and usable. Fourteen recommendations were not useful at all. Sixteen recommendations were not usable at all. Quality problems include recommendations that are vague or not actionable, and ones that may not improve the overall usability of the application. The paper suggests characteristics for "useful and usable recommendations," that is, recommendations for solving usability problems that lead to changes that efficiently improve the usability of a product.

Molich, Rolf, Robin Jeffries and Joseph Dumas. Journal of Usability Studies (2007). Articles>Usability>Assessment>Reports

Metaphor-Based Design of High-Throughput Screening Process Interfaces    

This paper describes work on developing usable interfaces for creating and editing methods for high-throughput screening of chemical and biological compounds in the domain of life sciences automation. A modified approach to metaphor-based interface design was used as a framework for developing a screening method editor prototype analogous to the presentation of a recipe in a cookbook. The prototype was compared to an existing screening method editor application in terms of effectiveness, efficiency, and satisfaction of novice users and was found to be superior.

Kaber, David B., Noa Segall and Rebecca S. Green. Journal of Usability Studies (2007). Articles>User Interface>Rhetoric>Tropes

A Methodology for Testing Voting Systems  

This paper compares the relative merit in realistic versus lab style experiments for testing voting technology. By analyzing three voting experiments, we describe the value of realistic settings in showing the enormous challenges for voting process control and consistent voting experiences. The methodology developed for this type of experiment will help other researchers to test polling place protocols and administration. Comparing the results from laboratory experiments with voter verification and realistic voting experiments further validates the procedure of testing equipment in laboratory settings. The methodology and protocol for testing voting systems can be applied to any voting technology. This protocol matches the real-world conditions of voting by replicating them for the experiment.

Selker, Ted, Elizabeth Rozenwieg and Anna Pandolfo. Journal of Usability Studies (2006). Articles>Usability>Testing>Civic

A Pattern Language Approach to Usability Knowledge Management    

Knowledge gained from usability testing is often applied merely to the immediate product under test and then forgotten--at least at an organizational level. This article describes a usability knowledge management system (KMS) based on principles of pattern language and use-case writing that offers a way to turn lessons learned from usability testing into organizational knowledge that can be leveraged across different projects and different design teams.

Hughes, Michael A. Journal of Usability Studies (2006). Articles>Usability>Knowledge Management>Language

Post-Modern Usability    

"Don't people in your field actually know anything?" From what I read on the list-servs, blogs, and other places where usability professionals hang out, there are many people even in our field who would answer "No!" They tend to see usability as a craft and question whether anything can be known about human-system design in a way that can be codified, or at least codified in a useful way. I would disagree, but I do think our field is passing through an evolutionary stage, where we have been working toward satisfying the business requirements for improved productivity.

Lund, Arnold M. Journal of Usability Studies (2006). Articles>Usability

Problems and Joys of Reading Research Papers for Practitioner Purposes  

Discusses reasons that practitioners read research papers and the obstacles that they face when reading research papers. Jarrett provides several examples and suggestions for improving the accessibility of research papers for practitioners. Her suggestions include writing clear titles, ensuring that the abstract states the study population and limitations of the study, and ensuring that the conclusions are written clearly. She also discusses her criteria for determining whether or not a research paper is relevant to her work.

Jarrett, Caroline. Journal of Usability Studies (2007). Articles>Publishing>Research>Writing

Reliability and Validity of the Mobile Phone Usability Questionnaire  

This study was a follow-up to determine the psychometric quality of the usability questionnaire items derived from a previous study (Ryu and Smith-Jackson, 2005), and to find a subset of items that represents a higher measure of reliability and validity. To evaluate the items, the questionnaire was administered to a representative sample involving approximately 300 participants. The findings revealed a six-factor structure, including (1) Ease of learning and use, (2) Assistance with operation and problem solving, (3) Emotional aspect and multimedia capabilities, (4) Commands and minimal memory load, (5) Efficiency and control, and (6) Typical tasks for mobile phones. The appropriate 72 items constituted the Mobile Phone Usability Questionnaire (MPUQ), which evaluates the usability of mobile phones for the purpose of making decisions among competing variations in the end-user market, determining alternatives of prototypes during the development process, and evolving versions during an iterative design process.

Ryu, Young Sam and Tonya L. Smith-Jackson. Journal of Usability Studies (2006). Design>Web Design>Wireless Web>Surveys

A Structured Process for Transforming Usability Data into Usability Information  

Much research has been devoted to developing usability evaluation methods that are used in evaluating interaction designs. More recently, however, research has shifted away from evaluation methods and comparisons of evaluation methods to issues of how to use the raw usability data generated by these methods. Associated with this focus is the assumption that the transformation of the raw usability data into usability information is relatively straightforward. We would argue that this assumption is incorrect, especially for novice usability practitioners. In this article, we present a structured process for transforming raw usability data into usability information that is based on a new way of thinking about usability problem data. The results of a study of this structured process indicate that it helps improve the effectiveness of novice usability practitioners.

Howarth, Jonathan, Terence S. Andre and Rex Hartson. Journal of Usability Studies (2007). Articles>Usability>Reports>Technical Writing

Surviving Our Success: Three Radical Recommendations    

The world of usability practitioners is undergoing massive changes. I know because I read it in the New York Times.

Spool, Jared M. Journal of Usability Studies (2007). Articles>Usability>Professionalism

System Usability Scale and Non-Native English Speakers  

The System Usability Scale (SUS) was administered verbally to native English and non-native English speakers for several internally deployed applications. It was found that a significant proportion of non-native English speakers failed to understand the word 'cumbersome' in Item 8 of the SUS (that is, 'I found the system to be very cumbersome to use.') This finding has implications for reliability and validity when the questionnaire is distributed electronically in multinational usability efforts.

Finstad, Kraig. Journal of Usability Studies (2006). Articles>Usability>Assessment>Methods

Towards the Design of Effective Formative Test Reports    

Many usability practitioners conduct most of their usability evaluations to improve a product during its design and development. We call these 'formative' evaluations to distinguish them from 'summative' (validation) usability tests at the end of development. A standard for reporting summative usability test results has been adopted by international standards organizations. But that standard is not intended for the broader range of techniques and business contexts in formative work. This paper reports on a new industry project to identify best practices in reports of formative usability evaluations. The initial work focused on gathering examples of reports used in a variety of business contexts. We define elements in these reports and present some early guidelines on making design decisions for a formative report. These guidelines are based on considerations of the business context, the relationship between author and audience, the questions that the evaluation is trying to answer, and the techniques used in the evaluation. Future work will continue to investigate industry practice and conduct evaluations of proposed guidelines or templates.

Theofanos, Mary Frances and Whitney Quesenbery. Journal of Usability Studies (2006). Articles>Usability>Testing>Reports