Assessment

Online Anonymous Rating Sites: Empowering Individual Voices

Rating sites empower people to make better choices. Obviously they are subject to abuse (either from the competition, from the the slandered source, or from biased friends). But even in the possible exaggerations from the participants, the ratings raise awareness of issues that you might otherwise not carefully examine.

Introducing Business Activity Monitoring

Typically, an organization's processes span multiple systems, channels, applications, departments, and external partners. In this case, how do we monitor such processes? What is the current state of the organizational processes? What is the benchmark for poorly-performing processes and exceptional processes? Most of the time, organizations are unable to answer such questions, or only have a vague idea for various reasons. Either they are monitoring the process with a very limited scope, or the mechanisms for monitoring the process are not in place to allow such details to be available. We rarely find organizations with process owners having an end-to-end view of a process. The big picture of a process is not available to the decision makers on a real-time basis.

User Value: Competing Theories and Models

In design research, the issues of what exactly constitutes user value and how design can contribute to its creation are not commonly discussed. This paper provides a critical overview of the theories of value used in anthropology, sociology, philosophy, business, and economics. In doing so, it reviews a range of theoretical and empirical studies, with particular emphasis on their position on product, user, and designer in the process of value creation. The paper first looks at the similarities and differences among definitions of value as exchange, sign, and experience. It then reviews types and properties of user value such as its multidimensionality, its contextuality, its interactivity, and the stages of user experience dependency identified by empirical studies. Methodological approaches to user value research and their possible applications in design are also discussed. Finally, directions for future research on user value are discussed giving particular emphasis to the need of tools and methods to support design practice.

Testing the User Experience: Consumer Emotions and Brand Success

The key to creating brand loyalty is developing a consistent and salient brand perception through the association of specific emotional experiences with a product or service. A classic example of this is the emotion of wonder and happiness people associate with The Walt Disney Company’s films and theme parks. By crafting amazing experiences for the people who enjoy their products, Disney has created such a favorable association, leading consumers to feel they can trust the brand and know what kind of experience to expect from a visit to a park, hotel, or movie theater. People can appreciate their intense focus on the user experience, whether watching Mary Poppins, meeting characters like Goofy and Minnie Mouse for the first time as a child, shown in Figure 1, or watching Toy Story characters leap to life in the amazing and spellbinding zoetrope at the California Adventure theme park.

Using EN 15038:2006 as an Assessment Tool

If you have struggled with a good way to assess the countless translation agencies vying for your business or looked for a way to assess your current provider, you’re not alone. Companies around the globe have longed for a standard objective means by which to carry out their assessments. Some help has arrived in the form of European Standard EN 15038:2006, “Translation Services — Service Requirements.”

Ensuring Quality in Outsourcing

Business Process Outsourcing has become a leading business model of our time. While the increasing pressure to cut cost is still among the primary drivers for this trend, today quality has become a major issue when it comes to choosing an outsourcing partner. Here is an overview of standards and models that help measure and improve the quality of outsourcing services.

Comprehensibility as an Economic Factor

How can you guarantee a clearly understandable user manual? Is it even possible to measure the quality of technical documents or does comprehensibility merely depend on the reader? To answer these questions for the Porsche AG, content analysis provider semiotis³ developed a model to help measure the quality of documents.

Critique: Ape Scapula

I stumbled upon this poster while reading the blog Anna’s Bones. She described as being finished “just in the nick of time.” A few more hours, and a stronger editorial hand, probably would have been welcomed.

Towards an Architectural Document Analysis

Information architecture (IA) and document architecture (DA) provide two, partly overlapping, perspectives on the creation of document structures. This article suggests how the architecture of a document can be analysed from these two perspectives. Literature on IA and DA has been examined in order to identify central ideas that are of relevance for analysing the architectures of digital documents. The article contains an overview of how IA and DA have been used and defined. The article shows how a model for analysing documents as sociotechnical artefacts can fruitfully draw on parts of the theoretical and practical complexes of IA and DA. The aspects that are identified as particularly important from IA are organisation systems, navigation, and labelling. From DA, logical structures, layout structures, content structures, and file structures are all applicable aspects. It is discussed how these various aspects may be interpreted in order to support an analysis of the organising principles of documents.

Design Reviews and Posting Without Answers

In our design review sessions, a couple of members from our eight-person team share what they’re working on and ask questions about challenges they’re facing. We provide feedback and critique their project. If you’ve ever participated in a creative writing group, the design review works similarly. Team members use common sense and experience to guide their questions and reviews. Somewhat in contrast to a creative writing group, though, you don’t have to bring a finished piece to share.

Website Testing Tools

Here is a collection of some testing tools that we have compiled to aid your testing handily grouped into categories. Look out for our reviews of some of these tools coming soon.

BrowserShots

Generates screenshots of how websites appear at 800x600 and 1024x768 resolution in six commonly used web browsers.

Analysis of Team Design Review

Every other team meeting, three team members get 30 minutes each to talk about projects they are working on, and they get to demonstrate some of the cool things they are integrating into the project. As a team, we look at the project and both learn from what they’ve done, and make suggestions on how they might improve the project.

How To Respond Effectively To Design Criticism

Unfortunately, not many people enjoy criticism. In fact, many have developed a thick skin and take pride in their ability to brush it off and move on. However, despite its negative connotation, criticism often presents an excellent opportunity to grow as a designer. Before you can respond effectively, you need to understand what those opportunities are.

Recovering Delivery for Digital Rhetoric

This article develops a rhetorical theory of delivery for Internet-based communications. Delivery, one of the five key canons of classical rhetoric, is still an important topic for rhetorical analysis and production. However, delivery needs to be re-theorized for the digital age. In Part 1, the article notes the importance of delivery in traditional rhetoric and argues that delivery should be viewed as a form of rhetorical knowledge (techne). Part 2 presents a theoretical framework for “digital delivery” consisting of five key topics—Body/Identity, Distribution/Circulation, Access/Accessibility, Interaction, and Economics—and shows how each of these topics can function strategically and heuristically to guide digital writing.

How Do Experts Assess Usability Problems? An Empirical Analysis of Cognitive Shortcuts

Discusses the cognitive shortcuts that may hinder technical communicators in empathizing with readers. Explores the issue of judging the severity of problems detected in a document evaluation. Demonstrates how cognitive shortcuts may affect technical communicators' capability to assess the likelihood and impact of reader problems.

Goal-Based Scenarios

Discusses the potential of goal-based scenarios as an approach to designing online learning environments. Explores practical applications of goal-based scenarios for online training. Presents a procedural approach to designing a goal-based scenario.

Running an Efficient CMS Evaluation and Procurement Process: Hands-on Tips, Insider Knowledge and Advice

Why is getting the process right, so important? Value for money, project success, Return on investment.

Typographic Design Patterns and Best Practices

To find typographic design patterns that are common in modern Web design and to resolve some common typographic issues, we conducted extensive research on 50 popular websites on which typography matters more than usual (or at least should matter more than usual). We’ve chosen popular newspapers, magazines and blogs as well as various typography-related websites. We’ve carefully analyzed their typography and style sheets and searched for similarities and differences.

Testing Search for Relevancy and Precision

Despite the fact that site search often receives the most traffic, it’s also the place where the user experience designer bears the least influence. Few tools exist to appraise the quality of the search experience, much less strategize ways to improve it. When it comes to site search, user experience designers are often sidelined like the single person at an old flame’s wedding: Everything seems to be moving along without you, and if you slipped out halfway through, chances are no one would notice. But relevancy testing and precision testing offer hope. These are two tools you can use to analyze and improve the search user experience.

Beyond Goals: Site Search Analytics from the Bottom Up

While goal-driven analysis is wonderfully useful, we’ll explore a different, “bottom-up” approach that relies on pattern analysis and failure analysis to help you understand your users’ intent in qualitative ways that complement the top-down approach.

Why Is It That Teams Do A Poor Job of Post-Writing-Project Analysis?

Project teams may recognize that reviews are not working well, though the may not understand why. A valuable solution is to conduct ”lessons learned” analysis following the end of the project. Too often, though, post-writing-project analysis receives little commitment or meaningful effort, but why?

IAに起因するタスク失敗は相変わらず不利益

タスク成功率は、2004 年のユーザビリティ統計と比べると大きく上昇した。しかしそれにもかかわらず、ユーザがタスクを完遂できないケースがあり、その原因の大半は情報アーキテクチャ(IA)の出来の悪さにある。

Improving the Practice of Document Review

Document reviews should be used as a tool to build quality into research and technical reports. In most handbooks for professional writers, review is recommended for clear and simple reasons: it is intended to identify problems and suggest improvements that enable an organization to produce documents that accomplish its goals and meet readers’ needs.

Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale

The System Usability Scale (SUS) is an inexpensive, yet effective tool for assessing the usability of a product, including Web sites, cell phones, interactive voice response systems, TV applications, and more. It provides an easy-to-understand score from 0 (negative) to 100 (positive). While a 100-point scale is intuitive in many respects and allows for relative judgments, information describing how the numeric score translates into an absolute judgment of usability is not known. To help answer that question, a seven-point adjective-anchored Likert scale was added as an eleventh question to nearly 1,000 SUS surveys. Results show that the Likert scale scores correlate extremely well with the SUS scores (r=0.822). The addition of the adjective rating scale to the SUS may help practitioners interpret individual SUS scores and aid in explaining the results to non-human factors professionals.

The Effect of Culture on Usability: Comparing the Perceptions and Performance of Taiwanese and North American MP3 Player Users

A study of how 23 Taiwanese and North American subjects use a consumer electronic product shows that culture strongly affects the usability of the product. Survey data shows that North American users had much lower levels of user satisfaction and perceptions of effectiveness and efficiency than Taiwanese users. On the other hand, results on performance were unclear, indicating similar levels of effectiveness for both cultural groups and conflicting results on levels of efficiency.

Conversing About Performance: Discursive Resources for the Appraisal Interview

Despite its acknowledged importance, performance appraisal (PA) continues to be one of the most persistent problems in organizations, especially the appraisal interview (AI) component of PA, for which many techniques have been attempted with only mixed success. The authors conceptualize the AI as a “conversation about performance” and draw on an extensive review of the communication literature to identify the discursive resources available to the organization, the appraiser, and the appraisee for improving the preparation for and conduct of a conversation about performance. The authors' conceptualization extends research on PAs by identifying methodologies and conceptual underpinnings with connections to interpersonal, organizational, and mass communication scholarship.

Symbolic Capital and Academic Fields: An Alternative Discourse on Journal Rankings

During my 30 years in the academy, I have seen universities subject to increased demands for accountability. These demands from both internal and external publics translate into added attention to quality assessment. To evaluate teaching, universities measure student learning outcomes and rely on standardized scores as indicators of teaching effectiveness. To assess research productivity, departments document publications that appear in top-ranked journals and presses and track dollar amounts raised through external funding. This focus on evaluation, in turn, lends new credence to independent ranking systems that provide unbiased indices of quality. An unintended consequence of these academic norms, however, is the pattern of treating standards as objective indices rather than practical guidelines.

Journal Rankings and Academic Research: Two Discourses About the Quality of Faculty Work

Peer evaluation is the hallmark of the academic profession. Hiring, advancement, and reputation in the university setting have traditionally depended on a scholar's work as judged by his or her colleagues. The emerging trend toward journal ranking as an indicator of research accomplishment poses an important challenge to professional academic standards and to higher education generally because ranking schemes diminish the professoriate and degrade knowledge work. We argue that when scholarly journals are ranked in terms of their desirability as publication outlets they take on the characteristics of commodities.

Measuring Content Strategy: Not a Piece of Cake

Since there was no way to measure the effect of the new content in terms of conversions, it wasn’t really worth doing. And this, my friends, made me sad.

How to Evaluate Your Own Web Site

Is your web site in need of improvement, but you don't really know where to start? Have you changed your web site recently, and want to make sure that it's actually been improved? Now you can perform a professional evaluation of your web site yourself. Here's how.

Evaluating DITA-Enabled Content Management Systems

This presentation describes how authoring DITA topics and managing those topics in a content management system (CMS) will contain translation costs while improving overall information quality. This is not a recommendation for any particular product. It is a guide to how one group built their candidate list and computes return on investment.

Best Practices in Managing Knowledge: Benchmarking Knowledge Management Within and Between Organizations

Benchmarking comprises prioritisation of strategic improvement need (the why), measurement (the what) and practices (the how). Re-measure tracks performance improvement.

Evaluation of an XML-Based Content Management System in the Translation Process

Translation companies typically embrace innovations in methods for efficiently creating final formatted documents. About a year ago a client asked if we would be interested in testing and evaluating a content management system (CMS) and how it would relate to our translation process.

Comparatif des CMS

Un site web pour afficher les différences entre des centaines de Progiciels de Gestion de Contenu.

Short Guide To Evaluation Of Digital Work

This short guide gathers a collection of questions evaluators can ask about a project, a check list of what to look for in a project, and some ideas about how to find experts for evaluators who are assessing digital work for promotion and tenure.

Is Your Web Site Working?

A badly designed and implemented Web site can cost your company more than money. Aside from the obvious costs of having the site developed, which in some cases may be quite a lot of money, a badly designed Web site can give existing and potential customers a negative impression of the company. Therefore, it is essential that your Web site is actually fulfilling its objectives.

Information Architecture Task Failures Remain Costly

Task success is up substantially compared with usability statistics from 2004. Bad information architecture causes most of the remaining user failures.

Practical .NET Unit Testing

When the concept of unit testing is presented to a manager or a client managing a project, their reaction is often formed from a naïve understanding of the process. They assume that it has about the same ROI as traditional system testing. Unit tests are absolutely critical to writing complex, reliable software. Try to avoid comparing unit tests with traditional software testing. They are NOT the same thing, and they have dramatically different purposes.

Comparing Featured Article Groups and Revision Patterns Correlations in Wikipedia

Collaboratively written by thousands of people, Wikipedia produces entries which are consistent with criteria agreed by Wikipedians and of high quality. This article focuses on Wikipedia’s Featured Articles and shows that not every contribution can be considered as being of equal quality. Two groups of articles are analysed by focusing on the edits distribution and the main editors’ contribution. The research shows how these aspects of the revision patterns can change dependent upon the category to which the articles belong.

Pitfalls of Web Accessibility Evaluation Tools

Automated web accessibility evaluation tools are hard to trust, understand and only provides feedback on a small amount of factors that influence accessibility. Also, a unified web evaluation methodology should be adopted to provide consistent results across tools.

Deque Worldspace

Worldspace is an accessibility analysis tool designed to identify errors with Section 508, and the Web Content Accessibility Guidelines.

Best Practices for Speeding Up Your Web Site

The Yahoo Exceptional Performance team has identified a number of best practices for making web pages fast. The list includes 34 best practices divided into 7 categories.

Comparison of Home Page Loadability Scores for Major WCM and ECM Vendors

YSlow assigns letter grades (A thru F) for a page in each of 13 categories of best-practice. I decided to run YSlow against the home pages of 35 well-known web content management and/or enterprise content management vendors, then calculate a Grade Point Average. The scores are posted below.

Toward Content Quality

How do we know whether content is any good? This simple question does not have a simple answer. Yet, I think having a good answer would help us show our employers and clients why their content needs to improve and how their content compares to the competition’s.

Differentiating Your Design: A Visual Approach to Competitive Reviews

A common activity at the outset of many design projects is a competitive review. As a designer, when you encounter a design problem, it’s a natural instinct to try to understand what others are doing to solve the same or similar problems. However, like other design-related activities, if you start a competitive review without a clear purpose and strategy for the activity, doing the review may not be productive.

Measure Audience Engagement with Internet Video

A video's Engagement Curve is a visual representation of the audience's cumulative interactions with the video. An Engagement Curve quickly reveals which parts of the video clip the audience finds compelling -- in the example above, viewers are clearly rewinding to re-watch a segment in the middle -- and which parts do not hold the viewers' attenion -- in this case, the end. An Engagement Curve is read from left-to-right, with the left edge representing the beginning of the video and the right edge representing the end of the video. The dashed grey line shows the view-count, while the blue line shows many times that particular segment of the video was watched

Is Corruption an Issue?

You might think corruption is mainly an issue in places like sub-Saharan Africa or Myanmar, but unfortunately I’ve been exposed to several cases of this inside the online industry.

Strategic Numbers: Discussing the Value of Design With Sara Beckman

We’re excited to bring Sara Beckman from the faculty at the Haas School of Business back into the Adaptive Path fold. We first worked with her in 2003 on our groundbreaking report, Leveraging Business Value: How ROI Changes User Experience.

Reviewing User Interfaces

While user interface (UI) reviews often occur at the end of the development cycle, I recommend that you get involved early in the process, preferably when the designers create the initial wireframes or paper prototypes. Why? Making changes early in the process reduces development costs. Plus, if you identify usability issues early, it’s much more likely the team can remedy them before launch, preventing bad reviews.

Google Insights: A Social Barometer for the Digital Age

Grant Whiteside looks at Google Insights for Search tool and how it can give you a pictorial breakdown of the social barometer.

Test Driving Your Next Employee's Skills

For the past few years, the buzz phrase in interviewing has been behavioral interviews. In behavioral interviews, the interviewer asks the candidate what has been done in the past in order to extrapolate what will be done into the future: past performance indicates future performance. I’m suggesting that the behavioral interview could be more than a discussion about behavior—it could be a demonstration of behavior. Test driving candidates places a demand on the candidate to exercise his or her current ability while under scrutiny. Thus, rather than hearing stories about behavior, test drives allow you to observe behavior.

The User Experience of Enterprise Software Matters

I can’t tell you how many frustratingly unusable enterprise Web applications I’ve encountered during my 12 plus years in corporate America. As important as the user experience of enterprise software is to a business’s success, why isn’t its assessment usually a factor in technology selection?

Communicating Customer and Business Value with a Value Matrix

If you’re like me, you’ve always felt something was missing once you finished creating your personas and scenarios. They communicate the heart and goals of the user, but miss out on a lot of details. And while it’s the intent of both documents to do just that, neither personas nor scenarios succinctly communicates to your business what features a product or service should have and why it should have them.

Relocating the Value of Work: Technical Communication in a Post-Industrial Age

This article analyzes the location of “value” in technical communication contexts, arguing that current models of technical communication embrace an outdated, self-deprecating, industrial approach subordinating information to concrete technological products. By rethinking technical communication in terms of Reich's “symbolic-analytic work”, technical communicators and educators can move into a post-industrial model of work that prioritizes information and communication, with benefits to both technical communicators and users.

A Technology Transfer Model for Program Assessment in Technical Communication

In this article we seek to reframe accountability by means of an emphasis not on auditing but on student performance, not on the development of databases but on the creation of reflective practice. We attempt to demonstrate one model of program assessment that focuses on student performance as the center of a reflective assessment framework that can act as a technology transfer model for the diffusion of program assessment knowledge.

Methods and Results of an Accreditation-Driven Writing Assessment in a Business College

This article describes a pilot effort for an accreditation-driven writing assessment in a business college, detailing the pilot's logistics and methods. Supported by rubric software and a philosophy of "real readers, real documents," the assessment was piloted in summer 2006 with five evaluators who were English instructors and four who worked or taught in business environments. The nine evaluators were each given 10 reports that were drawn from a sample of 50 reports completed in a writing-intensive course. They created 88 individual assessments using a 10-category rubric. While the overarching purpose of the pilot was to determine the effectiveness of the methods used, the results may also be of interest to those involved with the assessment of writing.

Communicating Customer and Business Value with a Value Matrix

What happens to the personas and scenarios once you’re ready to start requirements definition and design. Are you sure you’ve adequately communicated the type of system your users need to the Business Analyst and Interaction Designer on your team?

Corporate Blogs: Measure Their Value!

To date, ROI hasn't been applied to blogs. This is partly due to blogging recent introduction to the marketing mix. Many blogging experts have suggested calculating a blog ROI is impossible. As a professor, I teach students how to tie marketing to the bottom line. Calculating ROI for a blog should be no harder than calculating it for other marketing components. To place ROI measurements in context, you must first understand how blogs fulfill different business objectives.

Laatua Verkkoon: Quality Criteria

The quality criteria are intended to act as a tool for developing and assessing public web services. The purpose of the quality criteria is: to act as a tool for developing and assessing public web services; to improve the quality of public web services for both users and producers; to increase the benefits from public web services.

Web Application Solutions: A Designer’s Guide

Web Application Solutions is a guide that helps designers, product managers, and business owners evaluate some of the most popular Web application presentation layer solutions available today. We compare each solution through consistent criteria (deployment & reach, user interactions, processing, interface components & customization, back-end integration, future proofing, staffing & cost, unique features) and provide an overview, set of examples, and references for each.

Intranet Communication vs. Traditional Communication

A way to measure return on investment (ROI) for your intranet is to answer two basic questions. How does the intranet increase the level and quality of communication? How does it replace traditional forms of communication? To develop such an ROI model, you need to be clear on the current level and type of communication within your organization.

Intranet Review Toolkit

The Intranet Review Toolkit provides a comprehensive set of guidelines for assessing the strengths and weaknesses of corporate intranets. It contains a substantial set of heuristics, allowing a detailed intranet review to be conducted that focuses on a wide range of functionality, design and strategy.

Quantitatively Test the Effectiveness of Your Home Page

Staff should be able to confidently, quickly and accurately step from the home page of the intranet towards the information they require. If staff can’t achieve this without resorting to search, the home page needs to be redesigned. This article explains a quick and effective technique for assessing whether your home page is an effective gateway to site content.

If You Can't Measure It, You Can't Manage It

Intranets don't self-organize. Without planned, centralized information architectures and clearly defined published processes, they become unproductive. Intranets often have applications that either don't work properly, are too difficult to learn, or have no clear business benefit. Applications, like content, must be able to establish a clear return on investment.

Using JAWS to Evaluate Web Accessibility

This article is designed to help users who are new to JAWS learn the basic controls for testing web content, and to serve as a reference for the occasional JAWS user.

Testování Přístupnosti Webových Stránek se Screenreaderem JAWS

Tento článek je českou verzí článku Using JAWS to Evaluate Web Accessibility. V textu jsou zmiňovány prvky stránky, které jsou součástí struktury webu WebAIM.org a nemusí se vyskytovat na stránce s touto verzí.

Usando o Jaws Para Testar Acessibilidade

Este artigo destina-se a ensinar aos usuários não familiarizados com o JAWS os procedimentos básicos necessários a avaliar a acessibilidade do conteúdo web e servir como uma espécie de guia de referência para o usuário ocasional deste programa.

Determining Readability: Readability and its Implications for Web Content Accessibility

One area of accessibility often overlooked is the readability of the content of your web pages. Not every user may be familiar with terms or terminology being used. Others may not have the same socio-political background, literacy skills or capacity to fully comprehend what it is you are saying. One goal of the content author then is to try and identify their target audience, and then ensures that they are not "writing over their heads".

Toys 'R' Rushed: A Cautionary Tale

Website critic Lou Rosenfeld is shopping for a baby present, but the website he's using is making his task tougher than it should be. Lou takes on www.toysrus.com.

Assessing Assessments: The Inequality of Electronic Testing

Computer and Internet based tests are used for a variety of purposes. From entering education or employment, to improving basic learning, people everywhere are taking electronically formatted tests. With the advancement of testing from traditional paper-based tests to technologically advanced electronic tests, people reap the benefits of easier access to tests, faster response times, and greater reliability and validity of tests. However, persons with disabilities are being left out of the picture and out of many typically-administered tests.

Why Designers Fail: The Report

Last week I announced a study on why designers fail - exploring the reasons why designers, and people who work with designers, believe designers don’t achieve the results they desire. I presented the results as UIE 13 last week, and as promised here is a summary. Prize winners will be announced soon. Many top reasons for failure are not typically considered design issues, such as collaboration skills, persuasion skills, and receiving critical feedback.

Web Design by Designers

Designers are, as a rule, a fussy bunch, and when it comes to their own business communications they’re even more so. Designing a website for an award-winning design firm verges on the impossible. A design firm’s web presence primarily serves as a tool to attract new business from a global community—and, secondarily, as a means to show off. Designers are by far their own worst critics, and their websites have to tread a fine line between being cutting-edge so as to attract young new business, and more traditional so as to appeal to established or more conservative businesses.

Measuring Website Performance: Part 1

Establishing a set of reliable metrics for measuring the performance of your web site in the real world is a key success factor. In the next few articles, we will explore what can be measured, how to do it, and how to turn that data into some useful intelligence for your business.

Measuring Website Performance: Part 2

In this column we will continue with our examination of website metrics. Last column introduced the idea of performance metrics and the basics of what to measure. In this column I would like to go a little further into discussing the implications of those metrics.

Measuring Website Performance: Part 3

Your web server archives the information needed to generate these numbers and many others. The raw data is stored on the server in what is known as a log file. The statistics referenced above are best accumulated through the use of a log analysis program to convert your hard-to-read server log files into an understandable format.

How to Manage Out of Date Content

Organizations are in urgent need of professional review processes for their intranets and public websites. Out of date content is growing year by year, and there are many horror stories about out-of-date content waiting to happen. It’s time for management to get serious and professionally manage their websites.

The Sixty-Minute Guide to Evaluating Comparative Test Data

Mary wants to compare the average registration times between the two versions to see if the second version is faster than the first. The method typically used when comparing averages is called a t-test of independent means.

What Does a Good Web Page Need?

Starts to examine different pages, thinking about what items should appear on them, and considering issues such as consistency, usability, and accessibility.

Acid Redux

I fully acknowledge that a whole lot of very clever thinking went into the construction of Acid3 (as was true of Acid2), and that a lot of very smart people have worked very hard to pass it. Congratulations all around, really. I just can’t help feeling like some broader and more important point has been missed.

What You Need to Know If You Want a Job in Web Development

The true nature of Web development is a complicated metric to gage without being in the field for many years, and without reading about and listening to thousands who are in the field with you. It can be a frustrating experience for any one person graduating college and starting their career, or wanting to transition into a separate discipline. I decided to try and change that with some personal reflection.

Technical Communication Outsourcing: The Twelve Driver Framework Tutorial

Almost all IT, engineering research, financial analysis, and manufacturing industries are confronted with a question: to outsource or not? The outsourcing and offshoring trend is inspired by success stories of huge cost savings, decreased time-to-market, and better quality. Simultaneously, outsourcing-gone-bad stories highlight how hidden costs exceed benefits, cross-cultural problems impact quality, and intellectual property risks shadow project lifecycles. Managers in companies are presented with a confusing picture for which there are no easy answers. Companies, vendors, and policymakers need a framework to understand the outsourcing phenomenon and plan implementation strategies for outsourced projects. At present, many companies go with the gut based on the experience of others and media reports. But very rarely are two technical documentation tasks alike and never are the concerns of two technical communication tasks the same. This tutorial presents the twelve driver framework and the driver-model percentage matrix to assess the benefits and risks of outsourcing a technical communication task. In the end, qualitative decision-making will determine an organization's decision about outsourcing, but the use of such a framework and related metrics will greatly enhance the quality of the final choice.

On the History of Evaluation in Information Retrieval

This paper is a personal take on the history of evaluation experiments in information retrieval. It describes some of the early experiments that were formative in our understanding, and goes on to discuss the current dominance of TREC (the Text REtrieval Conference) and to assess its impact.

Bibliometrics to Webometrics

Bibliometrics has changed out of all recognition since 1958; becoming established as a field, being taught widely in library and information science schools, and being at the core of a number of science evaluation research groups around the world. This was all made possible by the work of Eugene Garfield and his Science Citation Index. This article reviews the distance that bibliometrics has travelled since 1958 by comparing early bibliometrics with current practice, and by giving an overview of a range of recent developments, such as patent analysis, national research evaluation exercises, visualization techniques, new applications, online citation indexes, and the creation of digital libraries. Webometrics, a modern, fast-growing offshoot of bibliometrics, is reviewed in detail. Finally, future prospects are discussed with regard to both bibliometrics and webometrics.

Design and Development of a Concept-Based Multi-Document Summarization System for Research Abstracts

This paper describes a new concept-based multi-document summarization system that employs discourse parsing, information extraction and information integration. Dissertation abstracts in the field of sociology were selected as sample documents for this study. The summarization process includes four major steps — (1) parsing dissertation abstracts into five standard sections; (2) extracting research concepts (often operationalized as research variables) and their relationships, the research methods used and the contextual relations from specific sections of the text; (3) integrating similar concepts and relationships across different abstracts; and (4) combining and organizing the different kinds of information using a variable-based framework, and presenting them in an interactive web-based interface. The accuracy of each summarization step was evaluated by comparing the system-generated output against human coding. The user evaluation carried out in the study indicated that the majority of subjects (70%) preferred the concept-based summaries generated using the system to the sentence-based summaries generated using traditional sentence extraction techniques.

Constructing Trust Between Teacher and Students Through Feedback and Revision Cycles in an EFL Writing Classroom

The authors' goal was to model the role played by the relationship between a writing teacher and her students in the feedback and revision cycle they experienced in an English-as-a-foreign-language context. Participants included a nonnative teacher of English and 14 students enrolled in her English writing class in a Korean university. Data came from formal, informal, and text-based interviews; semester-long classroom observations; and students' drafts with teacher comments. Findings showed that caring was enacted in complex and reciprocal ways, influenced by interwoven factors from the greater society, the course, the teacher, and the student. Students' level of trust in the teacher's English ability, teaching practices, and written feedback, as much as the teacher's trust in particular students based on how they revised their drafts, played a great role in the development of a caring relationship between them.

Proving Worth: What Technical Communication Managers Must Do to Prove the Value of Their Deliverables

If the documentation is not being used and used effectively, it will never help the bottom line. The trick to increasing value with internal and external users is to identify areas where documentation can save time and money, to create agreement that the documentation can save time and money, and to ensure that the documentation does save time and money.

Giving and Receiving Feedback

Feedback is important to business, but no one needed to tell us that. As Technical Communications managers, we regularly see feedback in many forms: user feedback, customer feedback, internal feedback, external feedback, feedback from testing, and feedback in performance appraisals. As beacons of information communication in our organizations, we are responsible to communicate well and, by extension, possess a solid appreciation of and ability to respond to feedback.

It's In the Numbers: Using Metrics to Plan Documentation Projects

It's in the numbers. Creating documentation is not an exact science, yet as communication leaders, we are expected to provide real estimates for how much time we need to document a project, or what we can produce given a predetermined timeline.

Testing Documentation

As part of the product, testing documentation seems like an obvious thing to do, but what does it really mean? I’ve fielded the question in a few different places now and it’s always interesting to delve deeper and understand the rationale behind the request.

Time Well Spent: The Magazine Publishing Industry's Online Niche

This article compares the uses of the print and online versions of the same magazine by its readership. Combining surveys of the readership and commercial data from the publisher and web designer, the study examines how one magazine has developed an online publication for its readers. Group Leisure is a niche magazine which has been in print for over a decade and online for two years. This article analyses the usage of the magazine in terms of age, gender and modal occupation of its readers and examines how their understanding of spending and saving time on the magazine underpins their perceptions of its value. The results and conclusions of this research have relevance to the publishing industry and to the study of online journalism.

How C.R.A.P is Your Site Design?

Eons ago when I was taking the Freshman web design course in college (okay, it was only 4 years ago) I was taught about the acronym of all acronyms, the one by which all other web design acronyms were judged. We learned that good design is based on the C.R.A.P. principles where C.R.A.P. stands for Contrast Repetition Alignment Proximity, and when Creative Directors tell you that your design is crap, they’re actually giving you positive reinforcement. Okay, that last part was made up, sorry. “Crappy work” is probably not a term of endearment but rather an indication that your pixels smell.

Using Calculators for User Engagement

Calculators can play important roles on websites. They are especially popular for financial sites, where they can help users calculate mortgage payments, retirement needs, interest earned, and so on. They also appear on other sites, where users can calculate things as varied as their BMI (body mass index), carbon footprint, life expectancy, or gas mileage.

Conducting Effective Team Technical Reviews

Mention team technical reviews to a group of tech writers and chances are good that you will either get a loud, collective groan, or the group will vie to tell the best review horror story. On the one hand, technical reviews are a vital part of our jobs because they help us to produce high quality product documents. On the other hand, technical reviews gone wrong are the bane of our existence. The good news is that we have the power to conduct consistently effective technical reviews. This article summarizes why we do reviews and what often goes wrong in reviews, and then summarizes steps to take before, during, and after technical reviews that can help you conduct effective team technical reviews. Although your process and team may differ from what's described here, you can apply the information in part or in whole to improve your current review process.

Eight Issues to Consider When Developing Metrics for Your Technical Communication Group

Wondering how you can assess the effectiveness and productivity of your work? Admittedly, it’s not easy and there are no simple approaches. But it can be done. As you develop a program, consider these issues, which arose from a review of literature on the metrics used to assess the productivity and effectiveness of software engineering, training, marketing communications, and technical communication.

Evaluating Online Help

Online help excels in providing quick access to concise information - but only when the users choose to access it. Delivering high-quality online help that satisfies all users is a hard task. Several good help authoring tools make help generation and maintenance easier, but to create good content that is highly effective is still a huge challenge. Experience shows that even after following quality guidelines or best practices, the final output may still not be good enough to satisfy the needs of your users. Heuristic evaluation of an online help system provides an initial assessment of both quality and usability. This article presents a summary of key points for evaluating online help, though you will likely want to expand the heuristics with company or product-centric metrics suitable to your application.

Comparing Powerpoint Experts' and University Students' Opinions About PowerPoint Presentations

Technical communication instructors want to help students, as well as professionals, design effective PowerPoint presentations. Toward this end, I compare the advice of academic and industry experts about effective PowerPoint presentation design to survey responses from university students about slide text, visual elements, animations, and other issues related to PowerPoint presentation design and delivery. Based on this comparison, I suggest some topics, such as PowerPoint's Slide Sorter view, that technical communication instructors and other presentation instructors might address when they cover presentations in their classes or seminars.

Contextualize Technical Writing Assessment to Better Prepare Students for Workplace Writing: Student-Centered Assessment Instruments

To teach students how to write for the workplace and other professional contexts, technical writing teachers often assign writing tasks that reflect real-life communication contexts, a teaching approach that is grounded in the field's contextualized understanding of genre. This article argues to fully embrace contextualized literacy and better teach workplace writing, technical writing teachers also need to contextualize how they assess student writing. To this end, this article examines some of workplaces' best assessment practices and critically integrates them into an introductory technical writing classroom through a method called student-centered assessment instruments. This method engages students, as workplaces engage employees, in the assessment process to identify local requirements for writing tasks. Aligned with theory and practice, this method is not only an effective classroom assessment method, but becomes an integrated part of students' genre-learning process within and beyond the classroom.

Seven Tips for Living with Technology

After living through more than a few technology acquisitions, variously as perpetrator, victim, and bystander, I’ve come across a few tips that can make the process a little easier.

How To Justify Conference Attendance

As a manager you may often be faced either with your own need to attend a professional conference or requests by your team members to attend one. Professional conferences can be expensive and not all budget managers understand their importance or the benefits derived by conference attendees. In this article, Mike Doyle discusses how to go about justifying the expense of attending a conference and provides some handy worksheets you can use to do so.

It's In the Numbers: Using Metrics to Plan Documentation Projects

Understanding the Need for Content Quality Management

An interview with Scott Abel, The Content Wrangler that's both a bit controversial and inspiring that looks at how good content quality management can be a great benefit for tech pubs departments.

What Do We Gain by Assessment?

The question, what do we gain by assessment, is one that has been asked more and more often by engineering educators. They ask the question even as the changes in accreditation brought on by ABET, Inc. and the Engineering Criteria have been cemented in programs both in the United States and abroad.

Practitioner's Perspective

The biggest challenge for auditors is to make sure that they're measuring the right things. All too often communicators measure only their outputs—the messages and channels they're producing—without connecting them to the outcomes of using these outputs.

Readability

I know from some years running a reading clinic in the United States that you can make more progress if you start the students out on relatively easy reading material so the the students can read the materials with some comprehension and success. But how do you tell these African instructors how to select “relatively easy” reading materials in technical English? The answer - use a readability formula.

Connecting Surveys to the Bottom Line

Most communication surveys pose questions about how well messages have been understood and how effective different communication channels are. What surveys usually lack are questions that link the communications you manage to the effect they have on employee behaviors, which result in improvements in the bottom line. Here are two examples of communicators who used surveys to analyze behavior and build a business case for their budgets.

Evaluating and Managing Surveys

While surveys aren't the only research tool available to HR managers, they are the most useful one when 'hard' numbers are needed on how many people see things a certain way and when it's important to track differences among subgroups or improvement over time.

Focus Research on Your Most Valuable "Capital"

The entire concept of human "capital" seems to have arisen during the last several years of booming economy and scarce availability of skilled employees. When any resource is scarce, it's valuable. Now with the highest rates of layoffs being announced in the U.S. since 1991, let's hope the mindset of management is not on the order of getting the most out of the human "liabilities" they're forced to retain.

Accountability and Return-On-Investment

Once viewed more as art than science, marketers are increasingly interested in measuring performance. Like it or not, there is a new wave of accountability in the world of marketing, and if you're not prepared, you could get swept under it. Companies are becoming increasingly concerned with ensuring that all activities are profitable. As a result, each dollar invested in marketing is being challenged to demonstrate bottom line performance. New forms of marketing, escalating ad costs and tools that purport to measure marketing effectiveness have all contributed to the pressure traditional media is facing to "prove its worth."

Balanced Scorecard Step-by-Step: Maximizing Performance and Maintaining Results

Paul Niven's book is invaluable for communicators whose companies are implementing a Balanced Scorecard, and it can also provide a great deal of useful information on setting measurable goals for a staff function like communication to ensure it aligns with a company's strategy. The book provides easy-to-understand summaries of how various business processes work for communicators who want to better understand their businesses.

A Checklist for Public Relations Practitioners When Considering Measurement or Evaluation Research

Read these 10 useful tips from Walter K. Lindenmann, Ph.D., an independent consultant specializing in public relations research, measurement and evaluation services.

Five Tips for Measuring Public Relations and Catching the CEO's Attention

Measuring the effectiveness of PR is critical to moving PR from a tactical function to a strategic component of your company's plan for success. But the old ways of counting clips just aren't good enough to convince today's management executives that their investment in PR and overall communications is paying off. Here are 5 Tips about how to measure PR in ways that will catch the CEO's attention and increase the stature of PR in any organization:

Forget ROI, Let's Show How We're Making Money

Throw a stone in a room full of communication professionals and there's a good chance you'll hit one that will back up this statement: senior management loves to see ROI measurements, but seeing how communication initiatives create sales trumps all other measurements. From a marketing communication perspective, simply receiving feedback from a sales team can help your team answer most senior-level frustrations. From the perspective of a sales force, understanding marketing efforts (and how those efforts actually work) aids in everyone's ultimate objective: securing sales.

Fun With Measurement Math!

Recent research tells us that 97 percent of all public relations departments are involved in media relations, and 88 percent evaluate their campaigns using media analysis. On one hand, industry leaders urge us to measure the results of our work via business outcomes; yet on the other, communicators are still asked to supply output results as 'proof of performance.' Is there some link between the two that can cover both? Here are some relatively easy ways to make your media results speak with numbers that management will respect and understand.

Hard Measures are Key to Gauging the Effectiveness of Communication on the Bottom Line

In conducting its landmark 2003 Communication ROI Study, which focuses on the relationship between an organization's internal communication strategy and practices and its shareholder returns, Watson Wyatt made some surprising findings regarding the relationship between effective external and internal communication.

How to Set Specific Goals for Your Public Relations Campaign and Calculate Your Precise ROI

In today's world, advertising is just one element of the marketing formula. In fact, there is a shift occurring away from advertising to other marketing that's less costly and more cost-effective and efficient. In all your marketing efforts, your goal is to gain that top-of-mind awareness position with your prospect/customer. PR can do this. Advertising can do this once awareness is attained. PR gets you there; advertising keeps you there. Knowing the return of PR truly justifies it as an integral part of the marketing arsenal.

Measuring Integrated Marketing Communication from Start to Finish

Many companies have taken a limited view of the impact that marketing communication can have on overall corporate objectives, reducing their understanding of the value of marketing communication. One reason for this resistance is that the value of IMC can be complex to measure in a world where marketing usually moves at a dynamic pace and is driven by a changing competitive landscape and seemingly unpredictable shifts in customer attitudes. The potential revenue and customer satisfaction benefits of implementing an IMC program can be so dramatic that companies shouldn't ignore the movement any longer.

Measuring Search Engine Marketing ROI

Spending on search engine marketing (SEM) is rising dramatically, yet surprisingly few companies are measuring the effectiveness of their campaigns. In a short survey conducted by web analytics vendor NetIQ, more than 800 participants responded to questions about their search engine marketing efforts and their attempts to measure success. The survey responses provide interesting insights into the state of search engine marketing ROI.

Build More Effective International Media Analysis Programmes with Market Research Disciplines

Ask communication professionals why measurement is important, and their answers are likely to involve accountability, measures of effectiveness, ROI and planning support. Ask market researchers what makes for good measurement, and they are likely to respond that it involves reproducible results, adherence to rigorous standards and objective impartiality. Within the communication process, however—especially within PR and media relations—there is a tendency to look more closely at the output of their programmes than at the methodology yielding the data charts and reports. While market research has a well-established body of theory to support its claims of delivering objective and authoritative data, media analysis as a commercial discipline is only just beginning to grasp the importance of these standards.

Quantifying the Impact of Communication on Your Organization’s Bottom Line

Consider this scenario: You’re making a pitch to the CEO that calls for more focus and resources on internal communication, citing recent studies that demonstrate that organizations with more effective communication have higher performance. And you get this response: “Great, I’m sold on the importance of communication. Tell me which communication channels have the greatest impact on our bottom line, and put together a strategic plan that will lead to increased revenues.” Did you get a little more than you bargained for? While the CEO’s request certainly sounds challenging, rest assured that it can be done.

Response Mechanisms—The Key to ROI

ROI still eludes many B2B communicators, despite the increasing pressure to prove it. What is the amount of revenue your company gains as a result of your communication after you’ve subtracted expenses? This is especially good to know if you integrate your marketing communication. What part of the mix is working, and what isn’t? If you know that, you can eliminate the duds and rev up the elements that really bring in revenue. Ultimately, over time, you can increase the return on your marketing investment by knowing how well the components of your program perform.

Alternative Ways to Measure the Effectiveness of Your Publications

If you want to go beyond the usual limits of a traditional readership survey that tells you how well received a publication is, first clarify your objectives. Then you might include additional "impact" questions on your next survey, conduct in-depth focus groups with readers, and conduct some objective, "audience-free" measurements of the publication to see how well those objectives were met.

Are You Spending the "Right" Amount?

To back up a request for more budgetor defend the existing one, you need to know exactly what you’re spending--and what you’re getting in return. But how can you tell if you’re spending too much on communication? This article suggests five approaches to weighing up the cost versus value of your communication activities.

Benchmarking: Ugly Truths and Unpredictable Outcomes

A walk through a benchmarking project, sharing some of the behind-the-scenes stories of benchmarking gone right, and gone wrong. So, here they are, complete with tales of terror, moments of madness and even some back-room horse-trading.

Buttress Your Benchmarking Efforts

A smart company understands that a favorable reputation improves its bottom line. From a PR perspective, a strong reputation acts as good will, giving you the benefit of the doubt with both journalists and the general public. To find out how strong your company's reputation is, it is helpful to compare it with the reputations of other companies, also known as benchmarking.

Choosing the Right Metrics to Benchmark

The assumption that financial analysts make is that low numbers on efficiency (communicators per employee, for example) would be better than high numbers. Unfortunately, that doesn't take into account that low-cost communication may have low impact on the bottom line. If your organization wants to track communication efficiency metrics, then I'd suggest tracking effectiveness measures as well.

Defining Benchmark Questions for Great Results

Part of the challenge of determining the questions to ask during benchmarking is to match the questions to the purpose of the study and the outcomes you are trying to achieve. Below is a breakdown of some of the issues regarding benchmarking questions that need to be addressed before beginning a benchmarking exercise.

The Myths and Methods of Reputation Measurement

If you are concerned about your reputation and want to measure its health, here's what to do. Get the communication people in your organization together in a room and get consensus on what you want to measure and which constituencies are your top priorities. Determine how a good relationship with each of those constituencies benefits your organization. Your success is measured by achieving those benefits. Figure out what you will be measuring and what benchmarks you will be measuring against. Undertake the appropriate research and voila, you'll have the answers you need.