Assessing the Value Added by In-House Technical Communication Courses  

A specially designed instrument that measures the effectiveness of written communication courses taught in-house has been pilot-tested with employees of a major power utility. The instrument showed that, one month afrr attending the course, participants’ written communication skills increased by twenty percent. A second measurement, recordedfour months aJer the course, showed there had been only a marginal drop in skills compared to the level recorded three months earlier.

Blicq, Ronald S. STC Proceedings (1997). Articles>Writing>Assessment

Comparative Assessment of Document Usability With Writing Quality Measures  

Measures of writing quality were developed on the basis of research findings on reading, writing, and cognition. From among the over twenty measures developed by the quality project, this paper illustrates theoretical and methodological issues for two kinds of measures: agents of action in sentences and task-oriented headings. When applied to a sample set of documents, these measures showed the writing to be inconsistent in style among the documents and only partly in conformance with suggestions derived from research. Though technical communication writing guidelines may be well thought out and grounded in years of practice, to have credibility in the new quality environment, writing guidelines need to be supported by testing. This paper discusses the development and testing of document quality measures that can be used as the basis of writing guidelines. The measures were tested both by using them to score technical documents, which will be discussed here, and laboratory testing with document users, which will not be discussed in this paper.

Krull, Robert. STC Proceedings (1994). Articles>Writing>Assessment

Contextualize Technical Writing Assessment to Better Prepare Students for Workplace Writing: Student-Centered Assessment Instruments    

To teach students how to write for the workplace and other professional contexts, technical writing teachers often assign writing tasks that reflect real-life communication contexts, a teaching approach that is grounded in the field's contextualized understanding of genre. This article argues to fully embrace contextualized literacy and better teach workplace writing, technical writing teachers also need to contextualize how they assess student writing. To this end, this article examines some of workplaces' best assessment practices and critically integrates them into an introductory technical writing classroom through a method called student-centered assessment instruments. This method engages students, as workplaces engage employees, in the assessment process to identify local requirements for writing tasks. Aligned with theory and practice, this method is not only an effective classroom assessment method, but becomes an integrated part of students' genre-learning process within and beyond the classroom.

Yu, Han. Journal of Technical Writing and Communication (2008). Articles>Education>Technical Writing>Assessment

Design, Results, and Analysis Assessment Components Nine-Course Program    

The case for assessment of college writing programs no longer needs to be made. Although none of us would have chosen the words, we all have come to accept the truth of Roger Debreceny’s words: the 'free ride' for America’s colleges and universities is indeed over (1). All writing programs face difficulties in selecting the means for the most effective evaluations for their individual programs. Key concerns include how appropriately, practically, and cost effectively various assessment tools address this problem.

Carson, J. Stanton, Patricia G. Wojahn, John R. Hayes and Thomas A. Marshall. LLAD (2003). Articles>Education>Writing>Assessment

Dynamics of Iterative Reader Feedback: An Analysis of Two Successive Plus-Minus Evaluation Studies    

A brochure that had been revised on the basis of feedback from readers using the plus-minus evaluation method was evaluated again using the same method. This article compares the results of these two successive evaluation studies to examine the dynamics of evaluating and revising using a troubleshooting method based on verbal self-reports. The findings show that the plus-minus method does not necessarily lead to a decrease in the number of problems readers find in a revised document. But the types of problems readers find are significantly different. For example, after the brochure was revised, it had fewer clarity and structural problems, and readers could focus more on credibility issues.

de Jong, Menno D.T. and Rijnks, D. Journal of Business and Technical Communication (2006). Articles>Writing>Assessment

Empirical Evaluation of Concept Mapping: A Job Performance Aid for Writers    

The usefulness of concept mapping as a job performance aid for writers of technical documents was examined. Thirty-four writers were randomly assigned to one of two groups. The experimental group received 2 hours of training in the use of concept mapping. Both groups revised the same chapter from a computer manual, and an experienced technical editor blindly evaluated each revision. In part two of the study, revised texts were given to two groups of users. One group received a concept-mapped revision, while the other group received a text revised by a writer who had used conventional revision techniques. Readers' comprehension was tested and compared. Revision time was not significantly different between groups, and the editor's ratings of quality were not different. However, readers' comprehension was significantly higher with the concept-mapped versions. These results suggest that concept mapping is a useful revision tool for writers.

Crandell, Thomas L., Naomi A. Kleid and Candace Soderston. Technical Communication Online (1996). Articles>Information Design>Writing>Assessment

Evaluating Training Workshops in a Writing Across the Curriculum Program: Method and Analysis    

Program directors could use data from protocols and interviews to identify 'natural sources of resistance', and 'translation and follow-up problems'.

Blakeslee, Ann M., John R. Hayes and Richard Young. LLAD (2002). Articles>Education>Writing>Assessment

Expert Judgments Versus Reader Feedback: A Comparison of Text Evaluation Techniques    

Are technical writers able to predict the results of a reader-focused text evaluation? In this article we report a study with fifteen technical writers, who were asked to point out the reader problems in a public information brochure. The brochure was also evaluated with thirty readers from the target audience (using a combination of the plus-minus method, a questionnaire, and user protocols). The results of both kinds of text evaluation show little overlap. The technical writers only predicted a small proportion of the reader feedback, and produced a lot of new problem detections. In addition, there was little agreement among the technical writers with regard to their problem detections.

de Jong, Menno D.T. and Leo R. Lentz. Journal of Technical Writing and Communication (1996). Articles>Writing>Assessment>Methods

Factors Affecting Readability

No one has found a way to really help writers create readable prose. Robert Gunning developed a method for calculating the 'Fog Index' and Rudolph Flesch worked out more than one formula for measuring the simplicity of writing. By one of Flesch's formulas (the one without personal pronouns), Ronald S. Lemos in the February, 1985 issue of Communications of the ACM (CACM) was able to prove that CACM required two less years of school to read than Datamation. Statistics can prove anything. I have no idea what Sophomore in High School could read the CACM cover to cover and even understand most of it. Flesch's book 'The Art of Plain Talk' was given to me at a Yourdon Systems Analysis course. The Instructor handed it to each of us, saying something like 'read this and you'll be a manager in no time' (supposedly, management is handed to the least efficient person who can also write well). The book is full of examples, mostly journalistic, showing how good writers evoke human interest. Of course, these writers had human events, thoughts and feelings as their focal points, not software, I doubt whether any of the graduates of that week ever used Flesch as a reference for grading their own documentation. How would Bernard Shaw have documented software? Or Mingus played it? This paper addresses these burning issues.

Hallgren, Chris. SIGDOC Proceedings (1986). Articles>Writing>Assessment>Formulas

From Writing Documents to Meeting User Information Needs  

As professional Technical Communicators, we constantly struggle with the question 'Is what we produce being used and is it making a difference?' Too often, we base our measures of success on our own views of what makes a good document, Help system, or other information product. Through broader and more detailed analysis, and by measuring success based on how our information products affect others, we can increase the effectiveness of our information products and increase our value to our users and companies.

Hurst, Bob, Cindy Line and Kerry Newberry. STC Proceedings (2003). Articles>Writing>Usability>Assessment

How In-Process Measures Can Help You Manage Quality  

Some technical communicators see productivity and quality measurements as threatening when these measurements are used as an evaluation of the person, not the process or the product. Communicators can also be frustrated by the time and effort it takes to collect quality measurements with no visible result or improvement of their own work. This paper discusses how managers of technical communicators can develop and implement a system of in-process measurements to help technical communicators manage the process and improve their own final documentation before it is shipped to the customer, without being threatened by the measurement system.

Fisher, Lori H. STC Proceedings (1997). Articles>Writing>Assessment>Technical Writing

The Impact of Student Learning Outcomes Assessment on Technical and Professional Communication Programs    

Because of accreditation, budget, and accountability pressures at the institutional and program levels, technical and professional communication faculty are more than ever involved in assessment-based activities. Using assessment to identify a program's strengths and weaknesses allows faculty to work toward continuous improvement based on their articulation of learning and behavioral goals and outcomes for their graduates. This article describes the processes of program assessment based on pedagogical goals, pointing out options and opportunities that will lead to a meaningful and manageable experience for technical communication faculty, and concludes with a view of how the larger academic body of technical communication programs can benefit from such work. As ATTW members take a careful look at the state of the profession from the academic perspective, we can use assessment to further direct our programs to meet professional expectations and, far more importantly, to help us meet the needs of the well-educated technical communicator.

Allen, Jo. Technical Communication Quarterly (2004). Articles>Education>Assessment>Technical Writing

Last Rites for Readability Formulas in Technical Communication    

Some reading researchers and technical communicators assume the efficacy of readability formulas. Reading researchers use such formulas to equalize the reading difficulty of texts used in experiments. Results of an informal Internet survey indicate that some professional writers and editors use readability formulas that are integrated into word-processing software. This article proposes that readability formulas fail to predict text difficulty. The results of an experiment demonstrate that "text difficulty" is a perception of the reader and therefore cannot be objectively calculated by counting syllables, word length, sentence length, and other text characteristics.

Connatser, Bradford R. Journal of Technical Writing and Communication (1999). Articles>Writing>Assessment>Formulas

Perceptions Of Memo Quality: A Case Study Of Engineering Practitioners, Professors, and Students    

One goal of college technical writing courses is to prepare students for real-world writing situations. Business writing textbooks function similarly, using guidelines, sample assignments, and model documents to help students develop rhetorical strategies to use in the workplace. Students attend class, or read and perform exercises in a textbook, with the faith that these skills will apply to workplace writing. In an attempt to better understand the similarities and differences between industry and academe's expectations of one genre of workplace writing, the memo, we compared the perceptions of memo quality by engineering faculty, students, and practitioners. All three groups responded to three sample memos taken from textbooks used by engineering professors in their undergraduate classrooms. The results indicate that students' and engineers' opinions of memo quality were more closely related to one another than to professors' comments, focusing on content, while professors were the most critical of style issues.

Amare, Nicole and Charlotte Brammer. Journal of Technical Writing and Communication (2005). Articles>Writing>Assessment>Engineering

Quality Metrics   

Technical communicators continuously battle with the problem of obtaining an objective and comparable representation of a document's quality. While everyone agrees on the importance of this issue, no definitive work exists on determining or representing the quality of a document. As Toby Frost states, 'It may take a long time to establish a baseline for quality metrics; we don't have adequate mechanisms for measuring quality today.' A quality metric provides a method for tracking a document through completion, helps ensure quality deliverables, and provides an additional criterion for personnel reviews.

Mallory, Eric. STC Proceedings (1999). Articles>Writing>Assessment>Heuristic Evaluation

Readability  

I know from some years running a reading clinic in the United States that you can make more progress if you start the students out on relatively easy reading material so the the students can read the materials with some comprehension and success. But how do you tell these African instructors how to select “relatively easy” reading materials in technical English? The answer - use a readability formula.

Fry, Edward. Impact Information (2006). Articles>Writing>Assessment>Formulas

Readability Formulas and Writing for the Web

There is a considerable amount of information published on the Web that is intended to be read by someone. There is evidence that much of the information may be too hard to read and understand for typical readers. Baker, Wilson and Kars (1997) reported that the readability scores of most articles in the 'Health Reference Center' ranged from 10th to 14th grade levels. Another study (Graber, Roller and Kaeble, 1999) included text-based information from commercial, academic and government sites. They found that the reading material averaged the 10th grade level. In a more recent study, a group of researchers (D'Alessandro, et.al., 2001) conducted readability analyses of pediatric patient education materials on the Web, and concluded that the information was not written at an appropriate reading level for typical users.

Bailey, Robert. Web Usability (2002). Articles>Writing>Assessment>Formulas

Readability Formulas Have Even More Limitations Than Klare Discusses    

A literature review reveals many technical weaknesses of readability formulas (when compared to direct usability testing with typical readers): they were developed for children's school books, not adult technical documentation;they ignore between-reader differences and the effects of content, layout, and retrieval aids on text usefulness; they emphasize countable features at the expense of more subtle contributors to text comprehension.

Redish, Janice C. 'Ginny'. Journal of Computer Documentation (2000). Articles>Writing>Assessment>Formulas

Readability Formulas in the New Millennium: What's the Use?    

While readability formulas were intended as a quick benchmark for indexing readabilty, they are inherently unreliable: they depend on criterion (calibration) passages too short to reflect cohesiveness, too varied to support between-formula comparisons, and too text-oriented to account for the effects of lists, enumerated sequences,and tables on text comprehension. But readability formulas did spark decades of research on what comprehension really involoves.

Schriver, Karen A. Journal of Computer Documentation (2000). Articles>Writing>Assessment>Formulas

Rhetorically-Based Heuristic For Technical Writing Students: A Guideline for Handling the Inundation of Information on the National Information Infrastructure  

In the age of the national information infrastructure, we are inundated with information from various sources. Each day, we come into contact with much more information than we can ever assimilate, and the amount of information that we have access to constantly increases. We generate more and more information daily, and with modern technology, such as Internet, we have immediate access to libraries and databases worldwide.

Jeansonne, Jerold. STC Proceedings (1994). Articles>Writing>Assessment>Technical Writing

A Syntactic Approach To Readability    

Focusing on the issue of readability, this article examines problems that readability formulas present to the technical communicator, especially in terms of interaction with government agencies, and focuses on readability formula requirements mandated by The Office of Health and Industry programs [OHIP] for medical technology product support literature. Because the Flesch Reading Ease and the Flesch-Kincaid formulas are widely available, they are probably the ones most frequently used. Contemporary readability scholars have overlooked the Golub Syntactic Density Formula, which evaluates prose according to a sentence's syntax at a deeper level than the number of words per sentence and the number of syllables per word. The authors recommend it as a tool for evaluating readability. How it might be applied with current computer applications is discussed.

Giles, Timothy D. and Brian Still. Journal of Technical Writing and Communication (2005). Articles>Writing>Assessment>Rhetoric

Toward an Informed Citizenry: Readability Formulas as Cultural Artifacts    

After World War II, the United States government and citizenry were concerned with truth, propaganda, democracy, and national security as they entered the Cold War era. This was a time when technocrats, engineers, and scientists could lead our free-world government through the perils of our tense relationships with Russia, Red China, and Korea. In the early 1940s, Rudolf Flesch began developing what he termed a "scientific rhetoric" to help writers of functional documents more effectively communicate technical information to a general public. He came up with a readability formula to help writers evaluate whether their writing was effective and this readability formula has profoundly shaped notions of "clear writing" for the last 60 years. This article explores Flesch's development of this readability formula, placing his work in a historical context, as well as discussing how the readability formula fit into a larger project to make effective writing more of a science than an art.

Longo, Bernadette. Journal of Technical Writing and Communication (2004). Articles>Writing>Assessment>Formulas

The Undervaluation of Writing Expertise  

Main argues that technical writing departments should value writing expertise more than tool experience when evaluating job candidates.

Main, Michael D. Intercom (2001). Articles>Writing>Assessment

Using New Technology to Assess the Academic Writing Styles of Male and Female Pairs and Individuals    

Background: Previous research suggests that there are advantages to writing in groups or in pairs compared with writing individually, and that men write differently from women. However, as far as we know, no one has yet used new technology to assess published academic articles written in these different modes. Method: We assembled 80 papers from recent issues of the Journal of Educational Psychology as follows: 21 authored by individual men, 21 by individual women, 19 by pairs of men, and 19 by pairs of women. We then used two computer-based measures to assess various textual features of the Abstracts, the Introductions, and the Discussion sections of these 80 papers. Results: Several differences were found between these various parts of the journal articles (e.g., the Discussions were more readable than the Introductions and these in turn were more readable than the Abstracts). However, there were few differences between the writing of pairs or individuals, or between that of men and women. Conclusions: There was no real evidence to support the notion that writing in pairs would lead to better quality articles or that there would be differences between the readability of papers produced by men and women. Such differences may occur, however, before peer review.

Hartley, James, James W. Pennebaker and Claire Fox. Journal of Technical Writing and Communication (2003). Articles>Writing>Assessment>Gender

You Can't Fix What You Can't Measure: Toward Better Content Metrics  

Fenstermacher writes that while there are substantial benefits to globalized content, it is difficult to develop metrics to analyze this content. He discusses key terms and processes that can be used when considering cost-reduction strategies to get you on your way to developing your own content metrics.

Fenstermacher, Hans E. Intercom (2006). Articles>Writing>Technical Writing>Assessment