George Lucas Educational Foundation

Technical Writing: How Usability Prepares Students for Essay-Scoring Software

Technical Writing: How Usability Prepares Students for Essay-Scoring Software

Related Tags: STEM
More Related Discussions
  • Facebook
  • Twitter
  • Pinterest
  • Share
T. R. Girill Society for Technical Communication/Lawrence Livermore National Lab. Technical Writing: How Usability Prepares Students for Essay-Scoring Software Now that essay questions are starting to appear on standardized tests, the high cost of human readers has accelerated the push to develop software that scans each student essay and outputs an evaluation just as a real person would. In fact, one recent foundation-funded study compared no less than eight such programs that automate essay scoring (Mark D. Shermis and Ben Hammer, "Contrasting State-of-the-art Automated Scoring of Essays: Analysis," April 2012, online at http://dl/ ). If this is the future that your students face, how can you prepare them to draft nonfiction science text that diverse essay-scoring software would rate highly? How Text Looks to a Software Robot Automated essay scoring usually focuses on one of four kinds of text features, relying on that feature to index or reflect overall essay quality (a few programs try to meld several feature types from this same set). The kinds are: Linguistic Features. Some programs look for likely linguistic mistakes (such as subject/verb disagreement in sentences or inappropriate use of English articles ('a' or 'the')), or count and compare such linguistic features as prepositions used or vocabulary diversity. These are just the hardest English features for many non-native speakers to deploy fluently and flawlessly. Statistical Features. During the 1980s, many "readability formulas" tried to estimate text quality by calculating such quantitative features as average sentence length (in words) or average word length (in syllables). Some essay-scoring programs today also rely on a similar statistical approach. Logical Features. When you read a text you are concerned about what it asserts (the logical propositions that it contains) and how those assertions fit together (the logical argument that the text advances). Some software likewise tries to identify propositions and arguments, even when expressed in diverse linguistic forms. Historical Features. This longitudinal or developmental approach to automated essay rating is the most indirect: the program scans the same 12 million words (but only about 80,000 unique words) of reading material that a typical high-school student would encounter in their lifetime, then compares their essay to that corpus looking for revealing trends. The Usability Response So how can your students prepare themselves for these feature- based rating schemes without obsessively, mechanically trying to manipulate each feature that the software might check? Fortunately, there is a strategic alternative. Readers of this technical-literacy series know that several decades of empirical usability research have yielded text-design techniques that help humans construct prose that other humans can really use to solve "far-transfer" science and engineering problems. Real people value such prose because they find it so effective. But, as a side benefit, it also ends up with just the features that essay-scoring software rates highly (but not installed in an artificial way). One crude but revealing indicator of this overlap between the usability approach to text engineering and the "computational needs" of automated essay rating appears in the index to Karen Schriver's influential handbook "Dynamics in Document Design" (Wiley, 1997). I easily found half a dozen index entries that reflected longitudinal/developmental topics in her book, seven that addressed linguistic features, and 12 index entries on readability/statistical issues. Schriver's entire 55-page Chapter 3 ("How documents engage readers' thinking and feeling") explores the impact of evidence and propositional clarity on text quality. So the strategic way of "beating the essay software" is to learn about and practice writing techniques that yield usable, effective technical text from the inside out. Your students are unlikely to hear the word "usability" in language arts class, however....that burden will likely fall to you. Want a quick review of usability issues and research? See "Usability: How Technical Writing Succeeds" online at

This post was created by a member of Edutopia's community. If you have your own #eduawesome tips, strategies, and ideas for improving education, share them with us.

Comments Sign in or register to comment Follow Subscribe to comments via RSS

Sign in to comment. Not a member? Register.