STEM education provides many opportunities and challenges. How can our practice evolve to meet the needs of 21st-century learners?

Technical Writing: How Usability Prepares Students for Essay-Scoring Software

T.R. Girill Technical Literacy Project leader, STC and LLNL

T. R. Girill
Society for Technical Communication/Lawrence Livermore National Lab.
trgirill@acm.org

Technical Writing: How Usability Prepares Students for
Essay-Scoring Software

Now that essay questions are starting to appear on standardized
tests, the high cost of human readers has accelerated the push to
develop software that scans each student essay and outputs an
evaluation just as a real person would. In fact, one recent
foundation-funded study compared no less than eight such programs
that automate essay scoring (Mark D. Shermis and Ben Hammer,
"Contrasting State-of-the-art Automated Scoring of Essays: Analysis,"
April 2012, online at
http://dl/dropbox.com/u/44416236/NCME%202012%20Paper3_29_12.pdf ).
If this is the future that your students face, how can you prepare
them to draft nonfiction science text that diverse essay-scoring
software would rate highly?

How Text Looks to a Software Robot

Automated essay scoring usually focuses on one of four kinds
of text features, relying on that feature to index or reflect
overall essay quality (a few programs try to meld several feature
types from this same set). The kinds are:

Linguistic Features.
Some programs look for likely linguistic mistakes (such as
subject/verb disagreement in sentences or inappropriate use of
English articles ('a' or 'the')), or count and compare such
linguistic features as prepositions used or vocabulary diversity.
These are just the hardest English features for many non-native
speakers to deploy fluently and flawlessly.

Statistical Features.
During the 1980s, many "readability formulas" tried to estimate
text quality by calculating such quantitative features as average
sentence length (in words) or average word length (in syllables).
Some essay-scoring programs today also rely on a similar
statistical approach.

Logical Features.
When you read a text you are concerned about what it asserts (the
logical propositions that it contains) and how those assertions
fit together (the logical argument that the text advances). Some
software likewise tries to identify propositions and arguments,
even when expressed in diverse linguistic forms.

Historical Features.
This longitudinal or developmental approach to automated essay rating
is the most indirect: the program scans the same 12 million words
(but only about 80,000 unique words) of reading material that a
typical high-school student would encounter in their lifetime, then
compares their essay to that corpus looking for revealing trends.

The Usability Response

So how can your students prepare themselves for these feature-
based rating schemes without obsessively, mechanically trying to
manipulate each feature that the software might check? Fortunately,
there is a strategic alternative. Readers of this technical-literacy
series know that several decades of empirical usability research
have yielded text-design techniques that help humans construct
prose that other humans can really use to solve "far-transfer"
science and engineering problems. Real people value such prose
because they find it so effective. But, as a side benefit, it also
ends up with just the features that essay-scoring software rates
highly (but not installed in an artificial way).

One crude but revealing indicator of this overlap between the
usability approach to text engineering and the "computational needs"
of automated essay rating appears in the index to Karen Schriver's
influential handbook "Dynamics in Document Design" (Wiley, 1997).
I easily found half a dozen index entries that reflected
longitudinal/developmental topics in her book, seven that addressed
linguistic features, and 12 index entries on readability/statistical
issues. Schriver's entire 55-page Chapter 3 ("How documents
engage readers' thinking and feeling") explores the impact of
evidence and propositional clarity on text quality.

So the strategic way of "beating the essay software" is to
learn about and practice writing techniques that yield usable,
effective technical text from the inside out. Your students are
unlikely to hear the word "usability" in language arts class,
however....that burden will likely fall to you. Want a quick
review of usability issues and research? See "Usability: How
Technical Writing Succeeds" online at
http://www.ebstc.org/TechLit/handbook/usability.html