automated essay grading

In the Times today:

Essay-Grading Software Offers Professors a Break By JOHN MARKOFF
Published: April 4, 2013 | New York Times

I'm actually in favor of essay grading software, in theory. I've been interested in automated essay scoring ever since reading Richard Hudson's paper Measuring Maturity in Writing (which I need to re-read, so nothing more on that at the moment):

Abstract
The chapter reviews the anglophone research literature on the 'formal' differences (identifiable in terms of grammatical or lexical patterns) between relatively mature and relatively immature writing (where maturity can be defined in terms of independent characteristics including the writer's age and examiners' gradings of quality). The measures involve aspects of vocabulary as well as both broad and detailed patterns of syntax. In vocabulary, maturity correlates not only with familiar measures of lexical diversity, sophistication and density, but also with 'nouniness' (not to be confused with 'nominality'), the proportion of word tokens that are nouns. In syntax, it correlates not only with broad measures such as T-unit length and subordination (versus coordination), but also with the use of more specific patterns such as apposition. At present these measures are empirically grounded but have no satisfactory theoretical explanation, but we can be sure that the eventual explanation will involve mental growth in at least two areas: working memory capacity and knowledge of language.

Maturity of writing, in this sense, can be measured by software, and I would be using automated scoring software myself if I could buy essay-scoring software on Amazon. EdX says it's giving software away free to 'institutions' (does that leave out individuals?) so I'll have to see if my department might throw its hat in the ring.

That said, a lot of this is nonsense:

Anant Agarwal, an electrical engineer who is president of EdX, predicted that the instant-grading software would be a useful pedagogical tool, enabling students to take tests and write essays over and over and improve the quality of their answers. He said the technology would offer distinct advantages over the traditional classroom system, where students often wait days or weeks for grades.

[snip]

“It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right,” said Daphne Koller, a computer scientist and a founder of Coursera.

[snip]

“One of our focuses is to help kids learn how to think critically,” said Victor Vuchic, a program officer at the Hewlett Foundation. “It’s probably impossible to do that with multiple-choice tests. The challenge is that this requires human graders, and so they cost a lot more and they take a lot more time.”

None of these things is going to happen. Students aren't going to write essay responses "over and over again;" if they do write essay responses over and over again it's not going to feel like a fun game; and nobody's going to learn to think critically from automated essay scoring software.

Oy.

a teaching assistant at a "top university on the East Coast of the USA" grades a paper

Huddleston and Pullum (scroll down)

When I say "no accountability," this is one of the things I'm talking about: a teaching assistant who does not know what the passive voice actually is given the power to grade student writing.

the passive voice files

Rodney Huddleston and Geoffrey Pullum on the passive voice:

In truth, the passive is very often exactly the right way to frame a clause in a particular context, and all competent authors use passives frequently. The people who recommend against it use it themselves, even while talking about how you should not use it. For example, in the act of explaining that you should "Use the active voice" because it is "more direct and vigorous than the passive", William Strunk and E. B. White assert that "Many a tame sentence . . . can be made lively and emphatic by substituting a transitive in the active voice" (see section 14 of their book The Elements of Style). Their sentence defies their warning; it contains an instance of the passive voice itself (can be made lively and emphatic).

from the archives: grading student writing

from 2009:

Students have long believed (on good evidence) that if the same paper is submitted to two teachers in two different sections of the same course, the paper is likely to receive two very different grades. In 1961, Paul Diederich and his colleagues proved that this student belief is no myth. When 30 student papers were graded by fifty-three graders (a total of 15,900 readings), more than one third of the papers received every possible grade. That is, 101 of the 300 papers received all nine grades: A, A-, B+, B, B-, C+, C, C-, and D. Diederich also reported that

94 percent [of the papers] received either seven, eight or nine different grades; and no essay received less than five different grades from fifty-three readers. Even when the raters were experienced teachers, the grades given to the papers by the different raters never attained a correlation greater than .40. Diederich, P.B., French, J.W., and Carlton, S.T. "Factors in judgments of writing ability." Research Bulletin RB-61-15. Princeton, N.J.: Educational Testing Service, 60 pp.
The Schools We Need and Why We Don’t Have Them
E. D. Hirsch
185-188