Thursday, May 1, 2014

Grading Software fooled by BABEL

BABEL, the Basic Automatic B.S. Essay Language generator, is software created by Les Perelman and others [] at MIT. Perelman was a "Director of Undergraduate Writing" at MIT. He has used BABEL to generate nonsense essays and feed them to automated essay grading software. The BABEL output gets high marks with sentences like "Privateness has not been and undoubtedly never will be lauded, precarious, and decent." I think Perelman has a point here. Clever students will no doubt learn to game any such grading system to their benefit. Teachers must question what, if any, benefit an automated essay grading system has.

Back in 2005, Perelman discovered an excellent predictor of score [] on an SAT essay test. It was the length of the essay. No other variable he examined correlated nearly as well with the score. The top scoring essays had many factual errors, too. No matter, according to SAT. The writing quality depends not on the correctness of any facts, according to SAT. Perelman's advice for scoring well is to practice writing fast and make up facts. Perhaps the high scorers can land a job with Fox News?

A paper by Perelman in the Journal of Writing Assessment critiques automatic scoring of essays.