Thursday, May 1, 2014

Grading Software fooled by BABEL

BABEL, the Basic Automatic B.S. Essay Language generator, is software created by Les Perelman and others [] at MIT. Perelman was a "Director of Undergraduate Writing" at MIT. He has used BABEL to generate nonsense essays and feed them to automated essay grading software. The BABEL output gets high marks with sentences like "Privateness has not been and undoubtedly never will be lauded, precarious, and decent." I think Perelman has a point here. Clever students will no doubt learn to game any such grading system to their benefit. Teachers must question what, if any, benefit an automated essay grading system has.

Back in 2005, Perelman discovered an excellent predictor of score [] on an SAT essay test. It was the length of the essay. No other variable he examined correlated nearly as well with the score. The top scoring essays had many factual errors, too. No matter, according to SAT. The writing quality depends not on the correctness of any facts, according to SAT. Perelman's advice for scoring well is to practice writing fast and make up facts. Perhaps the high scorers can land a job with Fox News?

A paper by Perelman in the Journal of Writing Assessment critiques automatic scoring of essays.

1 comment:

  1. I wonder if the essays for the Praxis tests are fed through an automatic grading system? I'm not sure what I can say about the test, given the confidentiality forms we sign when taking it. But I know that I made up facts, constructed plausible arguments based on those "facts" with the appropriate paragraph transitions, etc, and got a good score on the result. I'm glad I had read about the five-paragraph essay form, and how students were fooling it and spitting out B.S. yet still scoring well on multiple named exams. Otherwise, I would have felt far more anxiety before taking that test. Knowing that it didn't matter if what I said was true helped tremendously and I could focus on producing the writing.