Automated Essay Scoring – Robo-Readers that Mark Essays

by Carla Douglas

Image by MattHurst (CC BY-SA 2.0)

What if a software program could take the drudgery out of marking student essays by doing the work for you? Would you use it?

The technology is out there. Automated Essay Scoring (AES) technology or, more informally, the robo-reader, has been in development for some time.
Here’s how it works. The robo-reader compares an essay (it can’t actually read it) to other essays that have been scored by humans, and it “faithfully replicates” the human scores. The software can grade essays about as accurately as humans can, and it works a lot faster, too. For a detailed explanation of the robo-reader’s capabilities and a description of the study that tested them, read Justin Reich on the subject at EdTechResearcher.    
The phrase “faithfully replicates” is used by Dr. Mark Shermis, who authored the study that assessed a number of different automated scoring programs against human scorers. In conversation with Nora Young at CBC’s Spark, he explains that the software doesn’t evaluate a student essay the same way a human would, but uses a different logic entirely.
For instance, it may use language processing to look for keywords such as first, second, third and finally to assess structure. Or the program may look for a topic sentence and subsequent words that link to or support the topic sentence in order to assess the development of an argument or idea.
Sound familiar? Main idea, topic sentence, supporting details, transitional words and phrases — all the components English teachers focus on … ad nauseam. Even on higher-order tasks, such as essays that require students to read and respond to a passage of text, the software performs surprisingly well.
What does this mean for teachers? Some argue that it could further devalue and depersonalize education. Others, Shermis and Reich* included, believe it is a tool that could help to rapidly improve students’ writing skills by providing nearly instant feedback on elements such as structure, development of ideas, and so on. More writing assignments + faster feedback = improved writing skills.
So, would you use it? It certainly provides plenty to think about.
*Read Part II  and Part III of Reich’s post in which he discusses the possible effects of robo-readers on education policy and their implications for the classroom.  

Leave a Reply

Your email address will not be published. Required fields are marked *