Les Perlman and the E.T.S.: Taking on Automated Grading

Les Perlman offers a satirical commentary on the ineffectiveness of electronic grading.  The issue was initially brought to the national stage in March 2007 at the Conference on College Composition and Communication.  At this conference and subsequent annual ones, Perlman has demonstrated through panels, reviews and in this case–satirical writing– that grading systems being phased in by the Electronic Testing Service and College Board are ineffective and do not accurately grade writing  Therefore, Perlman crafted this essay (Exhibit B) in which he seemingly answers a College Board-styled writing prompt.  Many factors contribute to Perlman’s elegantly crafted scathing of the ETS’s methodology, but most critical to his work is proper rhetorical capability.  Perlman, under the constraints of the five paragraph form, criticizes electronic grading by deliberately writing nonsense into structure and subsequently forming a masked logical argument against electronic grading.

This entire essay is nonsense.  Many thoughts fail to agree or blend properly as Perlman shifts between ideas mid-paragraph demonstrating a lack of focus.  In fact, the majority of Perlman’s essay misplaces thoughts and follows inconsistent build up of evidence in order to prove his relatively genuine thesis.  In the second paragraph, he presents one of his humorous shifts, ”

In the Middle Ages, the University of Paris grew because it provided comfortable accommodations for each of its students, large rooms with servants and legs of mutton. Although they are expensive, these rooms are necessary to learning. The second reason for the five-paragraph theme is that it makes you focus on a single topic.

However, there is an immediate clash; Perlman is identified as a former professor at MIT.  This fact, along with many references to classic works and authors despite how farcical they are–Great Expectations, Heart of Darkness, Oscar Wilde, etc.– establishes Perlman’s credibility, or in terms of rhetorical analysis, ethos.  With these references and the identification as a professor, it seems illogical for Perlman to write such a nonsense essay.  This disconnect is humorous, so much so that the essay has proliferated the internet on blogs and news sites.

There is much more built in to Perlman’s humor which makes his essay an effective critique of the Electronic Testing Service.  From a rhetorical standpoint, examine the intended audience.  Noting that the essay was written to be graded by an electronic grader, one must determine that the legitimate intended audience was a computer.  This is a defining pillar the essay relies upon.  An electronic grading system looks for large words, complex sentences pristine organization and elegant parallelism, such as this well crafted sentence:

Second, most teaching assistants have political connections, from being children of judges and governors to being the brothers and sisters of kings and princes.

In a numerical structure, he uses active voice using a complex sentence consisting of both independent and dependent clauses and keeps the sentence rigidly parallel.  Perlman’s essay presents all of these factors beautifully, meeting his intended audience’s limited demands with ease and reportedly receiving a 6, the highest grade offered by the service for an electronically graded essay.1 Here is the disconnect: the actual audience is massive, including individuals across the world using the internet, and the basic presuppositions individuals have about the ideas and progression of an essay are fundamentally different than those of a computer.

What completes Perlman’s critique actually is not written by Perlman himself, but instead was crafted by the E.T.S. in response to the New York Times article which broke the story of Perlman’s essay online, ”

E.T.S. officials say that Mr. Perelman’s test prep advice is too complex for most students to absorb; if they can, they’re using the higher level of thinking the test seeks to reward anyway. In other words, if they’re smart enough to master such sophisticated test prep, they deserve a 6[…]

As for good writing being long writing, Mr. Deane [Principal Scientist at E.T.S.] said there was a correlation. Good writers have internalized the skills that give them better fluency, he said, enabling them to write more in a limited time.”

Finishing Perlman’s hidden logic for him, the E.T.S. determines Perlman’s hidden criticism to be of a “higher level of thinking” than their tests actually measure and therefore they deserve the highest grade.  In effect, Perlman has written a nonsensical essay with hidden logic and outward humor in order to force the E.T.S. to highlight the problem they created in the first place: their tests are no longer properly measuring the aptitude of students, they are merely testing their ability to organize text into an essay.  Overall, Perman perfectly uses advanced rhetorical techniques in a nonsense essay to write a satire criticizing electronic grading.

1.http://www.cbc.ca/spark/wp-content/uploads/2012/05/Essays-for-Robo-Reader.pdf (Exhibit B)

This entry was posted in Uncategorized. Bookmark the permalink.

9 Responses to Les Perlman and the E.T.S.: Taking on Automated Grading

  1. tksekf says:

    I couldn’t disagree with Perlman. This automated grading system is ruining what a ‘test’ is supposed to be. Instead of testing our own abilities and capacities, ETS and collegeboard are now testing how well we have analysed their exams and how well we are able to follow their grading system. His nonsense essay perfectly depicts those exams.

  2. sm1414 says:

    I agree with everything stated in the post. In general standardized tests do not actually test if a student knows how to write a cohesive, high-level essay, nor do they test whether students can actually function in the real world. The state from which I am from gives an exam written at an 8th grade level to high school juniors so that they can graduate. It is absolutely ridiculous. I was in AP Language and Composition and we learned to write a real essay that does not follow the schematic that Perlman criticizes here and I received a lower grade on this standardized test than some of the students in a regular College Prep English class. These tests are not about whether students have been educated, but rather how well they can take a test and how efficiently the state can grade them.

    • I agree with most of what you’re saying. However, I also took AP Lang and Comp and AP Lit, and in both those classes we were taught to write in the same 5 paragraph structure Perlman’s essay mocks. Although the AP essays are not computer graded, they are still standardized, and the courses which are meant to prepare you for them, do just that, meaning they teach almost exclusively to the test. Although these courses may teach you to write at a higher level, in the end your AP score boils down to how well you take their test. I would argue that although the AP tests grade more heavily on writing ability than the state standardized tests, they are still very restrictive, and do a poor job of preparing students for college and the real world. For example the first paper due for this UW class is a 4 to 5 page paper with a research paper style format. I have spent the past 2 years writing high level 5 paragraph essays in my English classes, without this style of writing being practiced once. This is a direct result of the fact that these types of papers are not on the AP test, and therefore not relevant to the most immediate goal of an AP Lit or AP Lang and Comp class, which is having students succeed on the AP test. Such a narrow focus on tests leads to student being unprepared to function in situations that don’t match up with the strict guidelines they were taught to follow.

  3. running95 says:

    I completely agree with both of these comments–the only concern of the ETS is ‘can students pass this test,’ and not ‘have students been taught the proper tools to succeed in college and life in general.’ I, too have to take a high school proficiency test that is set at an eighth grade level. The problem with that proficiency assessment is that it is set at a level at which the majority of students statewide can master which means that at least half of the state’s students can pass with ease and thus the test is not assessing students’ readiness to leave high school, but rather ascertaining whether or not they should be there in the first place. Moreover, the idea that being able to master the system is displaying an aptitude higher than what is being tested is absolutely ridiculous. It is a fact that as research and comprehension of the world improve the inhabitants of the world become increasingly more intelligent, and therefore, standardized tests should become more and more advanced to keep pace with the growing intelligence of students.

  4. drc1995 says:

    I agree as well with the above comments and the original post. It truly was interesting to see just how creative Les Perlman could get in getting around the ETS system by following the customary five paragraph format that I’m sure we all know about from our high school years, while all along making a mockery of the system itself. I think it just goes to show you just how much our education system in the US has seemingly been dumbed down and how this trend continues on. We have gotten to the point where we have a learning system that is focused on memorization, and a grading system that is based on format and structure, rather than being focused on actual content and overall worth to the student in both cases. For those who didn’t take AP English or similar AP’s such as myself, this 5 paragraph structure we’ve used our entire lives that Perlman so eloquently discredits really won’t be of any use to students neither in college nor in the outside world after college. I’m glad I have finally seen someone openly criticize it, and hope that more attention comes to the shortfalls of these types of systems in the future.

  5. roberly2 says:

    The post does a remarkable job highlighting the satirical undertones presented in the piece as well as forcing the reader to acknowledge that Perlman has undermined an educational staple that we have been taught our whole literate lives to believe is the ultimate tool of effective writing. While at first glance it is easy to dismiss Perlman as writing nonsense, or even as writing to specifically discredit the prompt topic itself- demands by college students are raising the price of education- but the deeper understanding of the piece requires us to also understand that the author is using structure in a novel and unique way. This post is very effective in demonstrating Perlman’s expertise as a write and innate understanding of both the faulty grading system as well as the ineffective five- paragraph style. Personally, I had to rethink a few truths after reading Perlman’s essay; while not necessarily a stickler for the five- paragraph structure, I do have a respect for a certain degree of rigidness in my writing, and i have a tendency to overuse words rather than be concise. Perlman has showed me both the freedom of construction in an essay as well as the power of using fewer words to convey a message.

  6. phishmonkees says:

    Les Perlman displays that there are flaws in the standardized testing system and that the writing section is manipulable. Regardless, I believe that standardized testing is a universal method that exhibits an applicant’s aptitude as well as his efficiency. I agree with previous commenters such as sm1414 that standardized testing does not test whether students can actually function in the real world. However judging someone’s functionality in the real world is somewhat impossible because it is subject to extreme bias. In my opinion there is no test that can measure someone’s true functionality. However there are other crucial qualities that are can be extracted from standardized test scores, such as whether someone can handle the academic rigor of a college.

  7. foldervral says:

    This post accurately describes how Les Perlman completely dismantled the credibility of automated grading systems, and I cannot help but agree. Standardized testing does force the test taker to format their essay into a rigid structure, but an essay consists of more than pure structure. There are some students that can make their essays flow quite well, and still makes structural errors. Other students have virtually no flow, yet strictly adhere to the structure of the standardized format. Even more students often include information that may not be completely relevant to the question at hand. A standardized test is meant to produce a comprehensive summary of a students overall skill. Les Perlman has proven that automated grading only addresses one of the three largest categories needed to be included when grading an essay. The fact that E.T.S states that if a student out-thinks the program than that student deserves the 6 is ridiculous. The test is meant to grade the students essay writing skills and understanding of the topic, not whether he or she can find a loophole in the system. Theotherhemingway is correct in praising Les Perlman on such a masterful job in proving that no matter how smart a computer get’s it will not compete with human analysis.

  8. aarongeiger says:

    Of note: a colleague at Carnegie Mellon demonstrated the that it was statistically and quantifiably possible to have a better-than-average chance of linguistically detecting sarcasm in text. Pearson can’t do even that, yet. We have a long way to go before robo-grading. Not everything needs to be automated. Even instructors and professors need the exercise of grading in order to effectively teach.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s