An Experimental Study of Placement Essay Examination Scoring by Writing Teachers and Non-Writing Teachers with Consideration of Content Effects on Ratings

Measurement experts have criticized the use of essay examinations,
chiefly because of their scoring unreliability. Research has emnhasized
scoring inconsistencies and that scorers of essay examinations are frequently
led by composition errors or other writing quality factors to
grade the subject matter--content--of the essays inconsistently. Studies
critical of essay scoring have emphasized the content of essay answers
and have used scorers with minimal expertise in the essays' subject areas.
Recent research reveals that, under certain conditions, i.e., when
writing experts are scoring for placement, using appropriate grading
criteria, reliability rates for essay scoring are improving. Besides
making distinctions between grading conditions and purposes, recent
studies distinguish between content and writing skills, and emphasize
examination of how content characteristics of student essays influence teachers' ratings.
This study focused on whether freshman placement essays would be
graded consistently by different groups of college faculty, experts in
writing and experts in the topic of the essays. Also examined were how
content designated as strong and weak would influence the overall scores
and the scoring of writing quality.