|
Q: What is the Intelligent Essay Assessor™?
A: The Intelligent Essay Assessor (IEA) is a powerful internet-based
service that has been proven to score written essays as accurately as
trained human scorers. IEA scores essays based on content as well as
on more mechanical aspects of writing, such as
grammar and spelling.
Q: How does IEA work?
A: IEA uses the Knowledge Analysis Technologies™ (KAT) engine, a
patented technology based on over twenty years of research and development.
The KAT engine is based on Latent Semantic Analysis, a computational
technique that provides a sophisticated analysis of text.
It assesses the content of an essay by comparing it against a set of
essays previously scored by expert human readers.
IEA assigns a score to each essay based in part
on the similarity of the content of the essay to the training set
essays.
Q: What is the KAT™ engine?
A: The KAT engine evaluates the meaning of text by examining whole passages.
The KAT engine is based on Pearson's unique
implementation of Latent Semantic Analysis (LSA), an approach that infers
semantic similarity of words and passages by analyzing large bodies of
relevant text. LSA can then understand the meaning of text in much the same
way as a human reader. More information on the KAT engine is available at
http://www.pearsonkt.com/prodIEA.shtml.
Q: Does IEA use keywords in its evaluation?
A: A common misconception is that IEA scores by recognizing
keywords. It does not. If two or more different terms accurately describe
a concept, IEA will recognize either or both and score them
equally as well. For example, IEA measures
"Understanding is very important when you read something you want
to know about"
as very similar in meaning to
"Comprehending's essential"
even though the sentences share no words in common. A keyword
based approach to similarity would find nothing in common between these
two sentences.
Q: How is the computer trained to score student
essays?
A: Partner companies wishing to offer an essay scoring service collect
100 - 200 student papers written in response to a given prompt. These
papers are then scored by
human graders and sent to Pearson, where the papers
and their scores are used to train the computer to score new student essays
in response to the prompt. The KAT engine learns to score the different
score points based on the human scored papers. The engine can be
trained to provide holistic as well as analytic or trait scores.
Q: How does the human scoring work?
A: Human graders assess a paper's overall quality using a specific rubric.
For analytic scoring, they examine a paper for important traits. Each essay
is scored by two graders for the holistic score and again by two graders
for each of the analytic traits. If the two graders diverge by more than
one point on any score, a third grader scores the paper to settle that
discrepancy.
Q: How does the computer recognize a good essay?
A: The Intelligent Essay Assessor uses the KAT engine to assess the
content of an essay, as well as more mechanical aspects of writing. When
a student
submits an essay for scoring, the system immediately measures the meaning
of the essay. It then compares the essay to the training essays, looking
for similarities, and assigns a holistic score in part by placing the essay
in a category with the most similar training essays. Analytic scoring
occurs in much the same way. For each trait, the system assesses that
trait in the student essay, compares it to the training essays, and then
categorizes the trait in question.
Q: How does IEA scoring compare to the way teachers
grade writing?
A: IEA's approach mirrors the way teachers grade essays. For example, when
teachers evaluate a student's essay, they look for characteristics that
identify an essay as an A or C paper. Their expectations are likely based
on their previous experience as a grader and on criteria for the assignment
in question. In other words, teachers search for a match between the essay
itself and the criteria for a particular grade or score. The Intelligent
Essay Assessor is trained to mimic this process.
Q: How does IEA score essays with highly unusual
writing styles?
A: An essay with a highly unique writing style or unusual construction may
receive an advisory message along with a score. If an essay is off-topic,
written in a language other than English, too brief or too repetitive, a
written refusal to write, or otherwise incomprehensible, a student will
receive an advisory and no score. These
advisory messages ask the student to discuss the essay and all feedback
with his or her teacher to ensure an appropriate evaluation of the writing.
Q: What research has been conducted on IEA?
A: Pearson has conducted extensive research on the reliability and validity of
the Intelligent Essay Assessor. The correlation between IEA and human
graders has been shown to be as high or higher than that between two
independent human graders in dozens of studies with over 200 prompts of
every type. The correlation and agreement rates of the scores produced by
IEA are better the more expert and reliable the human scores. IEA's scores
have been shown to reflect progress in writing skills and knowledge as a
result of instruction much more sensitively than human scores.
For more information, please see the
Reliability and Validity of the KAT™ Engine paper available on the
Research page at
http://www.pearsonkt.com.
Q: How long does it take to score an essay?
A: Most essays are scored within a few seconds. Keep in mind that a slow
or busy Internet connection may cause it
to take longer to receive the scores.
Q: What feedback do students receive?
A: Through our partners, students can receive a variety of feedback on
their essays, such as a holistic score and analytic scores on six traits
of writing. Students may also receive feedback on spelling, grammar
and redundancy, as well as on the length of their essays. An example of
student feedback is available at: http://www.pearsonkt.com/IEAFeedback.shtml.
|
