Skip to content Skip to navigation

Internet Grading Service Reduces Tedium For Teachers, Students

Tomorrow's Teaching and Learning

Message Number: 


The posting below is from the Stanford Report on-line edition of May

25, 2000. It looks at a new Internet grading service currently being

used in college-level logic courses, but which has the potential for

use in other courses across a wide range of disciplines.


Rick Reis

UP NEXT: The Case for an Anthropological Pedagogy

Tomorrow's Teaching and Learning


----------------- 1,657 words ---------------


By Kathleen O'Toole

Stanford Report

On-Line edition, May 25, 2000

After dinner when he needs a pick-me-up, John Etchemendy often sneaks

off to watch students submit their homework. Tapping into the

Internet from his home computer, the professor of philosophy and

author of logic textbooks and software reaches one of two Sun

workstations named Grade Grinder. He can watch as the wannabe

historians and lawyers taking logic from Professor John Justice at

Randolph-Macon Woman's College and the tech majors taking logic from

Professor Selmer Bringsjord at Rensselaer Polytechnic Institute (RPI)

submit answers to their problem sets.

These students and others taking introductory logic courses around

the country use the Internet to interact with Grade Grinder, a

robotic teaching assistant that doesn't give them answers to

problems, but gives them hints and reminders of principles they have

previously encountered. The robot's advice is personalized to address

the specific shortcomings of the last answer each student has

submitted, and it is delivered by e-mail in seconds. That compares to

the week or more that is typical of feedback from a human grader.

Grade Grinder is an Internet grading service that is provided with

purchase of a new textbook, Language Proof and Logic, and four pieces

of software. Etchemendy co-authored the package with the late Jon

Barwise of the University of Indiana and formerly of Stanford, and a

team of researchers at Stanford's Center for the Study of Language

and Information (CSLI) and Indiana's Visual Inference Laboratory.

Co-published last fall by CSLI Publications and a commercial textbook

house, Seven Bridges Press, the textbook/software package for

introductory logic is priced at $43.95, slightly less than most logic

textbooks alone. There is one catch: Because the purchaser is buying

lifetime tutoring help from Grade Grinder, each educational package

comes with a unique registration number, and the student cannot

resell that ID to another student. But the textbook covers more

ground than most introductory courses, so the student can continue to

receive tutoring from Grade Grinder years after taking a formal


"We are grading about half of the students' exercises live at a

central point, so I get to watch their progress, and that is one of

the most rewarding things," says Etchemendy, who has authored other

textbooks and software but without this interactive component. (He

also chaired the university's Commission on Technology in Teaching

and Learning, which funded proposals for developing learning

technologies on campus.) "First you'll see a student submit an answer

that is wildly incorrect, then get some feedback from Grade Grinder

and keep resubmitting until it's correct. As a textbook author, you

don't usually get that chance to see how students learn from it,

except your own."

When he finds a pattern of trouble, Etchemendy goes into Grade

Grinder's Java-language software and tinkers. For example, he noticed

a pattern last fall of students at California State

University-Northridge having difficulties with word problems that

took the form "neither . . . nor." Many of the students there

acquired English as their second language, and the grammatical

formulation confused them. Seeing the overall pattern allowed

Etchemendy to improve Grade Grinder's advice both to the students and

their professor.


Etchemendy got into this business by accident, but it has turned out

to fit well with his research on reasoning systems that use multiple

forms of representation. He began building teaching software in the

1980s out of frustration with some of the mistakes students made in

logic courses. Formal logic requires translating English sentences

into a language that lacks the ambiguities and subtleties of natural

languages. "As an example," he says, "I use the old 'Saturday Night

Live' joke that goes, 'Every five minutes, a man is mugged in New

York City. We're going to interview him tonight.'"

The joke is based on what logicians call a "quantifier scope

ambiguity." "People don't even recognize that the English sentence

has this ambiguity because we understand it correctly in context. If

I said to you, 'Every five minutes, a man from the L.A. Times has

been calling,' you would immediately interpret the sentence

differently than you interpreted the one about the man being mugged.

But when students try to learn an unambiguous language, they have

problems because they lack this understanding of English ambiguities."

Grade Grinder uses sophisticated computer algorithms to check such

things as the logical equivalence of the student's sentence with

expected answers, and its truth or falsity in a large number of

contexts. It can check files created using programs packaged with the

textbook -- Tarski's World, Fitch and Boole. It performs this check

much faster and with fewer errors than even an expert human logician,

says Dave Barker-Plummer, a logician and senior research scientist at


Grade Grinder most likely never will replace a human instructor,

Barker-Plummer and Etchemendy say, but it can free instructors and

students of their most tedious teaching and learning tasks.

Instructors still grade about half of the homework. "This isn't

classical distance education, because we think that no amount of

technology can replace an instructor's interactions with students

when they are trying to understand deeper conceptual issues,"

Barker-Plummer says.

The development team "stumbled on this minimally invasive approach,"

Etchemendy says, and "we also didn't see in advance that part of the

value of Grade Grinder would be allowing us to centrally analyze

common mistakes."

About a dozen of the professors who used Language Proof and Logic

this year were polled by e-mail for this article. They gave it high

marks, some saying it was a "revolutionary" development in the use of

technology in the classroom. All who responded said the grading

service made teaching logic easier on them and learning it easier on

their students.

"The automated grader worked flawlessly and freed my TA to help with

substantive issues in logic, rather than mechanical checking of

proofs. It's like having another TA -- for free," said Bringsjord,

director of the Minds and Machines Lab at RPI. "I believe this is the

future. It's the start of tutoring agents that handle parts of

teaching traditionally done by humans. My students also loved it."

Justice of the Department of Philosophy at Randolph-Macon agreed,

adding, "With the Grade Grinder always available on the Internet, the

student can know within seconds if she is doing the work correctly.

What's more, she can correct her work before she asks the Grade

Grinder to forward a report to the instructor. This instant feedback

makes learning logic quicker and less frustrating for the student."

Not all the students in the classes of Professor Tom Burke at the

University of South Carolina were pleased, however. "Some students

hate the software precisely because one cannot indulge in shortcuts

or sloppiness that pencil-and-paper homework easily permits," he said.

Burke said his only frustration with the program was "handling the

massive amount of information I get via e-mail on students' progress

through the homework. I wish I could get this information in a form

that could be easily imported into a database."

The CSLI team is addressing that issue. "We're hoping to have web

access to all the data for a particular instructor's course in a

compact form by the end of summer," Etchemendy says.


Instructors in the past also have been concerned that automated

assessment systems might increase opportunities for student cheating,

Barker-Plummer says. To address that concern, all Grade Grinder

homework has a time stamp that makes it highly unlikely for students

to share their homework. "We've made it difficult enough that any

student who is savvy enough to circumvent the system will probably

find it easier and less time consuming to do the work."

The Grade Grinder software could be adapted for use by other courses,

Etchemendy and Barker-Plummer believe. "Any type of computer file

that a computer can do something sensible to and give useful feedback

could use the Grade Grinder framework," Etchemendy says. "All you

have to do is write a single grading module for that type of file.

Down the road, we could supply chemistry professors with a software

framework in which to plug in their chemistry module, for example."

The grader would be especially useful, Barker-Plummer says, in

scientific courses where the range of possible answers is so large

that it is difficult to tell if a student has found a correct one. "I

can imagine situations in chemistry where you are asked to write down

a formula for a molecule, and there may be hundreds of ways to do it."


The logic course package was sought after by commercial publishers,

but Etchemendy says he felt it would be "unfair if not immoral" to

turn over the rights to a product that was developed with university

resources. "Fortunately, CSLI has a press that publishes academic

books, and we were able to have them publish it, through an

arrangement with another press to do textbook marketing and

distribution." About $2 from the sale of each textbook/software

package is set aside to cover the expected operation and maintenance

costs of the grader.

Dikran Karagueuzian, who directs CSLI Publications, says he views

Language Proof and Logic as a breakthrough in educational technology.

An academic press, CSLI Publications publishes 35 to 40 titles a

year, many of them books in the cognitive sciences and all of which

are reviewed by experts in their fields before acceptance (see Karagueuzian has rejected

other courseware proposals because referees found their quality

spotty, he says. "A lot of academic software is like a homemade

motorcycle. You can't take it on the freeway. The programs can be

used on the campus where they were born, but they can't pass the

campus boundaries.

"In this case, we have a package that doesn't annoy the best students

at the Harvards and Stanfords but which also doesn't alienate

community college students. The authors used lots of concrete

examples that don't go above the users' heads. The ideas sort of

unfurl, and the really nice thing is the instant feedback." SR