Scantrons not skewing grades

As students across campus recover from midterms, staff members in one University office work to… As students across campus recover from midterms, staff members in one University office work to dispel rumors that malfunctioning Scantron machines have altered some courses’ test scores.

Nancy Reilly, director of the Office of Measurement and Evaluation of Teaching, explained the source of the rumors.

She said the OMET office — which provides Pitt instructors with scoring and analysis of multiple-choice exams — recently updated its scoring machine to iNSIGHT 70, manufactured by the Scantron company.

Reilly said the new machine is more sensitive than the old one — so sensitive that it sometimes picks up answers that have been erased.

“We had a problem from the very beginning with students not erasing answers correctly,” Reilly said. “Let’s say a student wrote the wrong answer to a question, but then erased it. The machine was picking it up as a double mark, thus marking the question wrong.”

Workers installed the new machine during the summer semester, but the OMET office only started using it to grade tests in the fall.

Because there aren’t many tests at the beginning of the year, Reilly said, the problem didn’t come to her staff’s attention until the about halfway through September.

“We called the Scantron company as soon as we started having issues,” she said. “They told us how to play around with the sensitivity settings. After a few minor setbacks and a second phone call to the Scantron company, the machines started working correctly.”

Reilly said staff members at the OMET office fixed the problem quickly and that students should not worry about the machine affecting their grades.

She also said the problem had nothing to do with a change in answer sheet format. New sheets use hexagons instead of circles.

Steve Zumbrum, a psychology major, said he heard rumors that machine errors could negatively affect his grades.

“I was definitely a little apprehensive taking this test,” Zumbrum said shortly after completing a midterm Monday. “My professor kept stressing that if we change an answer, we have to make sure to erase it completely. A few times, I had to switch answers, and I couldn’t stop thinking about whether it would get picked up and lower my score.”

In spite of concerns like this, Reilly said she did not think the machine problems could negatively affect students’ grades.

“Our machines are as accurate as they can possibly be,” she said. “Of course, no machine is 100 percent accurate, so it‘s always up to the instructor and the student to make sure tests are graded properly.”

She said the OMET office notified all instructors who scanned their tests with the new machine to recheck their reports to see if it affected any students’ exams.

On the grading report given to instructors, a pound sign shows up for any question the machine reads as double-marked.

The pound sign, Reilly said, should let the instructor know that any answer double-marked is not necessarily wrong, but rather that the machine had a problem grading that question.

“If we ever come across a problem, we leave it up to the professor and the student to make a human judgment,” Reilly said. “A machine can’t do that. A machine is just a machine.”

The OMET office sent a letter to various deans in the academic community, explaining and apologizing for a recent delay in the office’s ability to return tests quickly.

The office is usually able to return tests the next day, but two weeks ago, it was only able to process tests in 48 hours.

Reilly said an influx of midterm tests, not problems with the scanning machine, caused the delay.

Pitt psychology professor Allan Zuckoff said he heard about the potential problems with the new machines in late September, after he had given his first exam. When he received his students’ grades for that exam, he said he noticed something strange about the number of questions recorded as double-marked.

“Because OMET provides us with raw data, where you can actually look at exactly which questions individual students got wrong, I was able to see there were an unusual number of questions where students had apparently marked two answers,” Zuckoff said.

Zuckoff did not have much trouble fixing the problem, though, and said he did not think it could have lowered any of his students’ scores.

“It was only a minor inconvenience having to go back through the tests,” he said. “If I had not had all the raw data OMET gives back to instructors and I actually had to go back through the Scantron sheets, it would have been an enormous amount of work. The way OMET provides instructors with the raw data made it easy.”

Zuckoff said his job would be much harder without the services OMET provides.

“This is my first semester teaching full-time in the psychology department,” he said. “I can‘t imagine having to grade my tests by hand. I have over 350 students in my Introduction to Psychology class. If I were to grade by hand, I would probably need four, five, a half-dozen TAs.”