Latest of many from California through Nevada to Minnesota and Georgia…

Georgia fiasco points to craziness of ‘bottom line’ based on once-a-year secret high-stakes tests

By George N. Schmidt

The first of the major testing scandals of the new testing cycle broke in mid-April when Georgia officials finally admitted that their testing program was so botched that it was useless in most grades and Nevada had to confront a major waste of money on high-stakes tests.

What wasn’t said was that the Georgia problem is probably true of half the states present using high stakes commercial tests — including Illinois — but that most states aren’t being examined as carefully as Georgia is.

According to an April 16 Education Week story (“Georgia suspends Testing Plans in Key Grades”):

“ Just as students were gearing up to take out their No. 2 pencils, Georgia officials decided to suspend most state tests in grades 1-8 this year after discovering that some 270 test questions were available on an Internet site for students, parents, and teachers.”

All elementary and middle school students were to have taken the Criterion-Referenced Competency Tests starting the week of April 16. Instead, the state gave the tests only in grades 4, 6, and 8 in reading, mathematics, and English/language arts. State officials said that those were the only grades and subjects for which there is no evidence that students and teachers had access to secure test items within a state “item bank” that districts use for practice.

“How could I ask students and teachers to go through [testing], knowing that the results would come back and we wouldn’t be able to use them?” Kathy Cox, the state superintendent of schools, told Education Week. “I just can’t see us putting people through the motions.”

“ The testing glitch is the latest in a string of such mishaps across the country that educators fear will only get worse, as states gear up to test hundreds of thousands of students under the “No Child Left Behind” Act of 2001,” the Educaton Week report continued.

Georgia was planning to use the results from this year’s tests for the first time to grade schools and reward or intervene in them based on their performance. Student-promotion decisions in grades 3, 5, and 8 also are to be based on test scores beginning with the spring 2004 administration of the exams, if Georgia’s plans go ahead as scheduled. State legislators are now debating whether to postpone the school accountability system for one year.

“ The state already had been scrambling to keep its tests on schedule,” Education Week continued. “On Feb. 21, the state administrative-services department rescinded a six-year, $84 million contract with Riverside Publishing, a division of the Boston-based Houghton Mifflin Co., after a competitor protested that all bidders had not received fair and equal treatment.”

Riverside is the same company that supplies Chicago with the controversial Iowa Test of Basic Skills, still used in the city’s elementary schools.

The Georgia department ordered a rebidding of the contract.

Less than a week later, on Feb. 26, members of the testing staff in the state education department were reviewing the printed test forms when they spotted a small number of items that also appeared in the nonsecure portions of the Criterion-Referenced Competency Tests’ item bank, Educaton Week reported. “On March 10, Riverside provided a preliminary report that led the education department to conclude the extent of item exposure was ‘unacceptably high in most grade and content areas.’”

On March 27, Ms. Cox and Wanda Barrs, the chairwoman of the state board of education, announced plans to sign an “emergency procurement contract” with Riverside, for $4.7 million, to enable a scaled-back version of this spring’s assessment to proceed as planned.

The debacle has led to a flurry of allegations about who’s at fault. Collin Earnst, the director of media relations for Houghton Mifflin, said Riverside was given the test questions by the previous contractor, Measured Progress, based in Dover, N.H., Educaton Week continued.

“So the questions were there prior to us being given the contract and prior to Kathy Cox’s administration,” he said. “If there were any problems with these items, if they were released on any practice tests prior to us getting the questions, that was obviously out of our control. It happened before we ever saw them.”

Stuart R. Kahl, the president of Measured Progress, retorted that “it’s premature for that statement.”

Measured Progress is being paid by the Georgia education department to review the remaining tests in grades 4, 6, and 8 and ensure that all the items are secure. The company will also keep the online item bank, being used by tens of thousands of teachers and students daily, operating through the end of this school year.

Measured Progress created the bank, which contains three levels of questions: Web-based practice tests that parents and students can access from home; nonsecure items for school use; and a secure item bank for high-stakes, statewide exams.

Company officials say they prepared a database of all test items available for the 2003 CRCT that coded those used for the practice tests or teacher-item banking system, and thus were not available to be used on the secure statewide exams. They also identified questions that had appeared on previous years’ editions of the tests and were subsequently released to the public.

“To be quite honest, I don’t know if we’ll ever know exactly what happened,” said Ms. Cox, a Republican who was elected last November and took office Jan. 13.

The imbroglio also raises questions about the resources and capacity within the Georgia education department to oversee state testing contracts.

In 2002, Georgia officials decided not to pay Harcourt Educational Measurement for that year’s administration of the Stanford Achievement Test-9th Edition, which it had been giving in addition to the CRCT, after concluding that the results were no longer worth trying to salvage because of scoring problems.

The state board of education already had decided not to renew its contract with Harcourt. In 2001, the San Antonio-based company had been late in delivering the scores from that spring’s administration of the test, prompting state officials to insert penalty clauses into the 2002 contract.

Proponents of high-stakes testing are regularly quoted in the media defending the testing programs and the one-size-fits-all approach of “No Child Left Behind” while critics are left out of the debate.

Education Week, for example offered the following in its Georgia report: “All of us have got to do a better job of staffing the state-level testing departments,” said Lisa Graham Keegan, a former Arizona state superintendent who is now the chief executive officer of the Washington-based Education Leaders Council. The ELC has held a number of meetings between representatives of states and test publishers.

“This stuff may have gone on before,” Ms. Keegan said, “but gone unnoticed because we just didn’t pay close enough attention and the stakes weren’t as high.”

“Rather than pointing fingers,” she added, “I think all of us need to decide this is a much bigger part of what our life is going to be like, particularly at the state level.”

Statements regarding massive waste that would once have gotten educators fired for misuse of public funds are also ignored when high-stakes testing is concerned.

“Ms. Cox said she does not think Georgia has lost money as a result of the current testing problem,” Education Week reported. “About $6.5 million of Riverside’s six-year contract was paid to the company through January of this year. The state paid the company another $1.1 million in February. The state superintendent said she is now exploring whether more test development, administration, and scoring could be done in-house, working with local universities that might be more responsive to state needs.”

“States have got to start looking to themselves,” she argued. “I think states are going to have to look to something different than these three or four testing companies.”

As the examples accumulate, it becomes interesting to see how long proponents of high-stakes testing will be able to sustain their claims.

Last September, the press reported problems in Nevada:

“A divided Nevada state school board agreed last week to a settlement that requires the San Antonio-based company [Harcourt Educational Systems] to pay penalties totaling $425,000 because of a mistake that threw off the scores of nearly 31,000 high school students who took the state’s high school-exit exam in mathematics last spring,” Education Week reported in September. “Harcourt also has agreed to change its quality-control procedures to prevent similar errors and to pick up any summer school costs incurred by the 736 10th and 11th graders who were mistakenly told they had failed the test.”

“It’s very unfortunate,” Jack W. McLaughlin, Nevada’s superintendent of public instruction, was quoted saying of the scoring error. “On the other hand, ... they’ve really gone the extra mile to ensure that it won’t happen again...”

“In an Aug. 26 statement about the settlement in Nevada, Harcourt President Dean Nafziger said the payment to the state would come in the form of “cash, teacher instruction, library books, and materials.” Nevada’s testing director, Paul M. La Marca, said that $275,000 of the penalty would be in the form of cash discounts on the contract.”

Another example: The costly $12 million settlement of the Minnesota case against NCS Pearson for its mis-scoring of the 2001 high school graduation test in that state has also been cited as an example of the problems that are arising.

In the context of recent academic and media events in Chicago, pressure will continue to build on the media to question whether the elusive goal of a national set of tests, as mandated by “No Child Left Behind” can be expected to do the jobs that are being assigned to such tests. Despite the agenda of the Education Writers Association (which guaranteed that only pro-test experts are available to reporters) criticism grows.

Despite claims from test publishers and the proponents of high stakes testing, the evidence is piling up that such a policy is simply fantasy and should be dropped.

The mistake occurred because of an improperly formatted computer file and took place during the process used to “equate” the results from the April 2002 administration of the math test to those from tests given previously, according to Mr. La Marca.

Results Deemed Too Late

Georgia education officials suspected a problem after seeing sharp spikes and drop-offs in students’ average scores in certain subjects from 2001 and 2002, Ms. Henson said.

Harcourt officials told the state that the company might be able to correct the problem, given more time. But after consulting with an independent panel of testing experts, state board members concluded that even if valid results were eventually made available, they would be too late to be useful, Ms. Henson said. Some districts use the results to make decisions about student placements and merit pay for employees, and the state wanted to use the results to help see how Georgia students stack up against their peers nationwide.

But the state also administers what are called criterion-referenced tests—ones, in this instance, that are aligned with its academic standards—and it is those that are used in making certain major decisions under the state’s accountability system, such as which schools are placed on the list of those needing improvement.

Harcourt officials said the raw scores from the 2002 Georgia tests were correct, but that the problem arose during the equating process, which is designed to allow for scores to be compared from one year to the next.

© 2003 Editorial Projects in Education Vol. 22, number 01, page 24