clock menu more-arrow no yes

Filed under:

Extraordinary gains, little investigation

A USA Today analysis of state test scores raises questions about extraordinary gains at nearly 70 schools between 2003 and 2009, but Colorado education officials do little to investigate such steep increases.

That may change as the state prepares to link growth in test scores to teacher and principal evaluations.

For example, third-graders at Aragon Elementary School in Fountain-Fort Carson School District 8 scored in the 31st percentile on the state math exam in 2005, meaning their test scores were on par with 31 percent of Colorado third-graders. A year later, in 2006, Aragon’s fourth-graders achieved at the 97th percentile in math.

The following year? Aragon fifth-graders had dropped to the 47th percentile in math in 2007.

Such fluctuations may be the result of top-notch teaching or they may raise suspicions of cheating – but state officials make comparatively little effort to find out. Colorado, unlike some other states whose test scores were part of the USA Today analysis, does not pay for the standard cheating detection software available as part of its $83 million contract with testing company CTB/McGraw-Hill for years 2007-2011.

And while Jo O’Brien, associate commissioner for accountability at the Colorado Department of Education, said the state conducts its own analysis of statistical variations in test results, it did not flag the changes at Aragon.

Nor did it detect other examples of extraordinary gains found in the USA Today analysis:

  • Third-graders at Trails West Elementary in the Cherry Creek School District achieved at the 30th percentile on the 2008 reading test. A year later, fourth-graders at Trails West achieved at the 76th percentile in reading.
  • At Bradley Elementary in Denver Public Schools, fourth-graders taking the state exams, known as CSAPs, in math in 2006 scored at the 11th percentile. A year later, fifth-graders at Bradley achieved at the 73rd percentile on state math tests.
  • Fourth-graders at Longfellow Elementary in Colorado Springs District 11 achieved at the 27th percentile in CSAP reading in 2007. In 2008, fifth-graders at Longfellow were reading at the 94th percentile.

Testing experts consulted by USA Today on the data analysis project say such dramatic changes certainly are not proof of cheating. But they may merit investigation, if only to figure out how such gains occurred so other educators can take note.

O’Brien said the state relies on districts to monitor their schools, although she acknowledged it may not always be in a district’s best interests to zealously delve into spikes in test scores.

“It’s an unusual situation that may change as we administer a new test,” she said, referring to the state’s move to a new assessment system, expected in 2014. “But for right now, the district investigates an incident and lets us know how it occurred.”

What the analysis revealed about Colorado test anomalies

Altogether, the USA Today analysis of reading and math scores found 69 schools where students moving from one grade to the next posted dramatic growth, or jumps greater than 99 percent of their peers in the state. Seven are charter schools. In the language of statisticians, that means those schools posted gains of at least three standard deviations from one year to the next. Only schools with at least 20 test-takers per grade level were included in the analysis.

Colorado students are now taking CSAPs.
Colorado students are now taking CSAPs.

In addition, 14 schools saw similarly dramatic declines in test scores on the Colorado Student Assessment Program during the eight years studied. Frequently, a soaring increase in scores was followed by a steep decline.

That was the case with Aragon Elementary, the school in Fountain-Fort Carson, where a group of students ranking in the bottom third on state math tests one year shot to the top of the state the following year. One year later, they had dropped back to the bottom half.

The district did recognize the spike as “a statistical anomaly,” said David Roudebush, the district’s assistant superintendent for academic support services, but he added, “I don’t think any real conclusions can be drawn due to multiple factors that can affect results.”

He had no more specific information, he said, since district and school administrators had changed since then.

In other schools, the pattern was the opposite – a group of students achieved average scores, dropped precipitously one year and then recovered the next.

Consider Bradley Elementary in Denver, where a group of third-graders in 2005 began what appears to be a roller-coaster ride on state reading and math tests:

  • 2005 – Grade 3 – 44th percentile in math, 64th percentile in reading
  • 2006 – Grade 4 – 11th percentile in math, 14th percentile in reading
  • 2007 – Grade 5 – 73rd percentile in math, 58th percentile in reading

Angelo Spataro, who retired as Bradley’s principal in 2007, said he noticed and responded to the fluctuations.

“We knew we had to do some added training in the fourth grade,” he said. “We felt with the training and some movement of staff, we would be able to solve that issue.”

He declined to be more specific about “movement of staff.” Asked if there any hint of cheating, Spataro said “Never.”

“During CSAP, I was in those classrooms myself,” he said. “Ultimately, it comes out. Some kid will go home and say, ‘Mr. So-and-so did this during CSAP’ and they find out. Even if you wanted to, it’s pretty hard to cheat.”

He added: “If I ever found anybody doing it, they would have been gone, I can tell you that.”

Denver is one of a few Colorado districts which already pays some teachers more based on improvements in state test scores. For example, teachers enrolled in ProComp, the district’s performance pay plan, could earn a $2,400 bonus in 2010-11 if their students’ CSAP test scores exceeded district expectations.

Of the 69 schools identified as showing extraordinary gains from one year to the next, 29 percent are in DPS. District officials said they do not conduct systemic analysis of scores, such as what was done by USA Today, for potential cheating.

They do investigate incidents brought to their attention by staff in schools or by district instructional leaders poring over results, said Connie Casson, the district’s executive director of accountability, research and evaluation.

“We’ve had largely small-scale cheating incidents,” she and other DPS staff said, citing examples such as an elementary teacher suspended ten days for posting a CSAP question online after the test was over.

DPS, in the midst of a teacher effectiveness initiative funded by the Bill & Melinda Gates Foundation, is focusing more on questions of the value teachers add to test scores, she said, than possible cheating.

“I’d say it’s driven by not feeling a need” to scrutinize further, she said. “If we did feel the need, budgets are always tight but we can afford what we prioritize.”

A handful of known cheating incidents, few repercussions

Colorado Department of Education officials do not formally track instances of cheating on the CSAP, O’ Brien said, and the state has no authority to discipline an educator or yank a license if an incident is confirmed. That’s up to each of the state’s 178 school districts.

State officials typically learn about possible cheating when they’re contacted by a district wanting to invalidate one or more tests, which can occur for reasons as varied as cheating by a teacher or student or mistakes in proctoring, such as giving a 7th-grade exam to 8th-grade students.

Between 2005 and 2010, the state lists 68 instances when at least 16 tests in a single school were invalidated through misadministration. But the reasons behind the invalidations aren’t reported – they’re all lumped together under the “misadministration” label.

That leaves anecdotal reports and media coverage as the best sources of cheating data. Among those that are known:

  • More than 150 eighth-grade science tests were invalidated at Carson Middle School in Fountain-Fort Carson school District 8 after a teacher used test questions from prior years as a study guide for the spring 2006 exam. The teacher, who is not named in media reports, resigned after an investigation.
  • Nearly 300 writing tests were thrown out at Silver Hills Middle School in the Adams 12 Five Star District after Principal Tracy Webber peeked at questions from the upcoming test and then talked to her teachers about them. Webber was demoted to assistant principal but continues to work at another district school, Westlake Middle. Her principal’s license remains active.
  • A first-year teacher at an Aurora charter school, AXL Academy, reportedly flipped open students’ test booklets after they had closed them and pointed out incorrect answers. A Ch. 7 investigation found conflicting accounts by the school and teacher – he claims he did as he was told, the school immediately contacted the district – and Aurora Public Schools officials invalidated 35 tests in spring 2010. The teacher no longer works at the school.

State officials have hired an outside firm to look into allegations of cheating in only one case, that of Cesar Chavez K-8 Academy in Pueblo, after the Pueblo schools superintendent publicly urged former Education Commissioner Dwight Jones to investigate. A $25,000 audit by Caveon Security of Utah found “extremely high” rates of special accommodations – more than half the students one year received extra time on the exams – but no evidence of answer sheet tampering or test coaching.

“Colorado has not had a lot of cheating scandals.” O’Brien said. “I’m not saying we don’t have something that goes on without us knowing about it … but our scores are so flat and so unremarkable.”

Last year’s passage of the Educator Effectiveness Law, which will base at least 50 percent of teacher and principal evaluations on academic growth measures, will raise the stakes, she said. State officials are talking about adding cheating software to the state testing contract and creating penalties, such as loss of license, for educators found guilty of cheating.

“We ourselves in the Department of Education don’t consider cheating to be in the top three problems we’re facing.” she said. “But we are thinking about how those policies will need to change because the stakes are higher.”

The COVID-19 outbreak is changing our daily reality

Chalkbeat is a nonprofit newsroom dedicated to providing the information families and educators need, but this kind of work isn't possible without your help.