number crunching

Some analysis left undone in data-driven education department

P.S. 199 in the South Bronx, one of the city's top-rated elementary schools in recent years. A high rate of its former students' test scores plummeted once they moved onto middle school.

The Department of Education crunches state test scores in dozens of ways to measure the performance of schools, principals, teachers, and students. But it does not perform a statistical analysis that can reveal whether an elementary school’s graduates have received test scores that far outstrip their actual skills.

Researchers say it would be relatively easy for the department to calculate “swing rates” to find the proportion of students from each school whose scores rise or fall by a statistically unlikely margin when they move to another school. Such an analysis could take some of the burden off of individual educators to report suspicions of cheating.

The city used to conduct swing rate analysis prior to the Bloomberg administration, according to a former testing official, and the state is poised to launch the measure as part of an overhaul of its own approach to test security.

But department officials say the analysis would be too onerous. They also say that they never launch investigations into cheating based on data anomalies alone. Instead, they say they will dispatch investigators only when they receive formal allegations of test improprieties.

The policy means that some top-rated schools whose students’ scores plummet at far higher than the average rate never have their testing practices scrutinized.

For all of the criticism of state tests as being arbitrary and imperfect measures of student performance, they are remarkably stable. In 2011, students who saw their scores fall by more than two standard deviations from the previous year made up just 0.6 percent of the sixth grade test-taking population in English, and 0.4 percent in math. That degree of decline is highly improbable under normal circumstances and is more likely to reflect externalities than real changes in academic proficiency.If one student’s test score plummets to that degree, it might be reasonable to conclude that he had a bad day when the second test was administered. But if an entire cohort of students see their scores plummet, it could be that testing conditions were especially favorable in the first year, maybe illicitly.

Some city schools have posted swing rates many times the 0.6 percent average, according to the New York Times. At the two elementary schools with the highest scores on the city’s 2011 progress reports, P.S. 257 and P.S. 31 in Brooklyn, the swing rates that year were 9 percent and 1.8 percent, respectively.

Neither school raised eyebrows at the department until a whistleblower at a middle school that received their students registered an official complaint about wide discrepancies between the students’ test scores and their actual skills. The allegation triggered investigations at both schools.

But the Brooklyn schools’ swing rates were not even the highest in the city that year. About 30 percent of the 34 students who graduated from P.S. 199 in the South Bronx and went on to a middle school up the road, I.S. 303, saw their test scores drop by more than two standard deviations. P.S. 199 had the 16th highest progress report score that year.

At I.S. 232, a nearby middle school that absorbed 29 students from P.S. 199, the swing rate was more in line with the city average.

But teachers at I.S. 303 said that the high test scores did not correlate to the basic English and math skills that many of the incoming P.S. 199 students demonstrated early on in sixth grade.

“You don’t just forget everything,” said one math teacher. “It just baffled me that they somehow got 3’s and 4’s” in fifth grade.

Multiple teachers at the school said students’ behavior during the sixth-grade state tests suggested that the students had received help during tests before.

Some of the students from P.S. 199 grew frustrated during the tests when I.S. 303 teachers did not tell them the answers to questions that stumped them, the teachers said. The teachers agreed to speak on the condition of anonymity because their principal had not given them permission to speak to reporters.

But even after noticing the score discrepancy, no one at I.S. 303 went to the department that year. It’s a common decision, according to educators from across the city who say allegations usually bring scrutiny first to the people who filed them, sometimes exposing them as whistleblowers to their colleagues.

An investigation at P.S. 199 is now open, a department spokeswoman said. Education officials contacted investigators after GothamSchools asked repeatedly about the school’s anomalous scores.

The department’s Office of Special Investigations is handling the case now after it was referred by the city’s Office of the Special Commissioner of Investigation, a SCI spokeswoman said.

Department officials said their concerns were not based solely on the swing rate. Data points such as swing rates alone are not enough to trigger investigations.

Many of the 37 schools to which the department dispatched testing monitors last year had seen their scores increase in unusual ways. But all but four of them had also been the subject of formal allegations.

In fact, the department does not even calculate schools’ swing rates as part of its regular analysis of schools’ performance. When the accountability division runs the numbers that feed into schools’ annual progress reports, which are based largely on students’ year-to-year growth, it does not aggregate results by sending schools.

Department officials say generating schools’ swing rates is a complicated endeavor. After learning about the score discrepancies at P.S. 199, GothamSchools requested swing data for other top-rated schools. But department officials said for months that running those numbers would be too difficult.

“We have provided you with a detailed analysis of P.S. 199,” spokeswoman Marge Feinberg wrote in an email this summer. “For the other schools, it would take a great deal of time.”

Researchers who have worked with Department of Education data before disputed that claim.

“It would be quite easy to do. Just about anyone with a computer and a basic knowledge of statistics could run these checks,” said Sean Corcoran, a New York University professor who has studied the city’s test score data. “This is something the DOE should do on a regular basis.”

A former testing director in New York City said that under his watch the city used data anomalies to trigger investigations.

“We routinely would look at the change in scores in schools from one year to the next,” said Bob Tobias, the Board of Education’s longtime testing chief who retired in 2002. Schools that showed erratic spikes, Tobias said, “would get a little more scrutiny.”

There are signs that the state might start conducting this type of analysis on its own. One charge given to the brand-new test security chief Tina Sciocchetti, at the State Education Department is to set guidelines for pursuing investigations using data methods that look suspicious test score patterns.

“I think those sorts of statistical analyses are simply a red flag and it’s absolutely true that additional investigation is necessary,” said Sciocchetti.

Weekend Reads

Need classroom decor inspiration? These educators have got you covered.

This school year, students will spend about 1,000 hours in school —making their classrooms a huge part of their learning experience.

We’re recognizing educators who’ve poured on the pizazz to make students feel welcome. From a 9th-grade “forensics lab” decked out in caution tape to a classroom stage complete with lights to get first graders pumped about public speaking, these crafty teachers have gone above and beyond to create great spaces.

Got a classroom of your own to show off? Know someone that should be on this list? Let us know!

Jaclyn Flores, First Grade Dual Language, Rochester, New York
“Having a classroom that is bright, cheerful, organized and inviting allows my students to feel pride in their classroom as well as feel welcome. My students look forward to standing on the stage to share or sitting on special chairs to dive into their learning. This space is a safe place for my students and we take pride in what it has become.”

Jasmine, Pre-K, Las Vegas, Nevada
“My classroom environment helps my students because providing calming colors and a home-like space makes them feel more comfortable in the classroom and ready to learn as first-time students!”

 

Oneika Osborne, 10th Grade Reading, Miami Southridge Senior High School, Miami, Florida
“My classroom environment invites all of my students to constantly be in a state of celebration and self-empowerment at all points of the learning process. With inspirational quotes, culturally relevant images, and an explosion of color, my classroom sets the tone for the day every single day as soon as we walk in. It is one of optimism, power, and of course glitter.”

Kristen Poindexter, Kindergarten, Spring Mill Elementary School, Indianapolis, Indiana
“I try very hard to make my classroom a place where memorable experiences happen. I use songs, finger plays, movement, and interactive activities to help cement concepts in their minds. It makes my teacher heart so happy when past students walk by my classroom and start their sentence with, “Remember when we…?”. We recently transformed our classroom into a Mad Science Lab where we investigated more about our 5 Senses.”

 

Brittany, 9th Grade Biology, Dallas, Texas
“I love my classroom environment because I teach Biology, it’s easy to relate every topic back to Forensics and real-life investigations! Mystery always gets the students going!”

 

Ms. Heaton, First Grade, Westampton, New Jersey
“As an educator, it is my goal to create a classroom environment that is positive and welcoming for students. I wanted to create a learning environment where students feel comfortable and in return stimulates student learning. A classroom is a second home for students so I wanted to ensure that the space was bright, friendly, and organized for the students to be able to use each and every day.”

D’Essence Grant, 8th Grade ELA, KIPP Houston, Houston, Texas
“Intentionally decorating my classroom was my first act of showing my students I care about them. I pride myself on building relationships with my students and them knowing I care about them inside and outside of the classroom. Taking the time to make the classroom meaningful and creative as well building a safe place for our community helps establish an effective classroom setting.”

 

Jayme Wiertzema, Elementary Art, Worthington, Minnesota
“I’m looking forward to having a CLASSROOM this year. The past two years I have taught from a cart and this year my amazing school district allowed me to have a classroom in our school that is busting at the seams! I’m so excited to use my classroom environment to inspire creativity in my students, get to know them and learn from their amazing imaginations in art class!”

 

Melissa Vecchio, 4th Grade, Queens, New York
“Since so much of a student’s time is spent inside their classroom, the environment should be neat, organized, easy to move around in but most of all positive. I love to use a theme to reinforce great behavior. I always give the students a choice in helping to design bulletin boards and desk arrangements. When they are involved they take pride in the classroom, and enjoy being there.”

moving forward

After Confederate flag dispute at Colorado football game, schools pledge to bring students together

PHOTO: Marc Piscotty
Manual High students.

Acknowledging “we may never have a conclusive picture of what happened,” two Colorado school districts sought to move past a controversy over whether a Confederate flag was displayed at a football game and open a conversation between the two school communities.

The principal of Manual High, Nick Dawkins, wrote in a community letter over the weekend that the visiting Weld Central High School team “displayed a Confederate flag during the first quarter of the (Friday night) game, offending many members of the Manual community.”

Officials from Denver Public Schools and Weld County School District Re-3J released a joint letter Tuesday saying that based “on what we have learned to date, however, the Weld Central team did not display the Confederate flag.” At the same time, it said, multiple Manual eyewitnesses “reported seeing spectators who attempted to bring a Confederate flag into the game and clothing with flag images.”

Going forward, students from the two schools — one rural and one urban — will participate in a student leadership exchange that has student leaders visit each other’s schools and communities to “share ideas and perspectives,” the letter says.

“At a time in our country when so many are divided, we want our students instead to come together, share ideas and learn together,” says the letter, which is signed by the principals of both schools and the superintendents of both school districts.

The alleged incident took place at a time when issues of race, social injustice, politics and sports are colliding in the United States, making for tough conversations, including in classrooms.

Weld Central’s mascot is a Rebel. Manual, whose mascot is the Thunderbolts, is located in one of Denver’s historically African-American neighborhoods.

Dawkins in his initial community letter also said “the tension created by the flag led to conflict on and off the playing field,” and that three Manual players were injured, including one who went to the hospital with a leg injury. He also said some Manual players reported that Weld Central players “taunted them with racial slurs.”

Weld Central officials vehemently denied that their team displayed the flag. In addition, they said in their own community letter they had “no evidence at this point that any of our student athletes displayed racially motivated inappropriate behavior.”

They said district officials “do not condone any form of racism,” including the Confederate flag.

Weld Central fans told the Greeley Tribune that they didn’t see any Confederate flag.

Read the full text below.