First Person

The Trouble With Not Releasing State Test Items

First Rule of Fight Club: Do Not Talk about Fight Club

Second Rule of Fight Club: DO NOT TALK about Fight Club

Has the New York State Education Department watched too many Brad Pitt movies? Okay, that’s a rhetorical question, but one that might be posed to other state education agencies also engaged in the business of high-stakes testing. This week, students in grades 3 through 8 across the state of New York are taking mathematics exams aligned with the Common Core State Standards. Following on the heels of last week’s English Language Arts exams, the math exams also promise to be unusually challenging, reflecting the complex skills and knowledge inscribed in the Common Core standards.

Regardless of broad pronouncements from policymakers and the media about the inherent superiority of the Common Core standards and the assessments designed to measure mastery of them, the truth is that no one really knows whether the standards will lead to higher student achievement, or whether the assessments will be good measures of students’ readiness for college and careers. In New York, although this year’s assessments are the first to be aligned with the Common Core standards, they have a short shelf-life: the state plans to administer the Partnership for Assessment of Readiness for College and Careers assessments in the spring of 2015, if those assessments are ready for prime time by then.

In the meantime, discussions about the content and quality of the assessments are hamstrung by New York’s decision not to release test items to the public. For educators, the issue is quite serious: Disclosure of secure test items by a teacher or school leader is considered a moral offense that can lead to disciplinary action, including loss of certification.

The strongest arguments in favor of keeping test questions and answers private are technical. It is desirable that different forms of a test, including those administered in different years, be scaled in such a way that a given score represents the same level of performance, regardless of the test form or year. Anchor items are used to link different forms of a test and equate them. Modern test theory uses the difficulty of test items, and their ability to differentiate higher and lower performers, as tools to estimate a test-taker’s performance. It’s important for anchor items to have a stable level of difficulty over time; if they become easier or harder over time, their ability to serve as a common anchor across test forms is compromised, as is our confidence that a given test score denotes the same level of performance over time. A change in the difficulty of a test item over time is referred to as item parameter drift.

Item parameter drift can occur due to changes in curriculum, teaching to a test, or practice. But the biggest risk is from the widespread release of test items, whether unintentionally, as in a security breach, or intentionally. If a wide swath of the test-taking population knows test questions and the right answers, the questions will be easier, even if the test-takers are not more capable. It’s for this reason that questions and answers in educational tests frequently aren’t released to the public: disclosing test questions would limit their ability to be reused and to serve as anchor items.

The National Assessment of Educational Progress is a case in point. The No Child Left Behind Act provides that the public shall have access to all assessment instruments used in NAEP, but that the Commissioner of the National Center for Education Statistics, which houses NAEP, may decline to make available test items that are intended for reuse for up to 10 years after their initial use.

Of course, one of the other features of the lovely NCLB law is that it prohibits the federal government from using NAEP to rank or punish individual students, teachers, schools or local education agencies. For this reason, NAEP is a low-stakes test — despite the ways in which pundits jump to draw broad policy inferences from comparisons of NAEP performance over time or across jurisdictions.

But one could argue that disclosure of test questions and answers may be justified when the test is used for high-stakes decisions such as student promotion, or the evaluation of teachers and/or schools. For most such high-stakes decisions, there are winners and losers, and when these decisions are made by agents of the government, the losers have a legitimate interest in whether the decisions were fair. One need look back no further than last week, when New York City announced that, due to a series of errors made by NCS Pearson, several thousand children were incorrectly classified as ineligible for gifted and talented programs.

Or, if you wish, reach back to last year, when the New York State Education Department discarded a series of items in the Grade 8 English Language Arts exam based on a passage involving a talking pineapple. Not too many people rose to defend the test items associated with this fable involving a hare and a pineapple, but Pearson, the firm contracted to develop and administer the exam, did. The choice of both the passage and the items, the company claimed, “was a sound decision in that ‘The Hare and the Pineapple’ and associated items had been field tested in New York State, yielded appropriate statistics for inclusion, and it was aligned to the appropriate NYS Standard.” Vetted by some teachers, too, I reckon. But with all of that, the passage and items were ludicrous.

One item following the passage asked which of the animals in the passage was the wisest: the moose, crow, hare or owl. Pearson claimed that it was unambiguous that the wisest animal was the owl, based on clues in the text. One such clue was that the owl declared that “Pineapples don’t have sleeves,” which, Pearson reported, was a factually accurate statement. So too, to the best of my knowledge, is that owls don’t talk.

High-stakes tests administered by governmental agencies call for a heightened sense of procedural fairness, including the ability to interrogate the tests and how they were constructed, and what counts as a correct response. The point is not so much that bad test items get discarded — although that may be appropriate from time to time — as much as it is that the procedures are subject to scrutiny by those they affect. New York does not have a great recent track record on this. The technical reports on the construction of last year’s state English Language Arts and math tests have not been made public yet, even though we’re in the midst of this year’s testing. And the technical manual for New York’s statewide teacher rankings, a modified version of value-added modeling, was released months ago—before the manual for the tests on which those rankings were based. It’s hard to know how much to trust the growth percentiles or value-added models without more information on the tests themselves.

Moreover, it may be especially important to have open and public discussions about tests that are aligned with the Common Core standards, which are new to educators and the public. The point of these tests, especially in their earliest administrations, is really not “ripping the Band-Aid off,” as New York City Schools Chancellor Dennis Walcott has declared — nor is it to document just how few students will meet the new standards, as a vehicle for supporting one policy reform or another. Rather, it’s to engage educators, policymakers and the public in a conversation about what we want our students to know, and how we can move them toward the desired levels of knowledge and skill.

And one good way to frame that conversation is to ground it in the discussion of particular assessment questions. Might teachers disagree with one another about what the best answer to an assessment question is? If they do, shouldn’t they be talking about it? Will students have an opportunity to discuss why a response is incorrect, what a better response might be, and why? Or will they simply receive a scale score telling them, and their parents, that they are well below grade-level?

Much has been made of the notion that assessments aligned with the Common Core standards are to be “authentic,” with real-world content that parallels what students might experience in adult daily life. (Ideally, something more sophisticated than “If Johnny has $5.63 and is wearing a pair of Nike Free Run+ 3 shoes, how long will it take him to run to the 7-Eleven to buy a delicious Coca-Cola product?”) If the content is indeed authentic, and reflective of what we expect students to know and be able to do as productive adults, we should be discussing that content, not hiding it under a rock.

There is a middle ground between total nondisclosure of test items and answers, and complete disclosure. It’s possible to retain the security of anchor items while releasing items that won’t be used again. But it’s easier to do this when there’s a more extensive bank of assessment items with known properties, and such an item bank for the Common Core does not yet exist. It may not be the most popular conclusion, but perhaps we should be investing more in the development of good assessment items.

First Rule of High-Stakes Assessments: Talk about High-Stakes Assessments

Second Rule of High-Stakes Assessments: TALK about High-Stakes Assessments

This post also appeared on The Hechinger Report’s Eye on Education blog.

First Person

What I learned about the limits of school choice in New York City from a mother whose child uses a wheelchair

PHOTO: Patrick Wall

As a researcher interested in the ways online platforms impact learning and educational decision-making, I’ve been trying to understand how New York City parents get the information to make a crucial decision: where to send their children to school.

So for the past six months, I’ve been asking local parents about the data they used to choose among the system’s 1700 or so schools.

I’ve heard all sorts of stories about the factors parents weigh when picking schools. Beyond the usual considerations like test scores and art programs, they also consider the logistics of commuting from the Bronx to the East Village with two children in tow, whether the school can accommodate parents and children who are still learning English, and how much money the parent-teacher association raises to supplement the school’s budget.

But for some families, the choice process begins and ends with the question: Is the building fully accessible?

The federal Americans with Disabilities Act requires public buildings constructed after 1992 to be fully accessible to people in wheelchairs. However, most New York City public school buildings were constructed prior to that law, and high construction costs have limited the number of new, fully accessible buildings.

As a result, a shocking 83 percent of New York City schools have been found non-compliant with the ADA, according to a two-year federal Department of Justice investigation whose findings the city Department of Education largely disputes. Recently, the city’s Office of Space Management has begun surveying buildings for full accessibility, but more work remains to be done.

One parent’s struggle to find a school suitable for her son, who has a physical disability but no cognitive issues, illustrates what a major role accessibility plays in some families’ decision-making.

Melanie Rivera is the mother of two and a native New Yorker living in Ditmas Park in Brooklyn’s District 22 who shared her story with me — and gave me permission to share it with others. Here is what she told me, in her own words:

My son Gabriel is seven years old. He was born with a condition called arthrogryposis, which affects the development of his joints. His hips, knees, and feet are affected and he has joint contractures, so his legs don’t bend and straighten the way most people’s do. In order to get around, he uses a combination of crutches and a wheelchair.

Before I had my differently-abled son, I was working in a preschool for children with special needs. The kids I worked with had cognitive developmental disabilities.

Despite my professional experience, I was overwhelmed when it was my turn to help my child with different abilities navigate the public school system. I can only imagine the students falling by the wayside because their parents don’t have that background.

When I was completing my son’s kindergarten application, I couldn’t even consider the academics of the school. My main priority was to tour the schools and assess their level of accessibility.

There are only a couple of ADA-accessible schools in my district, and there was no way of indicating on my son’s kindergarten application that he needed one. When we got the admissions results, he was assigned to his zoned school – which is not accessible.

I entered lengthy and extensive mediation to get him into an ADA-accessible school. At that point, I knew I would just have to take what I could get. For families whose children have special needs, “school choice” can ring hollow.

The process of finding any accessible school was a challenge. The DOE website allows families to search for ADA-accessible schools. But the site describes most schools as “partially accessible,” leaving it up to parents to call each school and say, “What do you mean by this?”

When I called the schools and asked, “Are you a barrier-free school?” the staff in the office didn’t know what the term meant. They might reply, “Oh yeah, we have a ramp.” I’d have to press further: “But can you get to the office? Can you get to every floor in the building?” The response was often, “Oh, I don’t know.”

Even the office staff didn’t know. But for my son’s sake, I needed to know.

Gabriel deserves the full range of academic and social experiences. So every day I make sure he’s learning in the least-restrictive environment — from the classroom, to phys ed, to field trips.

I believe the Department of Education also wants to make schools accessible and to place students with different abilities in settings where they’ll flourish, but the current system is not equipped to follow through on those good intentions. While I see gradual changes, I still know that if I don’t find the best placement for my son the system definitely won’t.

At the school level, administrators should know the details of their own school’s accessibility. Teachers should learn to include children with different abilities in their classrooms. Such a commitment means recognizing the value of inclusivity — not viewing accessibility as something ADA says you must do.

Before I had Gabriel, I never thought about accessibility. I never looked at street cutouts or thought about how to enter a store with steps. We’re probably all guilty of perpetuating exclusion at one point or another.

Recognizing that will allow us to change the status quo. It will allow every individual with a physical disability to fully participate in the public school system.

Claire Fontaine is a researcher at Data & Society, a research institute in New York City focused on social, cultural, and ethical issues arising from technological development. Kinjal Dave is a research assistant at Data & Society. You can read more about their project, which seeks to better understand the ways in which diverse New York City parents draw on school performance data, online dashboards, and school review websites when researching schools for their children.

First Person

I covered Tennessee’s ed beat for Chalkbeat. Here’s what I learned.

PHOTO: Marta W. Aldrich
Grace Tatter covers a press conference at the Tennessee State Capitol in 2015.

For three years, I covered the Statehouse for Chalkbeat Tennessee, reporting on how policies from Nashville trickled down into more than 1,800 public schools across the state.

Now I’m starting back to school myself, pursuing graduate studies aimed at helping me to become a better education journalist. I’m taking with me six things I learned on the job about public education in Tennessee.

1. Apathy is often cited as a major problem facing education. That’s not the case in Tennessee.

I heard from hundreds of parents, educators, and students who were passionate about what’s happening — good and bad — inside of schools. I covered crowded school board meetings and regularly scrambled for an open seat at legislative hearings where parents had filled the room after driving since dawn to beat the opening gavel. Not incidentally, those parents usually came from communities with the “worst” schools and the lowest test scores. While many disagreements exist about the best way to run schools, there is no shortage of people, particularly parents and educators, who care.

2. Tennessee has one of the most fascinating education stories in America.

I’ve had a front-row seat to massive changes in K-12 education under reforms ushered in by Race to the Top — an overhaul being tracked closely well beyond the state’s borders. But the national interest and import doesn’t end with changes stemming from the $500 million federal award. Tennessee is home to some of the nation’s premier education researchers, making its classrooms laboratories for new ideas about pre-K, school turnaround, and literacy instruction, just to name a few. And at the legislature, more lobbyists are devoted to education than to most any other cause. A lot of eyes are on Tennessee schools.

3. The education community is not as divided as it looks.

During the course of just a few years, I watched state lawmakers change their positions on accountability and school vouchers. I witnessed “anti-charter” activists praise charter leaders for their work. I chronicled task force meetings where state leaders who were committed to standardized testing found middle ground with classroom educators concerned that it’s gone too far. In short, a lot of people listened to each other and changed their minds. Watching such consensus-building reminded me that, while there are no simple debates about education, there is a widespread commitment to making it better.

4. Money matters.

Even when stories don’t seem to be about money, they usually are. How much money is being spent on testing, teacher salaries, school discipline reform? How much should be available for wraparound services? Why do some schools have more money than others? Is there enough to go around? Tennessee leaders have steadily upped public education spending, but the state still invests less than most other states, and the disparities among districts are gaping. That’s why more than a handful of school districts are battling with the state in court. Conversations about money are inextricable from conversations about improving schools.

5. Race is a significant education issue, but few leaders are willing to have that conversation.

More than 60 years after Brown v. Board of Education, Tennessee’s schools are largely racially segregated. Yet most policymakers tread lightly, if ever, into conversations about achieving real racial integration. And in many cases — such as a 2011 law enabling mostly white suburban Shelby County towns to secede from the mostly black Memphis district — they’ve actually gone backwards. Then there’s the achievement data. The annual release of test scores unleashes a flurry of conversation around the racial achievement gap. But the other 11 months of the year, I heard little about whether state and local policies are closing those gaps — or contributing to them — or the historical reasons why the gaps exist in the first place. To be sure, state leadership is trying to address some of Tennessee’s shortcomings. For example, the State Department of Education has launched modestly funded initiatives to recruit more teachers of color. But often, race and racism are the elephants in the room.

6. Still, there’s lots to celebrate.

If there were unlimited hours in the day, I could have written thousands of stories about what’s going right in public education. Every day, I received story ideas about collaborations with NASA in Oak Ridge, high school trips to Europe from Memphis, gourmet school lunches in Tullahoma, and learning partnerships with the Nashville Zoo. Even in schools with the steepest challenges, they were stories that inspire happiness and hope. They certainly inspired me.

Grace Tatter graduated from public schools in Winston-Salem, N.C., and received her bachelor’s degree in history from the University of North Carolina. She’s now pursuing a master’s degree in specialized studies at the Harvard Graduate School of Education.