Over the last decade, the District of Columbia school system has seen a plethora of major educational changes, from open enrollment and a burgeoning charter sector to overhauled teacher evaluation and pay systems.
A new Mathematica study suggests no single change, but all of them together, spurred public schools in the nation’s capital to improve faster than states and county districts that didn’t use so-called market-based education reforms since the 1990s.
And the educational changes contributed more to the increase in achievement than any gentrification or changes in the student population, the researchers said.
“We really focus on the overall picture rather than trying to break it into individual components,” said Duncan Chaplin, senior researcher at Mathematica, who co-authored the study with researchers Dallas Dotter and Maria Bartlett. “The improvements in grade 4 were very impressive, and they didn’t fade out by grade 8. ... [The benefits] didn’t fade out as kids got older, and they didn’t fade out over years” as the changes were implemented.
Chaplin said the study provides evidence in favor of market-based education reforms that have gained traction in many urban districts. However, the results also highlight potential pitfalls for less high-profile districts attempting to copy the capital city’s formula. The study did not, however, examine some other reforms, such as the common core, that were overlapped with the market-based changes.
Washington D.C.'s major education changes started with the passage of the Public Education Reform Amendment Act in 2007, which abolished the district’s board of education and created a mayor-appointed chancellor to run the schools. Then-Mayor Adrian Fenty and the first Chancellor Michelle Rhee launched aggressive and controversial structural changes, including implementing a high-stakes teacher evaluation system in 2009. The district gave the public charter school board control over approvals and renewals based on performance; the portion of students attending charter schools in the district has risen from 27 percent in 2007 to 46 percent a decade later. Finally, in 2014, the district opened a districtwide enrollment system for students.
The Mathematica study spans close to 30 years: five waves of 4th and 8th graders before reforms began in 2007 and three waves of students afterward, from 1996 to 2017 for math and 1998 to 2017 in reading. Researchers used expanded data from the National Assessment of Educational Progress in math and reading in both grades to compare the achievement of students in Washington, D.C., to that of two separate groups—one of states and one of counties—which participated in NAEP during that time. The researchers left out states or counties which had implemented similar systemic education changes to the District of Columbia’s: abolishing teacher tenure, increasing charter school enrollment, and using unified school enrollment systems.
A look at the numbers
During that time, the District of Columbia also saw an influx of more white and higher-income students. Since 2007, the share of Black students has fallen 15 percentage points and the share of white students has risen by the same amount, and the NAEP has changed the students it tests to reflect this. However, after controlling for those demographic changes, the researchers found Washington, D.C., students—and Black students in particular—made much faster academic progress compared to states and counties that did not implement the same collection of education changes. After controlling for demographics, public students in the nation’s capital performed on average 12 percentile points—about a third of a standard deviation—higher in math in grades 4 and 8 and more than a quarter of a standard deviation higher in reading in grade 4. The reading benefits faded by grade 8, however.
That meant Black students in District of Columbia schools scored on average 8.3 points higher in math at 8th grade and 9.5 points higher in 4th grade, while Hispanic students scored on average 9.5 points higher in math at 8th grade and 8 points higher in 4th grade. White students in Washington performed more than 6 points better on average in 4th grade math. It’s not clear why white students showed less benefit, but Chaplin noted that in highly racially segregated schools in the district, white students may have been more likely to have started out in higher-performing schools.
Evidence of Washington’s rapid improvement isn’t new. Earlier this year, the Council of the Great City Schools judged it the fastest-improving of all large urban school districts over the last decade. The district outperformed the CGCS’s expectations by a greater amount from 2009-19, and in more subject-and-grade combinations, than any other large urban district.
The Mathematica study does not analyze what specific elements may have driven the District of Columbia’s improvements, and does not compare the city’s students to other districts that used similar market-oriented education changes, but its results are in line with those seen in Detroit, Mich., and in New Orleans following Hurricane Katrina, when both urban districts boosted charter schools and school choice, and overhauled teacher tenure.
Douglas Harris, an economics professor at Tulane University and the author of “Charter School City: What the End of Traditional Public Schools in New Orleans Means for American Education,” who has studied New Orleans’ education changes post-Katrina, suggested the different policies may help build on each other over time to eliminate the lowest-performing schools or teachers, leading the improvements to show up after a few years.
For example, “at least in new Orleans, it was really driven by the process of taking over low-performing schools. ... Basically, they would start some [charters], some of them would work and were effective and did well, and those stayed open, and the ones that didn’t succeed, they gave over to another charter operator and let them try,” Harris said. “In the process, they kept winnowing out the low-performing schools and that lifted up the average over the first seven or eight years.”
However, Harris cautioned that both New Orleans and Washington have been high-profile, nationally watched turnaround districts in culturally rich cities, able to draw both foundation support and teacher and leader talent to support systemic overhauls to schools and districts. Smaller and less well-known cities and rural areas may have difficulty replicating and sustaining large-scale reforms.
“If it’s the nation’s capital, that’s also kind of an attractive place to be for the younger folks who tend to be involved in that kind of school reform. So that gives them an advantage in attracting people,” Harris said. “So I think that context is also important, and it also is relevant to the rural-urban piece” of implementing market-based education changes.
Harris said he expects to release a separate study comparing the implementation and progress of different districts, like New Orleans and Washington D.C., that implemented market-based reforms, in the next few months.
