Hypothesis Testing…and testing…and testing

Do higher scoring students get assigned
higher value-added teachers? Maybe— but more work is needed to really tell.

Early in 2014, Chetty, Friedman and Rockoff published a study based on New York City data that found that
value-added (VA) models which control for the previous years’ test scores can
accurately predict teacher impacts on current year student test scores. Along
the way, the team demonstrated the viability of a quasi-experimental design
that most districts— with student test and teacher assignment data— could use
to validate district VA calculations.

In the past several months, two studies have
replicated the Chetty et. al.’s
methodology and key results in two different samples: a study using data from Los Angeles by Bacher-Hicks, Kane and Staiger and one using data from North Carolina by Jesse Rothstein, who has been academia’s most
vocal critic of value-add methodologies.

Following the specifications and techniques of
the original paper, the two newer papers find little evidence of bias in
teacher VA measures (with the typical controls) and show that there are
differences in teacher VA across students and schools.

All three studies found a significant,
positive relationship between a student’s prior year test score and teacher VA.
In other words, higher-scoring students do get assigned to high VA teachers, on
average, with only some disagreement over what happens to special education
students.

Rothstein (North Carolina) found that minority students (Black or Hispanic)
and schools with higher fractions of minority students have higher VA teachers,
on average.  Chetty et al. (New York City)
found no such relationships. Bacher-Hicks
et al. (Los Angeles) found
significantly negative relationships at the student and school levels for
African-American and Hispanic students. In other words, in Los Angeles, African-American or Hispanic students or schools with
higher fractions of these students have teachers with lower VA, on average.

A key benefit of replicating research (which
happens much too rarely in education research) is that in addition to
comparing results in different locales, researchers can identify potential
weaknesses in a study’s design. While Rothstein
replicated the Chetty et al.’s
results using their techniques and specifications, he also went on to
demonstrate two potential flaws that should be the subject of future work: (a)
the results change depending on how one makes up for missing classroom or
teacher data; and (b) the “random” teacher moves— from classroom to
classroom or school to school— required for the quasi-experiment’s validity are
not so random after all. Taking these into account, he finds significant bias
in teacher value-added calculations: for some teachers, the models predict
better performance, on average, than they should, while for other teachers, the
models go the other way.

What’s the bottom line of these studies? There
seems to be clear evidence that in New
York City, Los Angeles and North Carolina, higher-scoring students
are assigned to higher VA teachers, on average. There is significant variation
across the three jurisdictions in the relationships between a student’s special
education or minority status and average teacher VA scores. Finally, Rothstein raises significant questions
about validity with respect to resolving missing data and the independence of
teacher turnover. All of these require further hypothesis testing— and more
replication studies.