New data shines a spotlight on what's happening in the black box of teacher prep – but what it reveals is just plain depressing.

A non-profit organization which describes itself as an education-focused version of Consumer Reports, the Educators Consumers Foundation is out with an interesting small study looking at the outcomes of graduates from Tennessee's teacher prep programs. While a few programs perform well, most programs in the state routinely graduate substantially more ineffective teachers than effective ones.

Let's sit with that for a moment: most of Tennessee's teacher prep programs are producing more teachers who fall in the bottom quintile of performance than those in the top quintile.

There are some caveats, but none which justify dismissing the grim findings.

First, one might assert that it's unfair to expect new teachers to be effective. The data compares teachers' performance over their first three years of teaching with the performance of all teachers in the state, so perhaps it's unrealistic to expect a program to produce effective new teachers relative to the broader population of teachers. In any case, if it were so unrealistic, how come some programs consistently produce relatively new teachers who perform just fine against more experienced peers? Here's looking at you, Lipscomb University—which NCTQ also identified as having a top-ranked undergraduate secondary prep program in the country in 2014. 

Also worth pointing out is that many of the findings aren't statistically significant. For example, the state's 2015 report card offers data on Lipscomb's teacher performance in 11 different grade/subject bands – but only two are statistically significant.

Finally, and this problem continues to be a major sticking point for us as we try to evaluate programs on student achievement outcomes, the state continues to lump all graduates from an institution into the same pot, even though they have graduated from programs within that institution which we know are substantially different in terms of their selectivity, coursework, and the professors who teach the coursework (as we show here). In other words, there's almost no similarity between graduate and undergraduate programs on the same campus. That's why NCTQ is unable to include VAM data for Tennessee programs in our Teacher Prep Review, tempting though it may be.

These three concerns aside, the findings paint a stark picture of programs that are not instilling minimal competency in their grads.