Finally, we received a letter from one of the authors of the certification studies that we criticized in the last issue of TQB, Edward Fuller of the University of Texas at Austin (no longer of the Charles Dana Center as we reported.) “In your recent bulletin, you critique a paper presented by Celeste Alexander and myself. Unfortunately, you made a few errors in reviewing our paper. First, our study was mandated by the Texas Legislature in 2002 when both Ms. Alexander and myself were Co-Directors of Research at the State Board for Educator Certification in Texas. This explains why the report is brief–the write-up was intended for legislative policymakers rather than academics.
“Second, the actual work behind the study took almost one year. Ms. Alexander and I ensured that only the most rigorous methods were employed in analyzing the data. We had the study reviewed by experts in the field and they did not have any major critiques of the methodology.
“Third, we actually have three years of data and used only two years in this first study. Had you read the paper, you would have noticed that we used two years of TAAS data in the study. The first year’s scores were used to control for prior level of achievement. Using the second year’s scores as the dependent variable and the first year’s scores as one of the independent variables gives you the same results (in terms of which variables that are statistically significant) as if you used the actual gain scores as the dependent variable. In fact, we ran a few models with the gain score as the dependent variable and the results were the same. If you read Sanders’ work carefully, you will see that only two years of scores are necessary to conduct value-added analyses, but three or more years are necessary when identifying the performance levels of individual teachers. Our study was not intended to identify individual high-and low-performing teachers.
“Finally, certified teachers in our study include only FULLY certified teachers. This means that teachers enrolled in alternative certification programs, teachers on out-of-state permits, and teachers on emergency permits are ALL considered NOT certified.
“Fourth, if you had examined the paper in detail, you would have noticed that we actually controlled for a large number of factors affecting student achievement, including student SES and race/ethnicity.
“One more thing–I am not at the Charles A. Dana Center, but rather the University of Texas at Austin in the Department of Educational Administration. Had you actually had a copy of the paper, you would have seen this on the front page.
“To imply our work is shoddy when you clearly did not read the actual paper (or did not read it very carefully) nor try to contact either one of us to ask details about the analysis is quite ironic, don’t you think? I expected better of you given your reputation.”
Ed Fuller, PhD
Department of Educational Administration
The University of Texas at Austin
Several points can be made in response.
First, the audience for which the paper was intended does not excuse its brevity and lack of crucial detail, especially given that it was then delivered at the annual AERA conference. It should not be necessary to contact authors directly to determine such basic methodological questions found lacking in the paper such as how alternatively certified teachers were classified…as emergency cert or as full cert? Or other key questions such as, when the data is disaggregated what are the various grade level effects? Was there a greater impact of being assigned an emergency certified teacher in high school than middle school? What is the impact of being assigned an alt cert math teacher over many years?
Second, now that we have learned from Professor Fuller how the alt cert teachers were classified–by grouping them with the emergency cert teachers–we are even more adamant that alternative certification teachers must be judged as a distinct group and not lumped in with emergency certification teachers who generally meet no academic standards. Studies that don’t do this just aren’t asking the right questions. No one is defending the hiring of teachers who can’t pass an easy licensure test. In fact, three out of four teachers who are on waivers in the state of Texas have met no academic standard; only one in four of these probationary teachers who have expertise in their content area. No wonder Fuller found an impact from certification, however tiny (the effect amounted to .045 standard deviations).
Lastly, Professor Fuller raises a couple of valid points that we will concede. We were not clear enough about the intent of our list of criteria that certification studies should meet. As Fuller noted, it looked as if we were accusing all of three studies of not meeting any of these criteria. The intent was to establish some ground rules, not to imply that none of the studies had met any of the criteria. Certainly in Fuller’s case his study did maintain the proper controls) and we take full blame for not making that clear. Also, while it would have been clearly advantageous for the study to have based its findings on more than one year’s growth (the study looked at student achievement gains from the 1998 math test to the 1999 math test), the fact that it did not is not sufficient cause for dismissing its findings.