The revisions process for the Teacher Prep Review (TPR) standards includes five phases, with each phase including engagement from experts and practitioners in the field. In developing the protocols to examine teacher preparation programs, NCTQ first establishes clear claims to then gather evidence from each program, making it possible to evaluate performance against the claim.230 The connection between the claim being made and the evidence gathered to evaluate the claim strengthens the basis of the TPR standards.
Below is an overview of the NCTQ standard revision process.
Below is an overview of the NCTQ standard revision process.
PHASE 1: Identify the Content of the Standard
Literature Review
In Phase 1, NCTQ researchers conduct a literature review related to each individual standard, paying particular attention to current, converging, and confirmed research. NCTQ consults with experts in each respective discipline to ensure all relevant, peer-reviewed research is captured. Research is synthesized to share with the External Advisory Panels.
In Phase 1, NCTQ researchers conduct a literature review related to each individual standard, paying particular attention to current, converging, and confirmed research. NCTQ consults with experts in each respective discipline to ensure all relevant, peer-reviewed research is captured. Research is synthesized to share with the External Advisory Panels.
External Advisory Panel
An External Advisory Panel is convened for each individual standard. These panels consist of experts and practitioners that may include teacher education faculty, researchers, classroom teachers, deans, and leaders from the K-12 system. Each panel is a diverse group in terms of gender, race/ethnicity, and role (higher education, K-12, etc.). Their charge is to provide input on the framework of the standard, such as what content is most important to evaluate based on research and national standards guiding the field (e.g., NCTM Standards for the Elementary Mathematics Standard), what claims are appropriate based on the data sources and evidence available, and what could be potential benchmarks (or decision rules for assigning an overall grade to a program) for the standard.
An External Advisory Panel is convened for each individual standard. These panels consist of experts and practitioners that may include teacher education faculty, researchers, classroom teachers, deans, and leaders from the K-12 system. Each panel is a diverse group in terms of gender, race/ethnicity, and role (higher education, K-12, etc.). Their charge is to provide input on the framework of the standard, such as what content is most important to evaluate based on research and national standards guiding the field (e.g., NCTM Standards for the Elementary Mathematics Standard), what claims are appropriate based on the data sources and evidence available, and what could be potential benchmarks (or decision rules for assigning an overall grade to a program) for the standard.
PHASE 2. Conduct Initial Tests
After receiving recommendations from the panel, a test protocol is drafted and piloted in a sample test. The sample represents a wide array of programs, including public and private institutions which represent the full spectrum of programs from large state-based systems to small, private colleges. The selection of programs used in the sample also considers evaluations from past NCTQ reviews and the geographic location of the institution. The data collected from this first test is summarized to inform further revisions to the standards and additional sample tests that may need to occur. Results of this test phase are then presented to the Technical Advisory Group in Phase 4.
PHASE 3. Facilitate an Open Comment Period
Concurrent with efforts to collect important test data, NCTQ engages stakeholders during a four-week open comment period. During this period, interested stakeholders review the potential revisions to the TPR standard and submit feedback through a survey designed to gather broad feedback from the field. The survey provides an opportunity to comment on the content construct of a standard, data sources and potential pieces of evidence, and to answer questions related to potential thresholds (decision rules for assigning an overall grade to a program). NCTQ seeks to engage a wide-variety of stakeholders during the open comment period to ensure the standard is setting fair, reasonable and accurate expectations for preparation programs.
After the open comment period closes, NCTQ summarizes the feedback and posts a summary on each standard revision webpage. In Phase 4, the feedback is shared with the Technical Advisory Group and subsequently shared with each External Advisory Panel in Phase 5 and posts a summary on each standard revision webpage.
PHASE 4. Convene the Technical Advisory Group
After the open comment period is complete, the Technical Advisory Group (TAG) reviews the test data and provides feedback on grading protocols and rubrics to ensure sound methodologies are being applied. The TAG is a group of six individuals who bring measurement expertise to review scoring protocols and grading protocols for each standard. While the Expert Advisory Panel's focus is on the content of the standard and expectations for programs, the TAG focuses on measurement practices and psychometrics. Results from the open comment period are also shared. After considering the impact data from the test phase, the TAG makes recommendations regarding the methodology and grading protocols. Additional test phases may need to be completed based on feedback collected during phase 2-4 (initial test phase, open comment period, Technical Advisory Group).
Phase 5. Reconvene the External Advisory Panel
Based on feedback from the open comment period, the data collected from sample tests, and the recommendations of the Technical Advisory Group, the standard and grading protocols are finalized. The External Advisory Panel is reconvened to review the final draft of the standard and provide any final feedback on the grading protocols.
After completion of Phase 5, the new standard is ready to be applied to the complete sample of elementary programs. Programs then receive preliminary findings and are provided the opportunity to submit new or additional evidence. This process further mitigates potential errors in analysis due to less information. Based on the standard revision process, a technical manual is published on the NCTQ website detailing final protocols and rationales for decision points along the way.
Endnotes
