Evaluation relies on two sources of data to produce a preliminary score for each program to review before the score is finalized:
Course syllabi
Course descriptions
To determine the validity of using course descriptions in lieu of syllabi, a pilot test evaluated 200 courses for which both course descriptions and syllabi were available to arrive at two independent assessments for how much time was dedicated to each of the five topics. The two forms of analysis produced similar results on average, within a range of 10 percentage points for each of the five topics. Further, the distribution of program grades under the two approaches was highly correlated, with no notable bias towards either approach. For more details from this pilot study, go here.
Methodology in brief
A team of analysts use course catalogs to determine the required coursework for each elementary program in the sample. Analysts then read course titles and descriptions to pinpoint all courses that are inclusive of mathematics. Teacher audience mathematics courses, inclusive of those that focus on both content and pedagogy are flagged for analysis. General audience mathematics courses (College Algebra, for example) are excluded from analysis.
A separate team of expert mathematics analysts evaluate syllabi and course descriptions using a detailed scoring protocol. Each course is analyzed for its coverage of five topics:
Numbers and Operations
Algebraic Thinking
Geometry and Measurement
Data Analysis and Probability
Mathematics Pedagogy
Two additional topic categories of "Other Content Topics" and "Other Pedagogy Topics" are also included in the coding process so that non-relevant material (science pedagogy, for example) can be accounted for while coding the totality of each course.
Course descriptions for every course are independently evaluated by two analysts. Twenty percent of syllabi are randomly selected to be scored by a second analyst. In both cases, disagreements are adjudicated by a third analyst.
When coding a Course Description |
When coding a Syllabus |
||
---|---|---|---|
Reference Count |
Count of the total number of references to each identifiable topic (e.g., Numbers and Operations) |
Unit Type |
Identify the type of calendar (daily, weekly, etc.) used in the syllabus. |
Unit Count |
Identify the total number of units (days, weeks, etc.) that are defined in the course schedule. |
||
Reference Count |
For each unit, count of the total number of references to each identifiable topic. |
Within the course description or details presented in the syllabus... |
Within each unit specified in the syllabus... |
||
---|---|---|---|
1. Numbers and Operations |
Count of the number of references to Numbers and Operations including any subtopics within that domain. |
||
2. Algebraic Thinking |
Count of the number of references to Algebraic Thinking, including any subtopics with that domain. |
||
3. Geometry and Measurement |
Count of the number of references to Geometry and Measurement including any subtopics with that domain. |
||
4. Data Analysis and Probability |
Count of the number of references to Data Analysis and Probability, including any subtopics with that domain. |
||
5. Mathematics Pedagogy |
Count of the number of references to mathematics instructional approaches. |
||
6. Other Content Topics |
Count of the number of references that address mathematics content topics outside any of the four content areas (e.g., references to trigonometry or calculus). |
||
7. Other Pedagogy Topics |
Count of the number of references that address general pedagogy or subject-specific pedagogy for subjects other than mathematics. |
After coding all courses, the percentage of each course dedicated to each topic is calculated. The resulting percentages are multiplied by the total instructional hours for the course (1 semester credit hour is equal to 15 instructional hours). The instructional hour counts are summed across all courses within a program and those values are measured against the scoring rubric.