Schools that use data to make decisions are following some of the best advice from both the world of business (Bass, 1981) and the world of education (Teddlie & Reynolds, 2000). Many schools implement this directive by focusing on student achievement, and this makes good sense. If schools are in the business of helping students learn, then the data used to guide decisions should relate directly to student achievement. Schmoker (2001), discussing Oak Park School District in Detroit, makes it clear that effective schools use data well:If there is anything we can learn from such districts as Oak Park, it is that successful organizations do not just collect data, they revere it. They aren't satisfied with data until data have life and meaning for every teacher, every pertinent party. They use data to create and to ensure an objective, commonly held reality. . . . The use of data allows for organized, simplified discussions that merge to create focused priorities and productive action. (p. 51)
As simple as this dynamic appears, schools and districts frequently err in at least two ways in their efforts to be data-driven.
Two Mistakes
Using Indirect Measures of Learning
Using measures of student learning that are not sensitive to the actual learning occurring in classrooms is the first mistake. This commonly happens when a school or district relies on what I refer to as “indirect” learning data, often provided by off-the-shelf standardized tests and even state-level standards tests. Such measures are indirect because they frequently do not adequately assess the content that is actually taught in a given school. A school might, in fact, be producing impressive student learning gains, but the test data do not pick them up.
Madaus, Kellaghan, Rakow, and King (1979) noticed that when schools used indirect tests to measure student achievement, the schools did not appear to be very effective, but when they used direct tests—those that actually measured the content that teachers taught in the school—some schools looked highly effective. In their discussion of a study of 52 schools, these researchers referred to direct measures as “curriculum-sensitive” measures:Our results clearly indicate that what we call curriculum-sensitive measures are precisely that. Compared to conventional standardized tests, they are clearly more dependent on the characteristics of schools and what goes on in them. (pp. 223–224)
They further warned that attempts to judge the effectiveness of a school using standardized tests could produce false conclusions.
Currently, many states have developed their own tests reflecting the content standards articulated for the state. Although state tests based on state standards are a better option than off-the-shelf standardized tests, they still don't live up to the challenge of providing a comprehensive and timely picture of student achievement. Neither a single test nor even a set of tests can ever address all the content that is taught within a given subject area at a given grade level. As the National Research Council (1999) notes,No test can possibly tap all the concepts and processes embodied in a subject area. . . . Instead, test makers construct a sample from the entire subject matter, called a domain. (p. 67)
The National Research Council concludes that standardized tests and state tests based on standards certainly have their place in the landscape of K–12 education, but schools should not use them as the primary indicator of student learning.
When schools and districts use indirect learning data from standardized tests or standards-based state tests as the primary measure of student learning, they are operating like an individual who wishes to improve his or her physical fitness and collects data by weighing him- or herself on a scale. Although a relationship between one's weight and one's physical fitness does exist, that relationship is indirect. A person can be quite slim and quite unfit at the same time. Resting heart rate, cholesterol level, strength, speed, and endurance are more direct measures of physical fitness.
How can a school avoid making this mistake? The solution is straightforward but not necessarily easy to implement. A school must use assessments that actually measure the content that teachers teach. One option is to use district-made or school-made tests that measure the content taught in specific courses. But my preferred option is to develop report cards that track student performance on specific knowledge and skills. These report cards might report on standards or more specific learning objectives (Marzano, 2000). For example, a report card might provide grades or rubric scores for five or six standards as opposed to a single overall grade for the course. Instead of relying on one summative test given at the end of a quarter, report cards and transcripts generate formative data over an entire quarter. They also put the classroom teacher at the center of the assessment dynamic. No matter how good a curriculum-specific test is, it cannot assess the rich variety of data captured by teachers who interact with students on a daily basis. Commenting on the effective use of formative assessment, researchers Black and Wiliam (1998) note that it can improve achievement by about seven-tenths of a standard deviation:Achieved on a nationwide scale, [this gain] would be equivalent to raising the mathematics achievement score of an “average” country like England, New Zealand, or the United States into the “top five” after the Pacific-rim countries of Singapore, Korea, Japan, and Hong Kong. (p. 61)
Having No Explanatory Model to Interpret the Data
The second mistake is less obvious than the first and more insidious as a result. It happens when a school or district has no system or plan for interpreting and using the data. Let's go back to the analogy of the person seeking to use data to improve physical fitness and assume that he or she is now using good direct data—resting heart rate, cholesterol level, strength, speed, endurance, and so on. The data are still not useful because there is no accompanying explanation of how to improve.
Fortunately, medical research provides us with a great deal of knowledge about those factors that affect physical fitness—diet, aerobic exercise, anaerobic exercise, resistance training—and an individual using that knowledge can make changes that would have a direct causal effect on his or her health.
What Works
Just as fortunately, education research has shed light on those factors that affect student learning. In fact, 35 years of research concretely identify 11 school, teacher, and student factors that are the primary determinants of student achievement. What Works in Schools: Translating Research into Action (Marzano, 2003) describes in some detail the research and rationale behind these 11 factors.
School-Level Factors
School-level policy and practice account for five factors. A guaranteed and viable curriculum refers to the fact that no matter who teaches a given class, the curriculum will address certain content. In addition, the curriculum is viable in that teachers can teach it adequately in the time allotted.
Challenging goals and effective feedback means that a school has a method of assessment that provides detailed information on specific learning goals for specific students on a timely basis—at least once every nine weeks. Schools use these data to set specific learning goals for individual students and to monitor student progress toward those goals systematically.
Parent and community involvement refers to structures that are in place to involve parents and community in the important decisions regarding the school as well as in the day-to-day functioning of the school.
A safe and orderly environment involves the employment of school-wide rules and procedures that create safety and order for students and teachers alike.
Staff collegiality and professionalism involves governance structures that allow for direct teacher input regarding school policy and a rigorous and comprehensive professional development program that encourages teachers to try new instructional strategies in an action research mode.
Teacher-Level Factors
Classroom teachers have direct control of three factors. Instructional strategies refer to the use of highly effective teaching techniques that enhance student learning. The effective teacher not only has a large array of such strategies at his or her disposal, but is also adept at determining which strategies to use with specific students and content.
Classroom management refers to establishing classroom rules and procedures and disciplinary interventions in the context of appropriate relationships between teachers and students.
Classroom curriculum design involves sequencing and pacing instructional strategies to build on the prior knowledge of students.
Student-Level Factors
Finally, the student-level category accounts for three factors.
Home atmosphere refers to how supportive the home environment is for the student in general and the academic achievement of the student in particular.
Learned intelligence and background knowledge addresses the extent to which students have attained an experiential base that provides them with a wealth of incidental learning. Such knowledge is, in effect, a type of academic intelligence that students can learn (see Ackerman, 1996).
Student motivation addresses how much students are interested in learning the content presented in school and their sense of their ability to master that content.
Even though some educators believe that these three student-level factors are characteristics beyond the responsibility of the school, schools and individual classroom teachers can intervene to alter them.
“What Works in Schools” Survey
- To what extent do we engage in this behavior or address this issue?
- How much will a change in our practices on this item increase the academic achievement of our students?
- How much effort will it take to significantly change our practices regarding this issue?
Figure 1. Do We Have a Guaranteed and Viable Curriculum?
Using Data: Two Wrongs and a Right-table
Not at all | Not oftern | To some extent | To a great extent | Not at all | Not much | To some extent | To a great extent | Too much to do | A lot but possible to do | Some | Not much | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IN MY SCHOOL . . . | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |
| "Guaranteed and Viable Curriculum 1. We have identified the content essential for all students to learn compared with the content considered supplemental and communicated this information to teachers." | ||||||||||||
| 2. Teachers can address the amount of essential content in the instructional time available. | ||||||||||||
| 3. The essential content is organized and sequenced in a way that students have ample opportunity to learn it. | ||||||||||||
| 4. Someone checks to ensure that teachers address the essential content. | ||||||||||||
| 5. We protect the instructional time available to teachers by minimizing interruptions and scheduled non-instructional activities. |
The first question focuses on how well the school is doing in terms of a particular school-level, teacher-level, or student-level factor. Scores near the low end of the scale indicate that the school does not engage in the behavior or address the issue well; scores near the high end of the scale indicate that the school does address the issue well.
Answers to the second question for each item provide perceptual data about the extent to which changing current practices will actually enhance student achievement. The logic behind this question is that researchers have noted that schools must determine whether those factors identified as important to student achievement in the research literature are actually important in their specific situations. In technical terms, many of the variables identified from the research are “situated.” When the research literature provides guidance at the general level, a school must determine whether that guidance is appropriate for its specific situation. Consequently, I advise against simply adopting without question the 11 factors that I propose as important in a specific school. As Reynolds and Teddlie (2000) point out, Sometimes the adoption of ideas from research has been somewhat uncritical; for example, the numerous attempts to apply findings from one specific context to another entirely different context when research has consistently demonstrated significant contextual differences. (p. 216)
Answers to the third question provide perceptual information regarding how much effort it will take to change current practice. The rationale for this question is that the most effective data-driven reform is incremental in nature—taking on issues that will have an impact on student achievement and have a high probability for success if the school addresses them (Fullan, 1982; Reynolds & Teddlie, 2000). Addressing an issue that an administrator perceives as important but for which change will require more time, resources, and energy than staff are willing to spend is a formula for failure.
