You need to make data-driven decisions!" intoned the workshop leader. I doubt that any of the teachers and administrators in the audience intended to make "ignorance-driven" decisions. In fact, few school leaders are experiencing a shortage of data. Most are actually drowning in data, with a wealth of test scores, student demographic information, and an increasing load of "formative" assessment data that may or may not be worthy of the name (Popham, 2008). The challenge is facing both an overabundance of data and a scarcity of information that educators can readily use to make better decisions.
Data Everywhere … and Nowhere
My observations in three recent school visits in the western, mid-western, and eastern United States illustrate the point. In the first, more than a ream of paper containing test data sat on the principal's desk, the paper still shrink-wrapped in plastic where it had been since last April, unopened by the overwhelmed principal. In the second district, I conducted a focus group of 15 principals who were promised confidentiality in exchange for candor; 14 of them confessed to never having used the multi-million-dollar data warehouse their district had purchased. The third district was loaded to the gills with advanced technology for downloading and analyzing data, yet the teachers of an elementary school with more than 800 students were printing every test score, grade, and demographic characteristic of students on chart paper for their "data analysis" meetings. "It's easier this way," the principal explained.
Navigating the Data River
Here are four tips for gaining control over the flood of data that threatens to engulf us. First, commit to data analysis as a continuous process, not an event. A study by the Bay Area School Reform Collaborative (Oberman & Symonds, 2005) revealed that schools that reviewed data several times each month were far more likely to close achievement gaps than those that reviewed data only a few times a year.
Second, start with a clearly focused question, such as, What are our greatest areas of strength in literacy for our 8th grade students? Which specific math skills are weakest in 4th grade? How do the vocabulary scores of our students who receive free and reduced-price lunch differ from those of our economically advantaged students? How are the scores of students who attend school at least 90 percent of the time different from those of students who attend less than 90 percent? What is the relationship between the two previous questions—that is, are students who are disadvantaged also more likely to be absent from school? Having a clearly focused question will avoid the tedious and time-wasting exercise of trolling through spreadsheets, papers, and databases without any direction.
Third, develop a schoolwide culture of hypothesis testing, in which teachers consider their assumptions before they look at the data. Consider two examples: "If a topic was on the pacing chart in October and the teacher has covered it, then students will have learned it; if the topic was not yet covered, then the students will not have learned it." Although this seems straightforward and even obvious, I have challenged teachers in several schools to test the hypothesis with data. If the hypothesis were true, then students taking a formative assessment in February should ace the material covered in October and fail the material scheduled to be covered in April. In fact, the reality of student mobility and student inattention may render these assumptions inappropriate.
Here's another common hypothesis: "If students performed poorly on a math test, then we need to drill them harder on math skills." That's possible, but it is also possible that the root cause of low math scores is students' inability to read the problem or, in the case of open-response items, students' inability to express their responses in writing.
Fourth, go beyond the numbers to consider causes of student success and failure. When the only data available are student demographic characteristics, then it is easy to assume that the causes of high or low achievement are related to family income, ethnicity, gender, and primary language. Such an analysis avoids a consideration of the powerful influences of teaching practices, curriculum, and feedback, just to name a few variables you won't find in a data warehouse. It's easy to create PowerPoint slides and wall charts showing data. It's more challenging—but more important—to have discussions about how the classroom experiences of students differ.
For example, one school system examined the time devoted to literacy in 67 elementary schools. Although the system leaders believed that there was a consistent districtwide literacy program, an examination of time and test scores revealed striking differences, with the time devoted to literacy ranging from 45 to 180 minutes. When teachers plotted the percentage of students who scored "proficient" or higher in reading, they learned that the schools that spent more time on reading usually had a higher percentage of students who were proficient.
The Stories Behind the Numbers
Although it seems paradoxical, the essence of a successful discussion about data is a commitment to examine not only the data, but also the stories behind the numbers. Only when we can articulate the "why" behind the data and turn the lens on our own teaching and leadership behaviors can we understand how to move from drowning in data to improving professional practice.
Think of it this way: If we were focusing on the epidemic of teenage obesity, the data we analyzed would include teens' weight. But no thoughtful health educator would regard those scores as sufficient for an analysis of health. We would also evaluate diet, exercise, self-image, and other information. Similarly, student test-score data is only one piece of a complex puzzle. There are, after all, faces, names, and real lives behind the data.
