Better data drive better decisions—or so we think. Yet all around us are examples of people and organizations with good information making bad decisions. Consider the case of Blockbuster. Less than a decade ago, Blockbuster had 8,000 stores in 17 countries and millions of customers. It was awash in data, including customer surveys that said people disliked going to video stores and hated late fees. Company leaders were also well aware of the rapid growth of Netflix, having turned down offers to acquire it. Despite all this telling information, Blockbuster failed to change with the times—and went bankrupt in 2010 (Satell, 2014).
Accounts of what happened inside Blockbuster reveal striking parallels to observations of school data teams. These observations show that putting data in teachers' hands doesn't guarantee better student performance. Many obstacles thwart the effective use of data in schools, including aspects of the data themselves and the mindsets of those expected to act on the data.
Data, Data Everywhere and Not a Thought to Think
Few educators complain about having too little data. A recent survey of 4,600 teachers found that teachers are overwhelmed with so much information that it's difficult to "separate the signal from the noise" (Bill and Melinda Gates Foundation, 2015, p. 18). Data often come too late, leaving teachers feeling as if they're driving away, looking at data through the rear window. Case studies of four schools in the San Francisco Bay Area, for example (Young, 2006), revealed that many teachers had little use for data from standardized reading assessments. One teacher noted that simply listening while students read provided more salient, real-time information than did formal assessments.
Teachers can be ill-equipped to parse and make decisions from data. A study that asked 230 teachers to make judgments from particular data sets found the teachers often "lost track of what they were trying to figure out" and relied on general impressions rather than empirical ones "if the calculation became at all complicated" (U.S. Department of Education, 2011, p. 61).
Even if teachers receive timely data and are skilled enough to analyze it, school culture can impede the ability to use data effectively. An in-depth examination of two urban middle schools engaged in mathematics reform (Horn, Kane, & Wilson, 2015) found that in one school, data conversations were mostly superficial. Discussions focused on how to present data differently or asked teachers to predict the percentage of students who might fall into various performance categories on the statewide test—with no reflection on how to change instruction to boost achievement.
Similarly, a team of researchers from Florida and California observed data teams simply going through the motions. Teams quickly worked through a data discussion protocol—without reflecting or identifying any changes in instruction—before exclaiming, "Yay, we're done!" (Datnow, Park, & Kennedy-Lewis, 2013).
However, the good news is that those same researchers observed other teacher teams digging deeply into data, respectfully questioning one another, and examining their own practices. This was usually thanks to a principal who clearly defined the purpose for data analysis and created a "we" feeling in the school (for example, stressing that math achievement was everyone's responsibility). These strong leaders created and modeled norms for data conversations, specifying what materials—and attitudes—teachers should bring to meetings. They set up ways teachers would hold one another accountable, argue productively in a safe and confidential environment, and ensure that conversations about students never turned to "nit-picking or trash talking" (Datnow, Park, & Kennedy-Lewis, 2013, p. 354).
Surfacing—and Challenging—Mindsets
Case studies have also revealed that data reports can be a Rorschach test for schools: What teams see in data says a lot about their mindsets about students and schooling.
In one urban middle school, conversations about improving student achievement focused mostly on sending "bubble kids" (whose results fell just below cut scores for proficiency levels) to tutoring or after-school programs (Horn, Kane, & Wilson, 2015). Teachers focused on what someone else might do and rarely touched upon how they might change their own instructional practices to boost student learning.
The school principal also revealed a behaviorist mindset about student learning by directing teachers to show students their test scores to motivate (in other words, scare) them into doing better. This leader also quipped that the after-school math program ought to be reserved for well-behaved students—showing no awareness that poor behavior might reflect student disengagement or difficulties. The school's mindset appeared to be, "We're okay; it's our students who are the problem."
Teachers in the other school profiled in the same study dug more deeply into student test data, examining questions students had missed to determine what concepts learners were struggling to grasp. Yet even as they surfaced possible misconceptions, the new teaching strategies these data teams came up with focused on helping students perform better on those particular test items, rather than on improving instruction (Horn, Kane, & Wilson, 2015).
Seeking "Kaizen"
Blockbuster's executives were smart people with good information. However, their mindsets got in the way. Many had come from retailers like Wal-Mart and 7-Eleven, so they focused on generating impulse buys in the video checkout aisle. They also faced pressure from shareholders demanding quick results—which may explain why they ousted Blockbuster's CEO when he proposed dropping late fees and investing heavily in a digital platform to rival Netflix's (Satell, 2014).
High-pressure environments—whether in schools or business—make it hard for people to own up to their shortcomings and work together. That's exactly what a study of high-stakes accountability in an urban district found. The longer low-performing schools faced the threat of sanctions, the less apt they were to examine their underlying assumptions and current practices—and pull together to improve them (Finnigan, Daly, & Che, 2012).
Educators who fear being judged by their shortcomings will be less likely to engage in self-reflection. Unless thoughtful leaders create a safe environment and set a tone akin to what Japanese manufacturers call kaizen—the view that "every defect is a treasure" that offers a chance to improve—teachers can have all the data in the world, but they won't be able to use the information to guide improvement.
