School leaders frequently hear calls to be data driven. At first glance, such an approach seems appealing, but when putting this idea into action, a number of questions arise: How do teachers use data? How can school leaders support teachers in this process?
For the past four years, our research team has studied how middle school math teachers use data when they work in grade-level teams. Looking across dozens of meetings, we saw teachers using assessment data to identify topics to reteach and students who would benefit from reteaching. Although this approach is "data driven" in that teachers use data to plan instruction, it does not necessarily help them improve their instruction.
More productive approaches give teachers opportunities to learn about both students' thinking and their own teaching. Learning about students' thinking can help teachers improve their practice by, for example, anticipating and responding to common misconceptions. Instructional leaders—coaches, administrators, or teacher leaders—often play a crucial role in helping teachers use data to support student learning.
Four Guiding Questions
When we study teachers' data use, we find that teachers use data to answer four guiding questions
- What do we need to reteach?
- To whom do we need to reteach it?
- Why did students struggle with this?
- How do we reteach it?
Typically, teachers tend to address only the first two questions. The most productive conversations, however, venture into the more challenging territory of why and how. Teachers learn a lot more about mathematics, students, and teaching in these deeper conversations.
Learning About Students' Thinking: What and Why
Most data-driven teacher groups use student performance data to determine what to reteach. Of course, learning what content to reteach is important, but productive groups delve deeper to address why students struggled. To do this, teachers might identify frequently selected distractors on multiple-choice questions and consider what those choices might indicate about students' thinking. Alternatively, teachers might look at a question's wording to determine things that might be contributing to student misconceptions—confusing vocabulary, an unclear prompt, or a nonstandard diagram. This information might help teachers make informed guesses about students' thinking.
Learning About Instruction: To Whom and How
Another common way teachers use student data is to identify to whom content should be retaught. Learning which students need additional help is an important first step, but productive groups also ask how to reteach the content more effectively to these students. For example, teachers might compare their instructional strategies for lessons on the standard, noting what worked (and didn't work) for students.
Putting the Guiding Questions Together
Teachers can use data conversations to learn about two dimensions of education: (1) students' thinking and (2) instruction. As Figure 1 shows, the way teachers combine the guiding questions—what, to whom, why, and how—determine how strong data conversations will be along these two dimensions. The best conversations about data are strong in both dimensions.
Figure 1. Four Ways of Using Student Performance Data
To illustrate, we present four vignettes modeled on actual recordings of teachers' data-use meetings. In these discussions, three math teachers named Sherry, Jodi, and Hewitt discuss assessment results with an instructional coach. In the meetings—as in most of these meetings in our study—teachers have copies of the assessment, assessment results showing how many students answered each item correctly, and a chart with a distribution of responses for each item.
Although each vignette starts out the same, the learning opportunities vary as a result of the differing approaches to data use. To highlight these differences, different instructional coaches facilitate each vignette: Alicia, Bob, Carol, and Duane. When the coaches ask questions, we mark them in italics to highlight how their facilitation steers the conversations.
Vignette 1. Targeted Reteaching
In targeted reteaching, teachers use data to discuss what to reteach and to whom, but they do not discuss why students struggled with a particular topic or how teachers could teach it differently. Such conversations provide few opportunities for teachers to learn about students' thinking or instructional practices and therefore provide minimal support for improving instruction. In the vignette below, the teachers look at the results of an interim assessment and decide how they should respond.
SHERRY: So it looks like proportional reasoning was pretty rough. We should go over that again.JODI: Yeah, I think we have to. Standards 1 and 3 were the worst, right? But I think Standard 2 is probably OK.COACH ALICIA: There were 290 kids who didn't master Standard 1.HEWITT: They missed all three questions?COACH ALICIA: Either three or two out of the three.SHERRY: OK.COACH ALICIA: Standard 2 is what they performed the best on. Only a handful of kids missed that.HEWITT: So let's reteach Standard 1 on Wednesday and Standard 3 on Thursday.JODI: Sherry, why don't you and I take the kids who didn't master it? Hewitt, you can take the kids who did master it and do an extension activity.SHERRY: OK, sounds good.
Here, the data are used to make logistical decisions, like what topic to reteach (Standards 1 and 3) or which kids need intervention (the 290 students who missed either two or three questions). Throughout this excerpt, the teachers emphasized what they would reteach and to whom, but they didn't consider why students struggled with Standards 1 and 3, nor how they might reteach to help students better understand.
Vignette 2. Tips and Tricks
In tips and tricks conversations, teachers use data not only to identify what to reteach and to whom, but also to discuss how to reteach. This approach provides stronger learning opportunities for teachers but is similar to the targeted reteaching approach in its weak emphasis on learning about students' thinking.
This vignette begins identically to the first, but takes a different turn when Coach Bob responds differently than Coach Alicia.
SHERRY: So it looks like proportional reasoning was pretty rough. We should go over that again.JODI: Yeah, I think we have to. Standards 1 and 3 were the worst, right? But I think Standard 2 is probably OK?COACH BOB: I think so. So Standard 1 is about finding unit rate, and Standard 3 is solving proportions. How do you want to approach that?HEWITT: Hmm. Sherry had a good way for teaching unit rate, right?SHERRY: I tell my kids to always write a proportion and then use the fish method to solve it.HEWITT: Yeah, that's good. Kids can remember it easily.JODI: OK, I'll reteach it that way. Can I borrow your worksheets for that?SHERRY: Absolutely. I'll run off some copies for you later.
This conversation went beyond targeted reteaching because in addition to identifying what to teach, Coach Bob asked teachers to think about how to teach it. In response, Sherry shares a method she used in her classroom. In tips and tricks conversations, teachers have some support to rethink their instruction. Such sharing begins to combat commonplace norms of privacy among teachers.
However, these conversations often offer superficial, one-size-fits-all strategies that do not adequately address the source of students' learning difficulties. Sherry's suggestion to use the "fish method" fits this description: It lacks specificity, and it's unclear whether and why this would help students develop a better understanding of proportional reasoning. Thus, their discussion of how to reteach proportional reasoning still fails to address students' mathematical understandings. This is a critical omission. Teachers still need to discuss why students struggled to develop criteria for whether Sherry's strategy will support students' understanding of proportional reasoning.
Vignette 3. Bounded Improvement
Bounded improvement conversations build on the what and to whom emphasis of targeted reteaching but add the question of why. This is an important addition, because teachers' knowledge of students' mathematical thinking is crucial for improving instruction. However, bounded improvement conversations still limit teachers' learning opportunities because they fail to connect this knowledge to future instruction.
In this vignette, Coach Carol presses teachers to discuss student understanding by drawing teachers' attention to this problem, Number 8 on their assessment:
- Susan
- José
- Mercedes
- They all ran the same pace
To answer this question, students need to compare different athletes' running paces. There are multiple strategies for doing so, but they all rely on a sense of how to compare proportions with different denominators (Mercedes ran the fastest pace).
Like Coach Bob, Coach Carol turns the conversation in a new direction by asking the teachers to think about why students struggled with solving proportions.
SHERRY: So it looks like proportional reasoning was pretty rough. We should go over that again.JODI: Yeah, I think we have to. Standards 1 and 3 were the worst, right? But I think Standard 2 is probably OK?COACH CAROL: Yeah. So what were the questions that kids struggled with?HEWITT: Number 8 was pretty low. It was on comparing unit rates. The one where they had to decide which kid ran the fastest.COACH CAROL: Why do you think that was so difficult?SHERRY: Well, they might not have remembered to set up a proportion correctly.COACH CAROL: OK. Are there any other reasons why it might have been hard for kids?HEWITT: It might also have been because the rates don't reduce easily.JODI: Right. And they might not have realized that they can use a calculator.SHERRY: Yeah. My kids always struggle with reducing fractions. We should definitely review reducing fractions. Let's hit that hard in our warm-ups next week.
In this conversation, the teachers identified several possible trouble spots, which is an improvement on the targeted reteaching conversation. This specificity comes from Coach Carol's line of questioning about why students may have struggled with this question. This provides supports for teachers to improve their instruction through a grounded knowledge of students' thinking.
But although they agree to reteach reducing fractions, they do not discuss how, specifically, to do so. Although some teachers may know what to do after this conversation, teachers who don't already know how to "hit" fraction reduction to further students' understanding would have little opportunity to learn. Without openly and explicitly discussing how to reteach, individual teachers' opportunities for improving are bounded by what they already know.
Vignette 4. Responsive Revisioning
In responsive revisioning, the teachers answer all four guiding questions: what to reteach, how to reteach it, to whom it should be retaught, and why students struggled with the assessed content. This approach is the most likely to lead to instructional improvement.
In this example, unlike the other coaches, Coach Duane presses teachers to make explicit connections between reasons students struggled and proposed instructional responses:
SHERRY: So it looks like proportional reasoning was pretty rough. We should go over that again.JODI: Yeah, I think we have to. Standards 1 and 3 were the worst, right? But I think Standard 2 is probably OK?COACH DUANE: Yeah. What were the questions kids struggled with?HEWITT: Number 8 was pretty low. It was on comparing unit rates. The one where they had to decide which kid ran the fastest.COACH DUANE: So what do you think made Number 8 difficult for students?SHERRY: Well, they might not have remembered to set up a proportion correctly.JODI: That might be an issue for some of them, but I also see that a lot of kids picked answer B. I think they might have gotten confused by what the "fastest pace" means—like, maybe they thought it meant who ran the longest. Or they might have divided minutes per miles and then went with the biggest number.COACH DUANE: So how do you want to address that?HEWITT: Let's do some more work around pace, speed, and distance. Our kids often get confused by those.JODI: Right. We could give them problems like Number 8 and see what strategies students are using to solve them.COACH DUANE: Yeah. I have some similar tasks I can give you all. Remember the Orangey problem from last year? Do you remember how that got out their misconceptions? That worked really well.HEWITT: Oh yeah! In my class, we compared the different strategies. It worked well because they could solve it in whatever way works best for them.SHERRY: We should rewrite it to include some rates that don't easily reduce so we can see who's still struggling with that.COACH DUANE: Good idea. How do you think kids will deal with those?JODI: Well, when I've used those on homework, I see some of them divide distance by time to get speed—like miles per minute. Then they want the biggest number. Sometimes they divide time by distance to get, like, pace—like minutes per mile. Then they'd want the smallest number.HEWITT: We should make sure to pull out both strategies. See if kids can see how they're related.COACH DUANE: OK. You might see some other ideas, too. I'll rewrite the Orangey problem and get it out to you.
Like Coach Carol, Coach Duane asked teachers to consider why students struggled with comparing unit rates. However, Coach Duane also pressed them to justify their planned response. He anchored the discussion in a problem they had done in the previous year to help them think more specifically about students' strategies in proportional reasoning.
Supporting Teachers' Learning: The Role of Instructional Leaders
This framework points to ways instructional leaders—administrators, coaches, and teacher leaders—can better support teachers' learning through data use. After identifying which approach a teacher group seems to be taking, leaders can facilitate conversations to push teachers' thinking toward responsive revisioning, adding the appropriate guiding questions.
Missing the Why
Both targeted reteaching and tips and tricks emphasize what content to reteach and to whom to reteach it but fail to consider why students struggled with the content. In these cases, the instructional leaders can press the conversation toward why by asking questions like these
- What patterns in student responses shed light on how they understood it?
- What makes this problem (or topic) difficult?
- What are some possible sources of confusion on the test items?
Additionally, when teachers share tips and tricks, facilitators can press teachers to explain their strategies through links to students' thinking:
- Why is that teaching strategy a good one?
- What misconceptions does the teaching strategy address?
Questions like these can help teachers uncover the reasons that students struggled with difficult content. Facilitators should also help teachers explicitly tie the sources of students' misunderstandings (the why) back to their proposed instructional strategies (the how).
Missing the How
Likewise, both targeted reteaching and bounded improvement approaches eschew questions related to how content should be retaught. In these cases, the instructional leader can ask questions like these
- How did you originally teach this content? What worked? What didn't work?
- How can we improve on the original approach to address what did not go well?
Additionally, when teachers have identified sources of student struggle without specifying future instruction, an instructional leader can ask questions like these:
- What are the best strategies for addressing these misconceptions?
- What curricular resources can we use to address these misconceptions?
Questions such as these can open the door for teachers to learn from one another. When Coach Duane asked what made Problem 8 difficult, Sherry guessed that the students did not follow her method, but Jodi based her interpretation on the distribution of student responses. Coach Duane's own knowledge of math instruction helped him see Jodi's comment as a more beneficial direction for the conversation. Having coaches who are knowledgeable about curricular resources supports teachers in understanding the how, as when Coach Duane suggested the Orangey problem. Not all school leaders have this expertise, but this framework can help them identify the kinds of instructional knowledge that are needed so they can find effective facilitators.
What, To Whom, How, and Why
Analyzing student data has become a common practice for teachers and leaders, and educators approach this task in many different ways. The most productive approaches address all four guiding questions—what to reteach, to whom it should be retaught, how to reteach it, and why students struggled. It is common for teacher groups to leave out one or more of these questions, but a strong facilitator can encourage teachers to dig deeper into data.
