Ten years ago, a group of 19 researchers and practitioners from the Harvard Graduate School of Education and Boston Public Schools developed a process for organizing the core work of schools. That process allowed teachers to collaboratively study a wide range of evidence and use what they learn to improve instruction. Since then, schools around the world have used this "Data Wise" process to drive improvement, with much of that work happening in meetings (see fig. 1).
Figure 1. The Eight-Step Data Wise Improvement Process
Source: Data Wise, Revised and Expanded Edition: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning (p. 5), edited by Kathryn Parker Boudett, Elizabeth A. City, and Richard J. Murnane, 2013, Cambridge, MA: Harvard Education Press. Copyright © 2005 by The President and Fellows of Harvard College. Reprinted with permission.
As we've supported schools in using this process, we've heard from educators that there is often much room for improvement in the way school meetings are planned and facilitated. So we've field-tested strategies for improving the quality of meetings—especially those in which educators analyze data.
One key insight has been that for a meeting to be effective, everyone involved must know and understand the meeting's purpose. As noted in Meeting Wise: Making the Most of Collaborative Time for Educators, it's appropriate for educators in meetings "to feel challenged by the exchange of ideas and the weight of the objectives, but not to feel confused by a lack of clarity about purpose, process, or next steps."
Here's where the Data Wise Improvement Process helps teams use their collaborative time well. Each of the eight steps has a clearly stated purpose, and each is a key part of a process that drives collaborative inquiry over time (see fig. 2). Data Wise clarifies the central purpose for each of a team's meetings—and focuses discussion and decision making during that meeting.
Figure 2. Purpose of Each Step of the Data Wise Improvement Process
Eight Steps to Becoming Data Wise-table
Step | Purpose of this Step | How This Step Played Out at Highland Academy |
|---|---|---|
| 1. Organize for collaborative work. | Establish structures and teams. | Schoolwide meeting agenda template and norms |
| 2. Build assessment literacy. | Increase comfort with data. | Professional development on interpreting assessment reports related to literacy, the schoolwide focus area |
| 3. Create data overview. | Identify a priority question. | "How do students approach finding the main idea in literature?" |
| 4. Dig into student data. | Identify a learner-centered problem. | "When answering questions about literature, students tend to zoom in on characters and their feelings about them without stepping back to consider the main idea of the story." |
| 5. Examine instruction. | Identify a problem of practice. | "As teachers, we tend to 'give away' the main idea of a story at the beginning of a lesson and devote most class time to encouraging students to identify personal connections to the characters." |
| 6. Develop action plan. | Create an action plan. | Instructional Strategy: Close analytic reading |
| 7. Plan to assess progress. | Create a plan to assess progress. | "Short-term: In-class presentations Medium-term: Teacher-designed written assessments Long-term: State English Language Arts assessment" |
| 8. Act and assess. | Document improvements in teaching and learning and adjust as needed. | After implementing the instructional strategy, teachers noticed that students improved in their ability to identify the main idea orally but struggled to capture it in writing. Teachers continued to adjust their instruction, and by the end of the year most students were proficient in the "main idea" sub-skill on the state test. |
How One School Used the Process
To illustrate this step-by-step approach, let's follow a school that we'll call Highland Academy. Our description of Highland's data analysis cycle is a composite case study drawing on our experience working with educators in various settings.
Step 1. Organize for collaborative work.
The base of the curving arrow in Figure 1 is extra wide because substantial foundational work must take place as a school prepares to engage in data inquiry. The purpose of Step 1 is to establish collaborative teams and structures that will enable educators to work together productively. This step involves adopting an improvement process, building a strong system of teams that communicate efficiently, and protecting time throughout the year for these teams to work. It also includes clarifying expectations for effective meetings, agreeing to norms for collaborative work, and acknowledging work style preferences. Finally, it entails creating a data inventory and an inventory of all the instructional initiatives already underway at a school, because no improvement effort happens in a vacuum.
At Highland Academy, a subgroup of the instructional leadership team was responsible for organizing data so that it was easily accessible and understandable to teachers, but the real data-analysis work was expected to happen in weekly grade-level team meetings. To ensure that those meetings were productive, the principal and her instructional leadership team kicked off the year with a full-day retreat before students came back to school. Throughout the day, faculty experienced a series of meetings that modeled the kind of structure and facilitation they would be expected to use in the coming year as they worked with data.
One of the day's most important activities was building shared meaning around three norms for interaction: assuming positive intentions, grounding statements in evidence, and taking an inquiry stance. Just as it sounds, assuming positive intentions means that each educator assumes every other member of the group is acting out of a desire to improve teaching and learning. Staying grounded in evidence means that colleagues rely on facts and low-inference information to guide their work (rather than previously held assumptions); taking an inquiry stance means that educators regularly ask one another questions—to understand, clarify, and stay grounded in evidence.
Teachers described what each norm would look like and feel like if they used it in daily interactions. They did a role-play to try out different ways they might hold one another accountable for following these norms. Throughout, the message was clear: Adhering to norms would help build the trust needed for the candid discussions about practice that a data-based improvement process entails.
Step 2. Build assessment literacy.
The purpose of this step is to increase staff members' comfort with the kinds of data they will be using throughout the inquiry process. Key tasks here involve reviewing the skills that will be tested on the assessments students will take and considering how these skills compare with the broader domain of skills and knowledge students need to master. Teachers also need to learn the principles of responsible data use and to practice studying assessment results.
The schoolwide focus for improvement at Highland was literacy. In the September faculty meeting, leaders facilitated grade-level team learning sessions in which teachers practiced reading and interpreting the literacy assessments their students would take. The goal of these sessions was to help teachers develop a common language to describe the kinds of inferences each data source would be best able to support. Teachers broke into groups, with each group assigned to look closely at one assessment and discuss what information it would give about student learning.
At one table, for example, teachers looked at a screening test that the school commonly used to place students in leveled reading groups. When they studied the assessment reports, they realized that the test did not provide enough detail to allow for truly strategic grouping of students with specific learning challenges.
Another set of teachers looked at a writing assessment. Although they agreed that it would allow them to make valid inferences about students' ability to construct grammatically correct sentences, they determined that it did not provide enough information to allow them to assess student progress in developing compelling arguments.
At the end of the faculty meeting, teachers at each table shared what they had learned about their assessment. Then the whole group engaged in a frank discussion about what the literacy assessments they currently used could—and could not—tell them about student learning.
Step 3. Create a data overview.
At this step, collaborative inquiry really begins as a broad faculty group identifies a priority question that members are committed to exploring. Typically, a small group of educators, such as the leadership team, conducts a thorough analysis of recent data pertaining to a focus area and finds a pattern—or "story"—they believe is important for the entire faculty to think about. They display the data in a few charts that make it easy for their colleagues to see the story. School leaders then engage teachers in making sense of the charts and identifying a specific question they want to dig into.
At Highland Academy, for instance, the leadership team culled through reports showing recent literacy assessment results and found a puzzle that they believed would lead to important conversations. They prepared a series of bar graphs that showed the percentage of students, by grade level, in the advanced, proficient, and warning levels on the state English language arts assessment, disaggregated by subskills, which included comprehension, vocabulary, finding the main idea, and identifying key details.
The leaders presented this data overview, and teachers discussed it in small cross-grade groups using the "I Notice, I Wonder" protocol. Having teachers start by making low-inference statements about what they noticed helped them practice the norm of grounding statements in evidence. Many teachers noted that they observed consistently low performance on "finding the main idea" questions for fiction texts, although students didn't have difficulty identifying the main idea of nonfiction texts. They then engaged in several rounds of wondering aloud about the results.
The purpose of the meeting wasn't to tell teachers what the data meant, but to allow them to express curiosity about why the data looked as they did and develop a sense of urgency to find out the answer. The priority question Highland teachers settled on was, "How do students approach finding the main idea in literature?"
Step 4. Dig into student data.
Once a priority question is chosen, the purpose of subsequent meetings is to identify a learner-centered problem that directly relates to that question. This involves examining and analyzing a wide range of data in the target area, including student work samples, performance on benchmark assessments, observations of students, or conversations with them about their learning. From these sources of evidence, teachers come to a shared understanding of what the data show about students' learning and identify a common learning challenge.
When digging into student data, many learning challenges often surface. Instead of getting hung up trying to find "the" learner-centered problem, the trick is to select "a" learner-centered problem that, if solved, would be an important step forward.
Working in grade-level groups, the Highland teachers continued to meet regularly to look closely at student work samples. They examined the kinds of questions teachers asked students—both in class and for homework—and how students answered them. Those observing in primary grades noticed that when students were asked to draw pictures or write simple sentences that captured the main idea of a story, the students focused their pictures and sentences on the primary characters but didn't necessarily communicate the main idea or theme of the story itself. Similarly, teachers in upper grades noticed that students tended to describe the protagonists' character traits rather than talk about themes. When they did address the main idea or theme, students used a limited vocabulary.
Ultimately, the faculty settled on the following learner-centered problem: "When answering questions about literature, students zoom in on characters and their feelings about them without stepping back to consider the main idea of the story."
Step 5. Examine instruction.
Once data teams begin to examine instruction, their main objective is to articulate a problem of practice that may be contributing to the learner-centered problem. Key tasks at this point include examining a wide range of instructional data (including lesson plans, assignments, and assessments) and observing teachers in classrooms. Teacher teams conduct these observations. Although administrators may sometimes participate, when they do so, it's always with the understanding that the objective isn't to evaluate a teacher's practice for accountability purposes but to reach a shared understanding of what's happening in classrooms.
At Highland, several teachers videotaped their reading lessons. Then teachers watched portions of each video with an eye to understanding why students usually didn't articulate the main idea of stories. At first, it seemed paradoxical: In every video, the main idea of the story was an important lesson topic. But when they analyzed what teachers and students were doing and saying, they noticed that teachers opened their lessons by summarizing the main idea of a story instead of challenging students to identify it. Lessons often included strategies for helping students identify literary devices and make inferences about characters' motivations—but not for helping them tackle big themes.
Teachers acknowledged that it felt easier to draw students in by having them get to know characters and explore their personalities. The faculty didn't want to lose these valuable parts of their teaching, but they wanted to find a better balance to ensure that students could also articulate main ideas. So they pinpointed this problem of practice: "As teachers, we tend to 'give away' the main idea of a story at the beginning of a lesson and devote most class time to encouraging students to identify personal connections to the characters."
Step 6. Develop an action plan.
At this point, educators create a complete, concise action plan for addressing the problem of practice. This work involves deciding on an instructional strategy, agreeing on what that strategy will look like in classrooms, and putting the plan in writing.
The Highland faculty researched instructional strategies for increasing students' ability to truly comprehend what they were reading. Although they had kicked off their inquiry cycle by looking at state test data, they were determined to select a strategy that would lead to the development of engaged and critical readers—not just a bump in test scores. In fact, they believed that if they continued to engage students as active participants in explaining the meaning of stories, students' comprehension would improve overall. So they chose a strategy called close analytic reading, which promoted student agency in investigating and expressing main ideas.
These educators knew that simply naming the instructional strategy and recommending it wouldn't be enough to ensure teachers would implement it well. They worked with a literacy coach to determine what kinds of professional development they would need in order to do close analytic reading regularly in class. In their grade-level data teams, they created a simple action plan table that clarified who would be responsible for doing what and by when (see fig. 3).
Figure 3. Highland Action Plan for Improving Instruction
ACTION PLAN
Learner-centered problem: When answering questions about literature, students tend to zoom in on characters and their feelings about them without stepping back to consider the main idea of the story.
Problem of practice: As teachers, we tend to 'give away' the main idea of a story at the beginning of a lesson and devote most class time to encouraging students to identify personal connections to the characters.
Instructional strategy: Close Analytic Reading
Eight Steps to Becoming Data Wise
Task | Who | When |
|---|---|---|
| Work with literacy coach to learn close analytical reading strategies and practice using them. | Grade-level team with literacy coach | Week of Oct. 19 |
| Draft a lesson plan that incorporates close reading strategies. | Grade-level team leader | Oct. 26 |
| Provide feedback on draft lesson plan. | Grade-level team | Week of Nov. 2 |
| Teach lessons that incorporate new strategies and videotape classes. | Each team member | Nov. 3 – Nov. 17 |
| Watch videos of team members teaching lessons that involve close analytic reading and discuss instructional adjustments. | Grade-level team | Nov. 23 |
| Review student work and discuss instructional adjustments. | Grade-level team | Nov. 30 |
| Adjust instruction, extend insights to other lessons. | Each team member | December |
Step 7. Plan to assess progress.
The purpose of meetings at this stage is to identify the short-, medium-, and long-term data sources teachers will use to evaluate how the changes they implement will affect student learning. This work includes setting student learning goals for each type of assessment.
At Highland, teachers realized they'd need to generate their own informal assessments to quickly gauge how their first attempts were affecting student learning. For short-term data, they decided to have pairs of students give classroom presentations about the main idea of a story, with the goal of seeing whether students could orally describe a main idea.
For medium-term data, teachers created three assessments that challenged students to express the main idea of a story in writing. They set a goal that by the third assessment, all students would successfully write about the main idea in a way that met grade-level standards.
For long-term data, teachers decided to return to the same state assessment that had been featured in the initial data overview. They set a goal that all students would be proficient in this test's subskill of finding the main idea for both fiction and nonfiction texts.
Step 8. Acting and Assessing
Now it's time for team members to carry out the action plan and for teachers to assess the extent to which they are doing what they committed to, and whether student learning goals are being met. Inevitably at this stage, the team must make adjustments to the action plan or the instructional strategies; once those adjustments have been made and teaching and learning are clearly improving, it's time to celebrate the success of using data to fuel change!
After Highland students had practiced close analytic reading and been challenged to find the main idea of a story, teachers were pleased to see that they became more confident in offering ideas for a text's main idea. Students' confidence was assessed not only by teacher impressions during class, but also through conversations with students and observation data from teacher peers. Initially, students' ability to describe that idea in writing using appropriate literary vocabulary was weak. In team meetings, and with guidance from the literacy coach, teachers realized they needed to include more direct instruction on how to write about the main idea of a story in a compelling way. Both the principal and the literacy coach supported teachers by reviewing lesson plans, observing instruction, and engaging in reflective dialogue with teachers about their instructional practice.
To document their journey through the improvement process, each teacher team concluded the inquiry cycle by creating a slide presentation that demonstrated how students and teachers had improved their work, and they shared it with parents, students, other teachers, and administrators.
Making the Most of Meetings
Throughout their inquiry work, the Highland data teams held purposeful meetings that eventually raised student achievement—and helped teachers evolve into a learning community. Purposeful is the operative word. The Data Wise process, with its deliverables that clarify the purpose of meetings at each step and its emphasis on collaboration, provides schools with actionable strategies to support doing such work.
Authors' note: For more information on the Data Wise process, see Data Wise, Revised and Expanded Edition: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning by Kathryn Boudett, Elizabeth City, and Richard Murnane (Harvard Education Press, 2013). The Data Wise project has released a massive open online course (MOOC) available free through www.edX.org that explores many of the ideas contained in this article.
