Many school districts underutilize one of the most powerful and common symbol systems available to them—numbers—to monitor, evaluate, and revise programs and policies. For example, in a school system committed to educating both sexes equally, no one bothers to figure out the ratio of boys to girls enrolled in advanced placement classes in science and mathematics to ensure gender equity. Or, on a larger scale, a school district that significantly increases its professional development budget makes no effort to determine quantitatively whether the extended professional development increases teacher knowledge and morale or improves student achievement.
Imagine instead that district decision makers pay close attention to numerical patterns to help determine how well schools are doing and what they should do next. As a result, schools recognize gender disparity, commit themselves to gender parity, and develop action plans to remedy the problem. In addition, decision makers request that the building-level administrators and teachers continue monitoring the representation and success of boys and girls in mathematics and science courses, charting the results periodically to measure policy effectiveness. They use a similar approach to determine and adjust the district professional development plan. When districts and schools use data to make decisions, they have the makings of a data-driven school culture.
A Data-Driven School Culture
What are the components of a data-driven school culture? What are examples from schools?
In Sound Public Schools (which has 4,500 students, 38 percent of whom are eligible for free or reduced lunch), the school district analyzed 4th grade scores from the high-stakes Massachusetts statewide exam, the Massachusetts Comprehensive Assessment System (MCAS). The district noticed that in each content area (mathematics, science, and English/language arts), writing fluency correlated strongly with overall proficiency. Specifically, the district compared the percentages of students who did not answer open-response questions, which require composing descriptions and explanations, with the percentages of multiple-choice response rates. For each of the three content areas, 35 to 45 percent of students did not even attempt to answer the open-response questions, whereas only 1 to 3 percent did not respond to the multiple-choice items.
This discrepancy prompted the district to examine student performance on other standardized assessments. To further diagnose student readiness for the 10th grade MCAS, Sound administered the Terra Nova exam at grade 9. An analysis of scores showed a close association between reading proficiency on the Terra Nova and subject matter knowledge as measured by the MCAS.
This close association led the school to implement performance assessments in reading and writing to students at each grade level. Teachers score the written work of students on an eight-level literacy scale, enter the information onto spreadsheets for their own analysis, and forward the data to the central office for use in the district's student database. After the reading and test trends are quantitatively documented, teachers provide extra support for students with the weakest skills. This helps teachers meet a crucial goal in the district improvement plan: reduce the lowest level of literacy by 25 percent each year.
Even relatively simple data patterns can suggest instructional responses. Sound's assistant superintendent reported that One major finding of the MCAS analysis was that many of our students were simply not familiar with the vocabulary of the assessment. For example, they didn't know that the word sum meant add. A systemwide list of necessary vocabulary has been prepared for classroom use at specific levels.
This example shows that in a data-driven culture, there is an institutionalized willingness to use numbers systematically to reveal important patterns and to answer focused questions about policy, methods, and outcomes. A central data team relies on input from teachers, who are ultimately responsible for student success. The team and the teachers link to the district's central office so that their work is not separate from other efforts to collect and use data. In other words, a data-driven culture is both localized and systemic.
In addition, a data-driven culture requires a few knowledgeable staff members, supportive administrators, and institutionalized procedures for distributing data-collection instruments, retrieving data, writing reports, and informing decision making. Many important queries require only paper, a pencil, and a calculator. More ambitious projects require desktop statistical software and a staff member who knows how to use it. Happily, such software is becoming more affordable, user-friendly, and powerful.
Regardless of whether the effort is small or large, the school district can become more informed and confident about the progress and impact of program policy and methods. With that knowledge and confidence, administrators, teachers, and stakeholders can respond proactively, rather than reactively, to demands for accountability.
Measuring the Impact of Curriculum Adoption
Beyond diagnostics, when a district wants to measure the impact of a large-scale initiative, it needs a data-driven culture. For example, in the Halogen Public School District (which has 2,500 students, 11 percent of whom are eligible for free or reduced lunch), numbers inform the district's decisions about curriculum and instruction. In monthly meetings, teachers, principals, and curriculum coordinators examine student performance on the MCAS, plan professional development, recommend curriculum revisions, and create strategies to engage parents.
The director of Halogen's elementary mathematics and science curriculum states the need: When committees of teachers began to align current curriculum and instructional practice with state expectations, it became clear that we had significant content gaps and inconsistencies between schools and teachers, and we needed to shift from didactic to inquiry-based pedagogy. In addition, we saw the need for student assessment data that reflected the content and instructional methodology envisioned in the state curriculum frameworks.
In another series of meetings, elementary teachers met in cross-grade clusters to review school- and classroom-level reports, as well as individual student data from the MCAS. Staff then met by grade level to develop plans for revising the curriculum to improve student performance on the MCAS. Middle and high school teachers also met monthly to map strengths and weaknesses of student performance on the MCAS and Terra Nova tests against current instructional content. Teachers then completed a self-assessment survey in which they specified what they teach in their classrooms and what their perceived level of expertise is in particular subjects. The data from the survey helped the district examine the alignment of its curriculum with the demands of the MCAS, locate gaps and redundancies, plan professional development, and recommend changes for content and instructional sequence.
Halogen has turned to its increasingly sophisticated data-driven culture to examine the impact of using a commercial standards-based curriculum in mathematics. Schools within the district are at different stages of implementing the curriculum. In all schools, scores on the California Achievement Test showed significant gains in mathematical reasoning and problem solving. But the district dug more deeply into its analysis by comparing demographically similar schools that were in different stages of the curriculum implementation.
One school has used the curriculum for three years; the other has only recently come on board. In the first school, 50 percent of the students were either proficient or advanced on the grade 4 MCAS, whereas 36 percent of the students at the second school were proficient or advanced. In fact, the first school outperformed the second school in every area of mathematics, with the greatest differences in patterns and relationships, algebra/mathematical structure, and open response.
Information from the Terra Nova test supported the finding that the first school was gaining ground. Mathematical performance in the first school exceeded national averages by a substantial 16 to 17 percent in communication, geometry, number sense, problem solving, reasoning and measurement—areas of instruction frequently neglected in more traditional mathematics curriculums.
Overall, with this kind of sophisticated data, the district is prioritizing its challenges more knowledgeably, allocating its resources more efficiently, and documenting its work to all stakeholders.
Promoting Data-Driven Cultures
Data-driven school cultures do not arise in a vacuum. They need a major motivator and technical and financial support. In Massachusetts, much of this has come from the Department of Education. The MCAS has riveted the attention of school districts because it is a high-stakes exam, it reports results at classroom and student levels, and it requires student-composed answers to its open-response and essay questions. Before the exam was mandated, student-composed answers were not a common instructional focus in Massachusetts schools.
To date, the Partnerships Advancing the Learning of Mathematics and Science (PALMS) project, a National Science Foundation state systemic initiative, has been a major provider of funds and advice for setting up data-driven cultures. Grants to 26 school districts, including Sound, Halogen, and Unified (described below), range from $4,500 to $65,000. Although focused on mathematics, science, technology, and engineering, PALMS support extends to all areas of the curriculum and to the data-driven needs of both the district and the teacher.
No One Right Way
In brief, PALMS finds that data-driven school cultures take hold in one of two ways. In larger school districts, data-driven efforts reside within institutional research and development units. In small districts, data-driven efforts usually begin with the work of one person who has a quantitative bent, enough curiosity to look for patterns, and, most important, a willingness to share results with colleagues and to solicit their views. Occasionally, outside consultants augment these efforts.
Data-driven teams organize in response to different needs and opportunities. For example, some data teams start by enhancing or developing information management systems within their district. Others collect and analyze data related to specific research questions: How do curriculum and instruction relate to student achievement, or what specific professional development is needed? Many districts try to determine which input and process indicators influence student performance in math, science, and technology at the building level (fig. 1).
Figure 1. Establishing a Data-Driven School Culture Checklist
Figure 1. Establishing a Data-Driven School Culture Checklist
What does the district want to know? Where to look:
Current district goals.
Patterns in data.
Upcoming district decisions.
Questions raised by teachers, administrators, or the community.
How will the district find out? What to do?
Form data team.
Conduct inventory of data currently compiled in the district and determine format (electronic or paper).
Assess technological capacity of the district to manage and analyze data.
Determine the extent to which personnel in the district have time, skills, and willingness to engage in data-driven projects.
Identify indicators of input, process, and outcomes variables related to goals.
Determine which additional data are reasonable to collect.
Train staff to collect and use data.
Analyze and disaggregate date.
What does the district do next? How to proceed?
Establish benchmarks and measure progress toward goals over time.
Develop action or school improvement plans.
Communicate findings.
The Importance of Teachers
Teachers are essential members of a data-driven school culture. Much quantitative information in schools derives directly from their work with students. Teachers, in turn, use the analysis of data to improve their practice (fig. 2).
Figure 2. How Teachers Can Be Data Savvy
- Identify questions related to student performance. You are likely to be most interested in information related to your classroom, but schoolwide, districtwide, or statewide patterns can also be informative.
- Identify data and gather necessary information. Take into account demographic information, such as gender, race, eligibility for free or reduced lunch, and language spoken at home. Looking at student responses to individual test questions tells you things that you can't learn from a single test score.
- Examine and use data. Look at student performance in focused areas to target particular groups for assistance. Examine data from previous years. Regardless of how convincing patterns look, do not jump to conclusions. Look beneath the surface and ask more questions.
- Ask useful questions.
- How does performance for individuals and groups related to state standards?
- Is there variation across content areas?
- How does the performance compare with other like groups, such as among students, schools, districts, states and across the nation?
- Are there data trends over time?
- Are there existing initiatives in the school, district, or classroom that might help improve student performance? On the basis of what evidence?
- What are the implications for your instructional practices or for your curriculum?
- Do your findings suggest that you need more professional development?
- How might other stakeholders benefit from this information?
A teacher doesn't need to be a statistician to be data-driven. Simple calculations, such as frequencies, averages, and percentages, provide enough information to answer many important questions. Advanced statistical procedures should be assigned to trained individuals.
An Urban Example
Unified Public Schools (which has 23,000 students, 45 percent of whom are eligible for free or reduced lunch) created a student information management system. The information system currently helps the district address two important issues: It provides quick and tailored responses to building and staff requests for student information; and it enables teachers to design classroom instruction that accounts for the needs of individual students on the basis of their reading, writing, and numeric proficiencies. For example, the district can monitor transience more efficiently and meet the needs of students who move among district schools within the year.
The cornerstone of the system is a student-specific, numerical ID. The ID tracks students through the school system. Each record contains student and parental background, coded as variables, as well as information about attendance, scheduling, grades, and disciplinary action. The new system all but replaces the cumbersome paper-based filing system previously maintained by the district.
At the classroom level, teachers have direct access to individual student data through student performance software. Previously, only district-level data specialists had access to student data and issued reports only when time permitted. The software provides teachers with a comprehensive quantitative profile of each student based on cumulative grade-by-grade records. A Unified social studies teacher who is facilitating this approach with colleagues affirms that Teachers can research areas of difficulty for various students. At weekly meetings, these teachers can target classwide areas of weakness and adjust their classroom instruction to address these problems. At the same time, teachers can address individual problem areas with those students through one-on-one instruction. We have found that making this connection between [software] in the lab and the regular in-class instruction has been very powerful for our students. With the help of software consultants, the district ensures the informed use of student quantitative records by training teachers for the different assessments.
At the building level, the information system produces reports with real-time data. This is extremely important to administrators, who regularly deal with student movement among buildings. The district ensures the requisite building-level expertise by training at least one individual in each school. This approach is powerful because it puts information analysis directly in the hands of those most responsible for making and effecting decisions. This strategy also frees up time for more complex analyses by the two data specialists who are employed districtwide. The fact that Unified is the highest scoring urban district in the state may be due, in part, to its growing data-driven culture.
Future Successes
More than anything else, a data-driven school culture means that the district actively uses quantitative patterns to make decisions, especially those related to programs, curriculum and instruction, and resource allocation. Administrators, faculty, and other key stakeholders support this work by allocating resources to establish and maintain a data-collection system and by training personnel who are as close to the classroom as possible. Typically, data teams carry out special projects related to a larger data system and to districtwide goals. Any district can expect gains in student achievement over time when it becomes data-driven.
