Data-driven decision making is here to stay. Throughout the last decade, educators have come to embrace data as an indispensable tool for school improvement. Like our colleagues in industry and medicine, teachers have learned that data help us identify priorities for improvement. When schools formally measure and publicize their weaknesses, addressing problem areas acquires new urgency.
The marriage between the data-driven movement and No Child Left Behind instigated many positive actions, especially on behalf of low-performing subgroups. As Tom Peters (1987) wrote two decades ago, "What gets measured gets done." But data-based reform has also had less happy consequences. In many schools, it has morphed into an unintended obstacle to both effective instruction and an intellectually rich, forward-looking education.
For data-driven instruction to transform schooling—which it can—it must serve a master very different from rigid accountability formulas. It must aim to help students from all backgrounds attain an authentic 21st-century education. A good model is the New York Performance Standards Consortium, a consortium of 28 high schools in New York State that uses data-driven decision making in service of 21st-century learning, with both simplicity and success. Advocates of data-driven instruction can learn much from this consortium. But first, let's look at what a 21st-century education requires and whether most schools are using data to help or hinder this kind of education.
What Is a 21st-Century Education?
The most prominent advocates of 21st-century education all stress the importance of learning essential content by way of authentic intellectual skills. These advocates' documents invariably contain the terms critical thinking andproblem solving. They emphasize the ability to argue, analyze others' arguments, conduct research, and acquire such "habits of mind" as the ability to invent or synthesize information. Literacy—the ability to read, write, and make effective presentations—is central (Berliner & Biddle, 1995; Kay & Houlihan, 2006; Pearlman, 2006).
College professors and business executives whom I've interviewed concur that such skills are crucial to students' success as college freshmen, professionals, and participants in a democracy. Yet these leaders uniformly lament that K–12 schooling does not emphasize such an education. This raises the question, Does the focus on data now present in public schools promote 21st-century teaching and learning?
When Data Interferes with Improvement
To answer this question, we must realize that our current data-driven decision making is to a great degree standardized-test-data-driven decision making. This is not all bad. As W. James Popham (2001) writes, standardized test items can operate as proxies—imperfect but useful indicators of legitimate learning. Indeed, when higher test scores are the result of sound curriculum and effective teaching, we should applaud these gains. There are many schools that use data to drive a curriculum rich in 21stcentury skills and that achieve both high test scores and deep learning. Tempe Preparatory Academy in Phoenix, Arizona, and View Park Preparatory in Los Angeles, California, (a majority-minority school with half its students living below the poverty line) are two prime examples (Hernandez, Kaplan, & Schwartz, 2006).
But as those of us who spend significant time observing in schools have seen (Schmoker, 2006), few schools truly focus on these key skills. Just as education leaders were beginning to win hearts and minds to the importance of authentic intellectual tasks and assessments, our attention was turned—too much, perhaps—to the numbers themselves. And we made an unpleasant discovery: Schools and even whole states could make steady gains on standardized tests without offering students intellectually challenging tasks.
Steadily improving scores didn't alert leaders to this paradox. In literacy, for instance, many of us assumed that data-based accountability would spur a general increase in reading and writing assignments relevant to 21st-century concerns in all disciplines. We believed that such assignments would promote the ability to write and think critically, evaluate, and discern facts from opinions. After all, these skills abound on state standards documents and corresponding assessments (Liben & Liben, 2005). Authentic literacy learning did increase in some schools. Standardized test data reflected this reality, and we should celebrate that fact (DuFour, DuFour, Eaker, & Many, 2006; Reeves, 2006; Schmoker, 2006).
But we can't discount evidence that fixating on data hindered instructional improvement in many schools. Teachers I've talked to in dozens of states admit that the quality of what they teach and how they teach hasn't changed that much since data grabbed the focus. Test-prep activities—not authentic teaching and learning—were responsible for much of the increase in test scores.
This explains why achievement gains on state tests are often at odds with stagnant performance on the National Assessment of Educational Progress (Cavanaugh, 2007). It explains why higher passing rates on standardized tests have had little effect on the high proportion of students who enroll in remedial college courses (Fitzhugh, 2007; Kollars, 2008).
In the last few years, at the invitation of school administrations, I have observed in several schools with good reputations. I found that in most classrooms, ill-devised lessons, aimless group activities, and busywork predominated. Worksheets were everywhere, and movies blared through school hallways. There was almost no emphasis on critical thinking, problem solving, reading, discussion, or writing. Ironically, faculties showed little interest in improving instruction because each of these schools enjoyed exceptional test scores, and some had received their respective state's highest rating for academic achievement. The data itself created a ceiling on instructional improvement.
A More Forward-Looking Approach
Fortunately, forward-looking institutions like the New York Performance Standards Consortium have begun using data to support instruction for authentic learning. Because consortium schools operate on a waiver, students only have to take one of the state's five Regents exams (the English exam). Regents exams determine whether New York high school students receive credit for core courses and graduate. Thomas Sobol, then New York State's commissioner of education, granted the waiver in 1991 because after observing the success of exemplary schools, including those in the consortium, he believed such schools could be even more innovative if they were relieved of the Regents requirements. With test-score pressure eased, these schools are free to concentrate on measuring the kinds of skills the world beyond the classroom demands.
Instead of test prep, students and teachers focus on work that culminates in four or more final projects in core academic areas: (1) a literary analysis, (2) a science experiment and related research project, (3) an extended mathematics problem-solving project, and (4) a research paper in social studies demonstrating the use of argument and evidence.
All projects require students to read, think critically, write, discuss, do research, construct an argument, and publicly present their knowledge. The intellectual richness of these projects is evident in sample titles, such as "Why Do They Have to Die? A Comparative Analysis of the Protagonists' Deaths inDr. Jekyll and Mr. Hyde, Metamorphosis, and Of Mice and Men" and "Finding the Parabolic Path of a Comet as It Moves Through the Solar System." A set of rubrics accompanies each project-related task. The entire process is informed by Wiggins and McTighe's (2005) notion of "backward design."
The consortium encourages any school to liberally borrow from its practices. The consortium's Web site (http://performanceassessment.org/performance/index.html) features links to rubrics used in consortium schools, guidance in implementing performance assessment, and pacing charts and planning documents (under the Forms and Charts link).
Rubrics listing criteria for satisfactorily accomplishing tasks like analyzing a literary work and performing scientific research were developed by consortium faculty, college professors, and local professionals—who also identified exemplar "anchor" projects. This group helps evaluate whether each student's work meets graduation requirements.
- Information on how many students are on track to successfully complete the major projects required for graduation and how many need guidance.
- Data on students' individual performance on key rubrics.
- For a science experiment: Hypothesis reflects a synthesis of primary background research. Creates graphs and charts that reflect the use of basic algebraic functions and multiple statistical tests.
- For a social studies paper: Includes detailed evidence drawn mainly from primary sources. Evaluation of substantial number of opposing/varied sources.
How Data Drive Consortium Teaching
Consortium schools have a process to ensure that their focus on rubric data drives instructional improvement. Every summer, a group of representative teachers, college professors, and other professionals conduct "moderation studies" of 150 sample projects representing each school. This group determines whether these projects as a whole meet the rigorous standards of the anchor projects and pinpoints where schools meet or fall short of criteria. The group gives feedback to individual schools so that they can adjust their teaching.
This process revealed to Urban Academy, for example, that its students' written work needed to contain more analysis and evaluation. According to the school's codirector, Ann Cook, "Some papers contained too much material that was merely summative, but not sufficiently analytical." Daniel Drmacich, principal of consortium member School Without Walls, which has participated in moderation studies for two years, has found that this focused review gives teachers an extra push and helpful direction in making student work come up to higher levels:Teachers all know that it is not only their students' work [that] reviewers are reviewing, but also their own. There's a subtle feeling of tension among teachers as we do the moderation studies because they feel pressure to make sure their kids are creating quality work. Almost all staff have interpreted this undercurrent as positive in their quest for helping students grow and develop.
Data review goes on continually during the school year. Teacher teams meet weekly to evaluate student work and compare it to both the anchor papers and graduation requirements. Academic departments at each school conduct interim assessments three times a year to ensure that the quality of student work is consistent with established criteria. Unlike typical end-of-year reviews of test scores, these analyses provide critical feedback on areas in which students are—and aren't—performing well enough to ultimately succeed on their graduation projects.
During such a team meeting, several teachers at Drmacich's school learned how a colleague was smoothly integrating the final academic projects required for consortium schools into her class's overall academic units, rather than adding these projects on as something extra. Inspired by her planning, this teacher's colleagues began integrating the social studies research paper and literary analysis projects into noncore classes, including courses in art and health.
Consortiumwide, this process of measuring 21st-century skills and using data to fuel change has had fortunate results. Although the percentage of students receiving free or reduced-price lunch in consortium schools (more than 60 percent) is higher than average for New York City schools (Foote, 2007), the schools' dropout rate is 9.9 percent, compared with 19.3 percent in New York schools overall. Ninety-one percent of consortium students are accepted into college compared with 63 percent in the city as a whole. According to Martha Foote, the consortium's research director, graduates report that once they get to college they are more competent in writing and revising than their classmates are (Olson, 2006).
Expanding the Model
Following the consortium model, we could easily create a system through which schools continue to administer standardized tests but subordinate the focus on test scores to assessments of 21st-century skills modeled on the consortium's rubrics or other effective systems. There is now consensus among educators that students are able to read, write, argue, and evaluate from the earliest grades. Schools could assess competence in such skills through what Grant Wiggins (1998) has called "educative assessments" in the final grades of elementary and middle school, not only in core courses, but also in art, music, and languages. For other grades, we could design performance-based assessments that show students have mastered relevant skills, and we could nurture these skills throughout the year.
Rhode Island has launched an assessment program similar to the New York consortium's, demonstrating that states can convert their creaking, 20th-century systems into systems promoting what students need now. We must harness the power of data to make school more interesting and relevant—and to prepare students to succeed as workers, college students, and informed citizens.
