What defines an excellent education is a question for the ages. From the Socratic idea of a life spent in search of the “good” to Samuel Clemens's observation that he “never let his schooling get in the way of his education,” people have always debated the characteristics of a quality education.
The New York City Department of Education, winner of the 2007 Broad Prize for Urban Education and a dependable lightning rod for education reform debate, has recently implemented an intriguing definition of school excellence in the form of its School Progress Report. This collage of measures reflects the complexities of defining and communicating what makes a good school.
New York City's definition of educational excellence is rooted in the nature of the city itself. On the wall of Mayor Michael Bloomberg's office are five clocks—as in some international airport lobby—but instead of the labels New York, Rome, Paris, London, and Moscow, the clocks read Manhattan, Queens, Brooklyn, Bronx, and Staten Island. This is not an insular view of the world. On the contrary, it recognizes and respects the diversity of New York City—from Little Italy to the Indian markets of Jackson Heights.
The city views its 1,400-plus schools no differently. Its overarching vision is to be “a system of great schools, not a great school system.” To that end, the city has made many organizational changes to increase schools' control of budgeting, staffing, and aspects of curriculum and instruction.
When a district increases each school's autonomy to make decisions like these and encourages schools to adapt to such local community needs as rapidly growing populations of English language learners, it shifts its definition of excellence away from compliance and uniformity. Everyone in the district comes to understand that conforming to a single way of running a school is no longer synonymous with excellence.
The city's evolving definition of excellence is expressed by the School Progress Report, which indexes a number of measures of school quality and academic achievement. These include standardized test scores; attendance rates; credit accumulation; graduation rates; results from New York State Regents exams; and perception data from students, parents, and teachers concerning such topics as school safety, respect, and student engagement. The measures are then combined into a single letter grade of A, B, C, D, or Ffor each school. (Go tohttp://schools.nyc.gov/Accountability/SchoolReports/ProgressReportsto review the latest School Progress Report.) Indexing makes it easy to compare performance. One need only line up the schools and look at their index rating, in this case a letter grade, to judge their relative performance.
The challenge of any index is to decide which measures to incorporate and the relative importance to assign each measure. The School Progress Report balances three key measures.
The Three Key Measures
Measure 1: Student Progress
In the 2006–07 report cards, student progress carried the greatest weight; it counted for 55 percent of a school's letter grade. Student progress is a school-level average of academic growth. It measures how each student's academic skills and knowledge have advanced or declined from one year to the next using state test scores at the elementary and middle school levels and number of credits accumulated and Regents exam scores at the high school level.
New York City's approach follows the progress of the same students, also known as amatched cohort of students, from one year to the next. By contrast, No Child Left Behind compares the results of different students from one year to the next in what is often called a cross-sectional analysis. Because no two groups of students are exactly the same, this latter approach is vulnerable to other factors, such as differences in socioeconomic status between groups, which can confound an accurate read on progress. Following a matched cohort appeals to common sense and aligns with best practices in assessment. Also, it's the only way to determine the growth of individual students over time. The same principles apply to current local, state, and federal efforts to develop growth or value-added models.
An additional rationale for placing the greatest emphasis on student progress is that it is a better proxy for school quality than absolute student performance at any single point in time. To use a running analogy, a runner's absolute performance in the New York City Marathon would be the time it takes her to run the race course. Progress, on the other hand, would be how the runner has improved her time from one running of the marathon to the next.
Absolute student performance, such as the percentage of students who are proficient in a given year, may well represent something other than the school's effect on students. Most would agree that socioeconomic status and other nonschool factors have a lot to do with students' readiness to learn. Therefore, high performance on absolute measures may be influenced by what students bring to school rather than what they receive at school.
Measures of student progress should reveal more about the school's contribution to learning. To return to the running analogy, who is the better coach—the one who improves the times of slow runners but still places only fourth in the league, or the coach whose runners don't improve but are already good enough to place first? New York City is placing the highest value on the former.
What organizational message about excellence does this send to the city's schools and communities? That realizing potential is valued above today's absolute performance. It is essentially an egalitarian message; it defines education as the act of extending to every student in each New York City borough, neighborhood, and family the opportunity to realize individual potential in a way that respects and responds to individual identities and needs. What approach could better embrace the diversity of New York?
Measure 2: Student Performance
This emphasis on progress is not without its difficulties. It runs in direct contrast to another concept the United States holds dear—meritocracy. Many believe that accomplishment—or absolute performance—should dominate the calculation of reward and future opportunity. That is why we award the gold medal to the fastest runner.
In the School Progress Report, absolute performance is represented bystudent performance and given a weight of 30 percent in the overall letter grade. This sends a message to schools that they can neither ignore nor depend solely on student performance to achieve a high index rating. This factor represents a large enough portion of the total grade to make it impossible to do very well without having some level of proficiency in the school. However, it is not weighted so heavily that it makes it impossible for a school that is helping lower-performing students improve to also achieve a good grade.
Measure 3: School Environment
The final 15 percent of the School Progress Index is assigned to school environment. Surveys of students, parents, and teachers rate how well the school provides crucial “preconditions for learning” in the areas of safety and respect, academic expectations, engagement, and communication. Attendance rates are also part of the school environment measure.
By including survey results, New York City broadens the types of data that inform the index. Survey data relate to perception and therefore are a distinctly different source from the credit accumulation, standardized test scores, and graduation rates that inform student progress and student performance. The fact that these data originate from school community members also means that the New York City school system is handing over part of the rating process, albeit a small portion, to those closest to the school. The relatively low weight given to school environment also sends a clear message that although they are important, environmental factors are secondary to the student progress and performance they produce.
The School Quality Review
The emphasis on academics as the key measure of school excellence is further emphasized by the fact that a fourth measure of school quality—the School Quality Review—is referenced in the school report card but doesn't carry any weight in the School Progress Index. All New York City schools undergo an onsite quality review conducted by an experienced educator. The purpose of the review is to measure “a school's progress in developing a culture of data-driven continuous improvement.”
Schools are rated on five “quality statements” that focus on the school's capacity to use data to meet the individual needs of its students. For example, the first quality statement reads, “Gather data: School leaders and faculty consistently gather data and use it to understand what each student knows and is able to do and to monitor student progress over time.” Here, too, the New York City school system has designed a process that is well aligned with its vision. The School Quality Review is intentionally neutral about a school's particular instructional approach. The focus is on how well the school can adapt to its students' needs through whatever approach might prove most promising. Like the overall focus on growth in the School Progress Report, the School Quality Review fosters the capacity to innovate in response to student needs.
Toward a New Model of Excellence
Has New York City struck the perfect balance in its definition of excellence? It is far too early to tell, but according to a November 2007 poll by Quinnipiac University, in Hamden, Connecticut, “By a 61–27 percent —, New York City voters say the recently issued report cards grading public schools A to F are fair. Parents with children in public school, who know the grade given their child's school, say by a 75–21 percent — that the grade is fair.” This is good news in the School Progress Report's first year of implementation.
Reactions to the new report cards have varied. Parents whose children attended a school that has always shown high absolute performance and little perceived need to further improve student achievement were often not satisfied with their school's letter grade. Some parents and community members were surprised to find other schools with less impressive rates of proficiency receiving higher marks than their school.
As the New York City school system shifts the meaning of excellence, it is not surprising that some schools and some parents are less comfortable than others. Any district that seeks to clearly define excellence will make the public think about what matters in education. A little pushback is a healthy sign that dialogue is occurring.
To mitigate concerns from schools that have long shown high absolute performance, the 2007–08 School Progress Reports provide both an overall grade and a grade for each aspect of the index (progress, performance, and environment).
Philip Vaccaro, the city's manager of progress reports, emphasizes that the process used to solicit feedback and refine the School Progress Report index has been a crucial factor in its success. The first report cards were piloted, with feedback gathered from principals and other stakeholders, before the report cards were made public in November 2007. Since then, Vaccaro has given presentations to scores of community groups around the city, collected feedback from thousands of e-mail messages, and conducted statistical analyses to test the inner workings of the index.
Beyond the mechanics of its index, the greatest strength of New York City's approach to defining excellence may be the cultural message embedded within it. By focusing on student progress, feedback from stakeholders, and data-driven decision making, the approach fosters the key behaviors necessary to create excellence in a diverse mix of schools, each aligned to the needs of its community.
It is ironic that New York City's emphasis on progress is expressed in a letter grade. Although it takes advantage of shared cultural experiences, the use of a letter grade is an old paradigm that connotes absolute performance as much as it does progress. As a nation, we need to push for new mental models to define excellence in education.
