In the United States, three presidents, the National Governors Association, national teachers unions, and numerous blue ribbon panels have called on states to implement standards-based reforms: content standards for core subjects, assessments aligned with standards, and accountability mechanisms tied to assessments for ensuring that students achieve standards. In 1999, 18 states required students to pass minimum competency exams for graduation; 14 states rewarded successful schools monetarily; 19 states had special assistance programs for failing schools; and 16 states had the power to close down, take over, or reconstitute failing schools (Gerald & Boser, 1999).
Disagreements about which reforms work—or even whether they are necessary—absorb the attention of educators and the public, especially on the question of how to assess student achievement. A close examination of how students have fared under a variety of assessment and accountability systems can offer some insights.
External Assessments
One assessment strategy—the curriculum-based external exit examination—has shown some promising results. These exams are designed by education authorities, not by teachers, and are administered at the end of specified courses.
- External assessments cover all or almost all students.
- Test results have real consequences for students and schools.
- The assessments define achievement compared to an external standard, not compared to other students in the classroom or the school. School reputations will depend less on parents' socioeconomic status and more on student success on these examinations; school reputations can have major consequences for administrators and staff, especially when students are able to choose which school they attend.
- The tests assess what students are expected to know and be able to do. The test designers expect that students will learn important material and develop valued skills as a result of studying for the exam. Teachers evaluate dimensions of student performance that cannot be reliably assessed by an external exam, such as the ability to write a research paper and make oral presentations.
- The education authority that establishes the K–12 curriculum and funds K–12 education coordinates changes in both instruction and exams, thereby facilitating curriculum reform. Unlike the SAT-I and the ACT, which sort applicants by aptitude, these exams reward the achievement of students who have learned what schools are teaching.
Benefits of External Exit Exams
- The results of an external exit exam report multiple levels of achievement in each subject. In contrast, a minimum competency exam that generates only a pass-fail grade when passing is necessary to graduate requires standards low enough to allow almost everyone to pass after multiple tries. Such low standards will not stimulate the majority of students to greater effort.Exit exams, on the other hand, report each student's achievement level in the tested subject. As a result, all students, not just those at the bottom of the class, have an incentive to do well on the exam. The prospect of exit exams is more likely to improve classroom culture and all students' motivation to study.
- The external exit exams assess more difficult material. Because they are supposed to measure the full range of achievement in the subject, these exams contain more difficult questions and problems. As a result, teachers spend more time on cognitively demanding skills and topics.Minimum competency exams, by contrast, usually identify only those students who have failed to meet a rather low standard, so the tests do not include questions or problems that students near that borderline are unlikely to be able to answer or solve. These low expectations may result in too much class time being devoted to practicing low-level skills.
- The curriculum-based external exams also function as end-of-course exams. Because each test assesses the content of specific courses, the teachers of a course or course sequence will inevitably feel responsible for how well their students perform on the exam. Grades on these tests can be a part of the overall course grade, a policy that further integrates the external exam into the classroom culture, maximizes the alignment between instruction and assessment, and enhances accountability. Proponents argue that teachers will want to set higher standards and that students will be more attentive in class and more likely to complete demanding homework assignments. Teachers drop their traditional role as evaluators of student success and instead join the students' team as coaches who help the team succeed on the state exam.
The Success of Curriculum-Based Exit Exams
Analysis of data from the Third International Mathematics and Science Study (TIMSS) and the International Assessment of Educational Progress indicate that 13-year-old students from countries with curriculum-based exit exam systems are ahead of students from other countries at a comparable level of economic development—including the United States—by 0.67–2 grade levels in mathematics, science, geography, and reading. Differences show up even among jurisdictions within the same country: Students in Canadian provinces with diploma exams in math and science, for example, are one-half year ahead of comparable students studying the same subjects in other provinces (Bishop 1997, 1999; Woessman, 2001).
Other studies (Bishop, 1997, 1999) have examined the impact of these exams on school policies and instructional practices and have found links between curriculum-based external exit exams and higher minimum standards for becoming a teacher, higher teacher salaries (30–34 percent higher for secondary school teachers), and more teachers who have majored in the subject that they are assigned to teach. Schools whose students take these exams also better equip science labs, devote more hours to math and science instruction, and provide after-school tutoring to more students.
Fears that such externally set exams have diminished the quality of instruction appear to be unfounded. Quizzes and tests are more common, but in other respects pedagogy is no different. Surveys of students in science classes (Bishop, 1997, 1999) show that those who undergo these exams are less likely to say that memorization is the way to learn the subject and are more likely to have worked on experiments in science class. Students are no less likely to enjoy the subject, and they are more likely to agree that science is useful in everyday life. Students also talk with their parents more often about schoolwork and are more likely to report that their parents have positive attitudes about the subject.
Accountability Systems in the United States
What do these positive findings regarding the effects of curriculum-based external exit exams in other countries suggest about the likely success of such accountability systems in the United States? Although a number of states—Maryland, Virginia, Tennessee, and Michigan—are considering implementing curriculum-based external exit exams, only two states—New York and North Carolina—actually implemented them during the 1990s, including the exams in a policy mix with other student accountability measures, such as minimum competency exams.
Most states pursuing standards-based reforms have set up other kinds of school or student accountability programs.
Impact on College Enrollment and Job Success
Proponents of standards-based reforms often cite improvements in student performance on statewide tests as evidence that school accountability initiatives are working. Opponents disagree. Test scores have gone up, they say, because test preparation is displacing the teaching of other skills and knowledge that are more important to succeed in college and in jobs. The opponents' assertion poses a testable hypothesis. We examined a representative sample of former 8th grade students and measured the effects of accountability systems on students' subsequent college enrollment and success in the labor market after high school (Bishop, Mane, & Bishop, 2000; Bishop, Mane, Bishop, & Moriarty, 2001).
Our primary data set for this part of our study was from the National Education Longitudinal Study of 1988 (National Center for Education Statistics, 2000), which provides six years of longitudinal data on 14,000 students who were 8th graders in 1988. Family background is a powerful predictor of high school completion, academic achievement, college attendance, and labor market success, so our analyses included controls for many socio-demographic characteristics. We also controlled for the characteristics of the high school and the community, such as type of school, teacher salary, pupil-teacher ratio, mean 8th grade test scores, and ethnic and socioeconomic composition of the student body.
Effects on college attendance. Except for one student category, the students who came from states with minimum competency exams or external exit exams were more likely to attend college six years later. States with minimum competency exams showed a significant increase (2–4 percent) in the percentage of 8th graders who were attending college six years later; this increase showed up whether the students' grade averages had been A, B, or C when they were in 8th grade. For students from New York's exit exam system, the college attendance rate of students who had C averages in 8th grade was significantly higher—6 percent—than the rates for comparable students in other states, whereas the college attendance rate of New York students who had A averages in 8th grade was about the same as the rates in other states.
Effects on labor market success. Controlling for high school completion and college attendance, students who attended high school in states with minimum competency exams earned significantly more—9 percent more in the calendar year following high school graduation—than students in states without minimum competency exams. We can infer that employers in these states responded to the enhanced reputation of high school graduates by paying them significantly higher salaries.
Impact on Test Scores
Test score gains. To measure the relative impact of different reforms on academic achievement, we first analyzed test score gains that the students in the study made between 8th grade and 12th grade on the 1988 National Educational Longitudinal Study achievement tests (Bishop, Mane, & Bishop, 2000; Bishop, Mane, Bishop, & Moriarty, 2001) (see fig. 1). We compared the test-score gains of students in states with different graduation requirements: states requiring completion of a certain number of Carnegie unit courses; states requiring minimum competency exams; and New York's policy mix of state minimum competency exams or voluntary external end-of-course exams. Although we used achievement tests different from those used by the state accountability systems being evaluated, we were able to identify significant trends.
Figure 1. Effects of the State Graduation Requirements on Test Score Gains
New York's students showed the most significant gains. Test score gains from 8th to 12th grade were nearly 40 percent of a grade-level equivalent greater in New York. This success confirms earlier findings that New York students performed significantly better on SAT tests and on the 1992 8th grade National Assessment of Educational Progress (NAEP) math tests than did students in other states with demographically similar populations (Bishop, Moriarty, & Mane, 2000).
Students' test score gains in states with minimum competency exams showed some improvement compared to gains in other states, but the most striking finding was that state-imposed Carnegie unit requirements had no effect whatsoever on test score gains during high school. Simply requiring students to take certain courses does not improve their level of achievement.
Test scores. We then analyzed the effects of several standards-based reform initiatives on 8th graders who took the NAEP tests in 1996 and 1998 (Bishop, Mane, & Bishop, 2000; Bishop, Mane, Bishop, & Moriarty, 2001) (see fig. 2).
Figure 2. Effects of Standards-Based Reform Initiatives on NAEP 8th Grade Test Scores
The policy that clearly had the biggest effect on test scores was the combination of end-of-course exams and minimum competency exams that has been in place in New York since the early 1980s and in North Carolina since about 1991. In comparison to students in states without these exams, 8th graders in New York and North Carolina were about 45 percent of a grade level ahead in math and science and 65 percent of a grade level ahead in reading.
The next most effective intervention was state-imposed stakes for teachers and schools, particularly when rewards for successful schools were combined with sanctions for failing schools. Our estimate of the effect of a state that sets up reward-punishment stakes for schools is that the students in these states were 20 percent of a grade level ahead in math and science and 24 percent of a grade level ahead in reading when compared to demographically comparable students in states that did not impose rewards or sanctions. Public reporting of schools' results on state tests is necessary to implement these policies, but reporting without applying rewards or sanctions had no discernable effect on student achievement.
When other state reform policies were held constant, the positive effects of state-imposed minimum competency exams on 8th grade achievement were relatively small. Nonetheless, students with C averages tested better in math and science when they lived in states that required minimum competency exams.
Whose Predictions Were Correct?
Our analysis of college attendance rates, labor market success, and test scores overwhelmingly rejects the hypotheses that such test-based accountability systems as minimum compentency exams or external exit exams hurt students by inducing teachers to teach to severely flawed tests. Indeed, contrary to the predictions of the critics of standards-based reforms, the impacts of test-based accountability policies on indicators of success after high school—attending college and attaining better jobs—are positive. In addition, scores on achievement tests that are not part of state accountability systems are higher in states with strong standards-based reform policies, showing that most students benefit from these reforms. Unfortunately, however, some students will lose out—those who would have graduated under the old rules but cannot pass the new required tests. How large are these effects?
Effects of Accountability Measures on Dropout Rates
We first analyzed the longitudinal study and found that students who had C averages in 8th grade and lived in states that required minimum competency examinations were less likely to graduate from high school within the next six years. The high school graduation rates of students who had B or A averages in 8th grade were not affected.
We then analyzed 1994–97 state data on the dropout rates reported by public high schools. States with more course requirements had significantly higher dropout rates, and those with minimum competency exam requirements also showed somewhat higher dropout rates. The dropout rates of states with end-of-course exit exams were somewhat lower than average.
Surprisingly, holding schools accountable for student test scores was not associated with higher dropout rates; in fact, the policy tended to be associated with lower dropout rates, especially for the state programs that apply rewards and sanctions to schools on the basis of both test scores and dropout rates. Our estimates imply that a well-designed "stakes for schools" system that includes dropout rates in the accountability policy more than offsets the tendency of exam requirements to increase dropout rates (Bishop, Mane, & Bishop, 2000; Bishop, Mane, Bishop, & Moriarty, 2001). Further research is necessary for understanding all of the variables in reporting dropout rates.
Policy Implications
State accountability systems. We found that states that reward successful schools monetarily and apply sanctions to failing schools had significantly higher achievement levels and lower dropout rates. We also confirmed the finding of David Grissmer and associates (2000) that the biggest gains in NAEP mathematics scores were in North Carolina and Texas, the two states that established the nation's most comprehensive systems of school and student accountability in the early 1990s. Education authorities should carefully examine these school accountability systems.
Student accountability: Minimum competency exams. Overall, we found that systems that require students to pass minimum competency exams had positive effects on nearly all students. Eighth graders from states with minimum competency exams were 2–4 percent more likely to be attending college six years later and were earning 9 percent more in salary after high school.
Student accountability: External exit exams. Curriculum-based external exit exams had the largest positive impact on improving test scores. Although New York's students of the early 1990s were more likely to get alternative high school certificates or to take longer to get their high school diploma, they were also as likely as students in comparable states to graduate from high school. Achievement levels of all New York students at the end of high school were roughly the equivalent of one grade level ahead of the achievement levels of students in comparable states.
To develop high standards for all students and to institute fair and effective assessments of how students meet those standards, education authorities should closely examine the benefits of curriculum-based external exit exams in New York, North Carolina, and outside the United States.
