In 1991 the Oregon legislature adopted a comprehensive educational reform bill calling for significant changes in educational practice in Oregon's schools by the end of the 1990s. A central element required schools to implement a Certificate of Initial Mastery (CIM) program by 1997.
As originally conceived, the certificate would validate a student's ability to apply what he had learned in several disciplines to a variety of challenging open ended problems. For the first time, students would earn an academic credential based on performance gauged against high standards. Because of the applied nature of the CIM learning goals, results from performance assessments would play a major role in the certification process. In 1995 the legislature added a requirement for students to demonstrate a breadth of knowledge and skills in several disciplines. The legislature also extended the implementation date to 1999 and chose not to make it a requirement for high school graduation. (The original legislation had been silent on this last issue, leading some to speculate that the CIM would replace the current high school diploma.)
Early on, the Oregon Department of Education asked several school districts to begin designing and piloting a performance based certification program. Lake Oswego, a school district with 7,000 students in suburban Portland, is one such district. After four years, we have learned several lessons about what it takes to successfully implement a performance based certification system.
The Certificate of Initial Mastery
To earn a Certificate of Initial Mastery, a student must accumulate a sufficient body of evidence displaying a high level of achievement toward each of several learning goals developed by the state. Students begin accumulating evidence in elementary school and continue until they have met all requirements for the certificate.
The law says that "most students will complete the CIM by about the end of 10th grade." However, in Lake Oswego, earning a CIM is viewed as a "continuous progress" activity, meaning that the certificate is to be awarded when all the requirements are met, regardless of age or grade level. Because the certificate reflects work related to several different standards, we may possibly award intermediate "merit badges" as a student reaches the desired level of proficiency in a specific area, such as writing or mathematics, and award the overall certificate after the student has met all of the component goals.
The body of evidence has two parts. The first will directly address "content standards." In Oregon a content standard describes the knowledge and skills a student should master in a discipline. So, for instance, in social studies, a student eligible for the certificate will have a broad grounding in the facts of history, geography, economics, and civics, as well as appropriate analytical skills. Typically a student would demonstrate these through examinations that sample for breadth but do not demand much beyond recall and comprehension. A statewide assessment in reading and literature, mathematics, science, and social studies will be a primary source of this evidence. Other standardized exam scores may possibly be included, provided that an appropriate "CIM level score" can be determined.
The second part of the body of evidence will focus on a student's ability to apply knowledge and skills to a variety of tough problems. This is where performance assessments will play a major role. Some of the evidence will come from on demand performance assessments that are part of the statewide assessment program, and other pieces will come from performance assessments developed at the district level and embedded in the local curriculum (see fig. 1).
Figure 1. Performance Standards/Mathematics
State Multiple Choice Tests. On state multiple choice tests, students must achieve these scores to exceed the performance standards in mathematics.
Meet Standard: 239
Exceed Standard: 249
State Problem-Solving Tests, Certificate of Initial Mastery, LEVEL 10. On state problem-solving tests, students must achieve the following scores to meet or exceed the performance standards in mathematics.
Mileposts on the Road to a Certificate of Initial Mastery - table
Meet Standard | Exceed Standard | |
|---|---|---|
| Conceptual Understanding | 4 | 5 |
| Process and Strategies | 4 | 5 |
| Communication | 4 | 5 |
| Interpret Reasonableness | 4 | 5 |
Classroom Assessments, Certificate of Initial Mastery, LEVEL 10. On classroom assessments, students must achieve the following scores on the state coring guide to meet or exceed the performance standards in mathematics.
Within five mathematical problems, solve accurately and demonstrate understanding of statistics and probability, algebraic relationships and geometry.
In each show the following:
Mileposts on the Road to a Certificate of Initial Mastery - table 2
Meet Standard | Exceed Standard | |
|---|---|---|
| Understanding of the mathematical concepts present in the problem | 4 | 5 |
| Use of appropriate mathematical processes and strategies to solve the problem | 4 | 5 |
| Review of the work and support for the reasonableness of the results | 4 | 5 |
| Clear communication of the steps to the solutions | 4 | 5 |
The current plan calls for using the same discipline based scoring guides for all performance assessments, whether state or locally developed (see fig. 2). Scoring guides can help ensure that every school and classroom in the state uses the same high standard. Such guides for reading/literature, writing, speaking, mathematics, science, and social studies are all being developed at the state level and will be used uniformly by every district in the state. Scoring guides for foreign language and the arts will be developed by local districts that have the responsibility for administering local assessments, setting local performance standards, and certifying student performance. There will be no state performance standards for these two goals.
In addition to scores from performance assessments, Oregon law also requires that actual scored samples of student work be included in the body of evidence, so that the quality of student work becomes public. A parent, employer, or college admissions officer should be able to look at a student's body of evidence and independently reach the same conclusion about the level of performance represented as did the educators originally scoring the work.
Lessons Learned
During the past four years, Lake Oswego teachers and administrators have worked hard to achieve several objectives, including creating quality curriculum imbedded performance assessments, learning to use standardized scoring guides, implementing a workable filing system for certification materials, and gaining community support for a certification system. Although we have made progress on all fronts, the rate and quality of that progress have been uneven. We have struggled with a number of thorny issues and, through trial and error, have learned some important lessons.
When to certify. One unresolved question involves timing. The reform law calls for all students to be reviewed at grades 3, 5, and 8 to check their progress toward the learning goals. These benchmark reviews are intended to trigger help for those students not making adequate progress. The benchmarks, and the law's reference to 10th grade as the point when most students will complete the CIM, have led some educators to conclude that all students will basically move forward together and that 10th grade is when the actual certification should happen.
Lake Oswego's "continuous progress" approach disagrees with this view. Besides being suited to the obvious reality that students will acquire proficiencies at different rates, the continuous progress model minimizes the political problems associated with the expectation that students will acquire certification by about the same time. Such an expectation ensures a "winners and losers" dichotomy and the emergence of constituencies to defend the "losers." The political viability of a certification system depends on not adopting a grade level approach.
The role of curriculum documentation. Pre existing agreements among teachers about what is to be taught at various grade levels and in courses at the secondary level are very helpful in establishing a system based on standards. Existing curriculum agreements in Lake Oswego have helped in the development of performance assessments and fostered the collaboration needed among teachers to more uniformly judge the quality of student work. Districts without such agreements seem to be moving more slowly toward a certification system.
The quality of performance assessments. We have produced several hundred performance assessments covering several disciplines and spanning grades from primary through 10. Most of these have had significant educational value, but the vast majority fall short of what we need. The high quality performance assessments needed for certification must include clear specifications, standard expectations for documentation, careful field testing, and adequate security measures. Our own strategy of decentralized development individual classroom teachers developing and using their own assessments appears inadequate. We are moving to a collaborative approach in which small districtwide teams of teachers jointly develop common assessments for disciplines and grade levels.
Differences among educational levels. At first we thought that implementing the elements of the certification process would be generally the same regardless of the educational level: elementary, middle, or high school. In reality, we found a sharp difference in what was required of teachers working in self contained classrooms and of those teaching specialized subjects such as mathematics.
Because of the broad scope of the CIM learning goals, a teacher in a self contained classroom must assess, collect evidence, and monitor progress in several areas. This poses a significant logistics problem. Our current working solution is to phase in the areas in which evidence will be gathered and to reduce the amount collected. For instance, a 3rd grade student may accumulate evidence only in reading, writing, and math, with perhaps only three scores and work samples per area. By 6th grade, the student may gather evidence in all CIM learning goal areas, with a minimum requirement of six scores and work samples per area, none more than 18 months old. These somewhat reduced expectations are justified at the elementary level because monitoring, not certification, is the focus at this level. The key is to collect just enough information to make sound remediation decisions.
Political realities. The legislature's enactment of reform legislation is a classic example of top down change. Not surprisingly, some constituencies both inside and outside the education community at the local level had no interest in being "reformed." In fact, some constituencies are actively opposed to the implementation of high standards or any standards at all, preferring the current system of many alternative courses that substitute for the more rigorous core courses.
A strategy that has worked to build support for the certification system has been to deliberately connect the goals and elements of the legislation with our local values as presented in a number of strategic documents. By initially decentralizing our development activities, we sought, with some success, to garner bottom up involvement and support. But as far as we can tell, the job of monitoring and cultivating support for significant changes is neverending.
Ownership and logistics. A key question we have grappled with is, Who has primary responsibility for maintaining and updating the collection of evidence? Typically, school officials have had custody of high stakes data transcripts, for example. Our current view says students should be custodians of their own collections of evidence after they reach the secondary school level. This has introduced several difficult problems yet to be solved, such as finding ways to make work samples secure from unauthorized change. But we have found that the benefits are enormous. In some cases, this strategy has significantly increased the level of student effort and commitment.
Another challenge is how to manage all of the CIM certification records, which include actual work samples as well as test scores. One student's collection of evidence can be substantial. Two strategies have helped. First, we have adopted the concept of a "rolling" collection of work; when new evidence is added, some old evidence is removed. What is left is a "critical mass" of a student's highest quality work to date. Second, we have used technology to store large amounts of information in a compact form. Our current procedure, which works on a small scale, is to store data on a recordable compact disk in an organized presentation format. Each disk costs about $7, a cost we expect to transfer to parents. We will soon see if we can make this system work with several thousand students instead of several hundred.
The Road Ahead
We have two more school years in which to complete our preparations to issue Certificates of Initial Mastery. We have a long way to go. Several issues will occupy our attention.
Implementation strategy. We will work on developing common performance assessments in district teams while supporting implementation through school based action plans. The goal is to speed and standardize development while maintaining good buy in and support at the building level.
Scorer calibration. A system of standards requires that work be scored consistently. This is difficult, particularly for performance assessments, which use scoring guides to score the work. Ten years' experience in Lake Oswego with performance based writing assessment has shown that teachers even teachers outside the English/language arts discipline can learn to use a writing scoring guide consistently. We are betting that this will be possible with the other scoring guides as well.
One key to consistency appears to be collaborative scoring sessions for teams of teachers. Even experienced scorers benefit from "recalibration." In 1996 97 we will collect samples of student work already scored by classroom teachers and rescore the work within district sponsored scoring groups. Teachers will discuss their use of the scoring guides and the selection of range finders for the various score levels.
Rules of evidence. Many "rules of evidence" must still be established that will influence the final form of student certification files. How many pieces of evidence will be sufficient to justify certification? How will various pieces of evidence be weighted in the decision making process? What role, if any, will student self reflection play in the certification process?
In addition, we need to address various security and authenticity issues. A current plan calls for every piece of evidence included in a CIM file to be authenticated so that reviewers of student work can be assured that the work is genuine.
Remediation. As we track student progress from elementary school into the secondary schools, we will need to identify and support students who are not making adequate progress. We will need to invent and prove the effectiveness of remedial strategies. These may involve intensified instruction or extended school year opportunities. In addition, we will need to invent a method for retesting and making up assessments perhaps at a district or even a regional testing center. To be credible, our certification process must show that any student denied the CIM had ample additional support and opportunities to demonstrate the required level of performance.
Although we having been working on developing a performance based certification system for four years and will not issue our first Certificate of Initial Mastery until the 1998 99 school year, we barely have enough time to get ready. One overall lesson is abundantly clear: Implementing a performance based certification system takes time, resources, and patience, particularly when there are no working models to follow.
We have successfully passed several milestones on our journey. Although we are pretty sure we are on the right road, we are still a little uncertain how many miles are left to go and whether unexpected curves remain to be negotiated. Time will tell.
