Are you facing a challenging accountability dilemma in your school or classroom? Do you have a question about assessment? In a new twist on a popular column, Educational Leadership invites readers to submit their questions about assessment or accountability to noted assessment authority W. James Popham. Popham is the author of 25 books, including Test Better, Teach Better (ASCD, 2003). In addition, he is a former test developer and an outspoken critic of poorly designed and badly used tests. Address your questions toAskPopham@ascd.org. Responses to selected questions will appear in subsequent columns.
Question: Is it better to use teacher-created assessments or tests that companies create?
—-Rob Ackerman, Principal, Massachusetts
Answer: During the past several years, I've heard numerous educators from both the United States and Canada pose this very question. Clearly, Rob, the reason you and other educators are concerned about such an issue stems from a dramatically increased need for education assessments. If we didn't require more tests, there'd be little need to worry about whether we should get such tests from commercial vendors or, instead, have teachers whomp them up themselves. But we do need more tests—lots of them—so where should they come from?
With few exceptions, education assessments should be supplied to teachers, not pried out of them. Teachers are professionals, and professionals should be able to carry out their work without being required to create—from scratch—all the tools they need to function professionally. Would you ask surgeons to construct their own scalpels before they operated?
Of course, this approach only makes sense if a serious corollary is satisfied, namely, that commercial test-development firms supply good tests. More specifically, we need tests that enable teachers and administrators to arrive at valid inferences relevant to the day-in, day-out decisions they must make. For example, there are surely teachers in your school who want to monitor their students' emerging mastery of the curricular aims that the Massachusetts Comprehensive Assessment System (MCAS) measures. To do so, they'll typically administer classroom tests, then make inferences about students' MCAS progress on the basis of their test scores. But at the moment, the sad reality is that many test vendors are not providing the sorts of assessments that educators need.
So what kinds of tests are most in demand these days? For openers, there is currently a crying need for more tests that teachers can use for classroom formative assessment. Fortunately, ample research has shown that formative assessment works well in classrooms. That's because it supplies evidence enabling teachers to adjust their instructional procedures and students to adjust their learning tactics. Ideally, most of those assessments should be genuinely diagnostic in the sense that students' responses will reveal why “in-difficulty” students are having those difficulties. We also need interim tests that schools can administer every few months to predict students' performances on upcoming accountability tests. And, finally, we desperately need instructionally sensitive accountability tests that can accurately evaluate school quality. Most test vendors do not produce defensible tests of these sorts.
And that's where our “who-builds-them” plot thickens. Although teachers, as professionals, shouldn't have to create their own assessments, they sure as heck need to know how to rigorously evaluate vendor-developed assessments to see whether those tests are truly appropriate for the function they're supposed to fulfill, whether that function is formative, predictive, or evaluative.
Some states take great pride in the fact that all items on their annual accountability tests are constructed by the state's own teachers. Although the teachers may be enthusiastic about the items they produce, many of these items are less than lustrous. Far too many teacher-built items, for instance, contain content that gives a subtle advantage to children from more affluent families.
However, teachers can become enthused about judging the quality of a test vendor's items—as long as those teachers know what to look for. They should approach test evaluation with four questions in mind: (1) Does the test measure a manageable number of instructionally meaningful curricular aims? (2) Do the descriptive materials accompanying the test clearly communicate the test's assessment targets? (3) Are there sufficient items on the test that measure each assessed curricular aim to let teachers and students know whether a student has mastered each skill or body of knowledge? (4) Are the items on the test more likely to assess what a student has been taught in school rather than what that student might have learned elsewhere?
So to answer your question, Rob, education tests should be supplied to teachers by assessment firms specializing in that mission. However, for those tests to contribute to improved schooling, educators need to acquire a modest amount of measurement moxie so they can decide which tests to use and which tests to shred.
Here's one suggestion: Why not encourage several of your school's teachers to form a long-term learning community focused on assessment literacy? Once a small cadre of your teachers possesses assessment acumen, they can help you select vendor-produced tests that are worth having.
