Of all the technological innovations aimed at transforming teaching, one of the more controversial is the computer-based integrated learning system (ILS). These systems include courseware and management software and run on networked hardware. They cover one or more curriculum areas across grade ranges, and usually include a management program that tracks and reports student progress (Wilson 1990). Proponents of such systems argue that they increase student motivation and enhance individualized instruction, thereby improving student learning.
After five years of studying the effectiveness of ILSs, we have concluded that the systems do have enormous potential but are infrequently implemented fully. This conclusion is based on several studies we conducted for local school districts, as well as a two-year national study conducted for an ILS agency. (In the latter study, we were employed by an independent research firm. Additional external researchers critiqued and verified the validity of the study design, instruments, analysis, and interpretation.)
The Kinds of Instruction Possible
Most ILSs include instruction and practice problems in basic school subjects such as math and language arts, although some also include computer skills, science, writing, social studies, foreign languages, and even entire English-as-a-second-language (ESL) programs. Several systems provide for the entire K–12 curriculum. The instructional design of most systems follows behaviorally-oriented programmed instruction. Each lesson is tailored for the specific learning level of the student and builds in a sequential fashion in the same manner as a basal textbook.
Recently, some ILSs have moved beyond this simple drill and practice by adding materials that require students to delve into complex problems in ways that promote deep reflection and genuine understanding (Becker and Hativa 1994). For example, some systems include science simulations that teach the scientific method by illustrating the beginning of an experiment and requiring students to hypothesize about outcomes before completing the simulation. Some language arts lessons encourage discovery learning by allowing students to explore topics through a branching program before defining specific vocabulary or presenting the overall theme.
Such ILSs follow more of a constructivist view of learning by providing a rich learning environment. Word processing programs, spreadsheets, mathematical graphing programs, encyclopedias, and thesauruses are only a few examples of the software and other resources that allow students to construct meaning and enhance critical thinking.
The Assessment Capabilities
One essential feature of all ILS management systems is their ability to track student progress on learning activities from the point at which students log on the computer to the point at which they log off. When the student begins a lesson the next day, the ILS automatically adds to the assessment file for that student. The most basic function of these files is to provide feedback and to record each student's response to traditional class-type tests. In addition, many systems can also evaluate student learning with alternative forms of assessment. Holistic computer scoring of student writing samples and computer management of student portfolios are examples of assessment approaches that are currently being refined for use with an ILS.
The Physical Configuration
The ILS software is used on a central file server computer that is networked with as many as 30 or 40 student computers. From the system's central server, specific lessons are automatically sent to each student's computer when the student logs on. The lesson selection is based on the ILS's cumulative assessment of the student's prior performance and current learning needs in that particular curriculum strand.
Most systems are designed to be used in either a computer lab or a distributed configuration. In a lab, 20–40 student workstations are contained in one area, along with the file server to which they are networked. Teachers may bring their entire class to the lab, where they—or trained lab attendants—use the ILS file server to deliver the tailored instruction to each student and then monitor to ensure progress. Labs may also be used by a teacher to deliver whole-class instruction by calling up the same lesson on all computer stations.
In the distributed configuration, a few computers are typically located in each classroom and are networked to the centrally located file server. Only a few students work on the computers at any one time, while the remaining students are engaged in other activities. Although teachers can use the in-class computers to deliver the same lesson simultaneously to a small group of students, students in distributed configurations usually move at their own pace within their individually prescribed curriculum.
What Do We Know About ILS Effectiveness?
Proponents of these systems argue that they improve student learning. Whether or not one views research as supporting this claim depends in large part on the type of evidence one chooses to trust. For some educators, testimonials of informed users— students, teachers, curriculum specialists, and administrators—carry great weight. Thus, questionnaires and interviews that convey the sentiments of such groups are accepted as important evidence. Experts' observations of the system in use are, to some, far more trustworthy than typical quantitative data such as students' test scores. They prefer case studies rather than classic, experimental control group studies.
Other educators view case studies and anecdotal evidence as quaint and provocative, but not scientifically defensible. Opinions may be interesting to such persons, but they carry little weight when compared to quantitative evidence collected by tests or more direct measures of student performance. Little short of carefully controlled experimental or quasi-experimental research is accepted by such educators as evidence that any instructional system or approach is effective.
ILS researchers have used both approaches. Numerous studies have shown positive student, teacher, and administrator satisfaction with the systems (Sherry 1990, Trotter 1990). However, many of these studies did not address the impact of the ILS on student achievement. Of the studies that did examine student learning outcomes, many reported large gains on standardized test scores (Trotter 1990). However, Becker (1992a) has noted methodological flaws in many of these studies and, in a meta-analysis of nearly 100 of them, concluded that they provide little conclusive evidence of ILS impact on achievement. Where differences were found between the achievement of ILS users and comparable non-users, Becker concluded they were too small to have any educational significance.
Once again we seem to be left with the limp conclusion that “research results are mixed.” For those who are comfortable with casually conducted comparative studies or who accept testimonials of practitioners and opinions of expert observers, the effectiveness of the ILS has been amply demonstrated. The jury is still out for those who demand the rigor of decisive experimental evidence.
The Underutilization of ILSs
Our research (Van Dusen and Worthen 1994) has led us to yet another conclusion: We believe that the reason prior research results are equivocal may be that they have underestimated the impact of these systems. This would explain the apparent contradiction between testimonials of ILS users and the results of rigorous experimentation.
Why have most scientifically rigorous studies of integrated learning systems failed to demonstrate their impact? We believe that the “no impact” findings may reflect a failure of such studies to examine one very critical question: Has the ILS been implemented correctly and in the manner its designers proposed as necessary for its effectiveness?
This underutilization of such systems—the failure to use them in accordance with specific guidelines and practices designed to achieve optimal performance—has been documented (Sherry 1990), and it became apparent to us during the first year of our national study. In none of the sites participating in our study was the ILS being used to its full potential. In some schools, students were spending as little as 10 minutes per week on the system! Given such astonishing underutilization, we were not surprised to find that students in the ILS schools showed no greater gains in achievement or attitudes than students in non-ILS schools.
During the second year of our study, the schools were encouraged to improve their implementation. As a result, students spent significantly more time on the ILS, and teachers were much more involved in integrating the system with classroom activities. The results were startling: Students participating in the ILS classes scored higher on standardized tests than did those in traditional classes. The effect size for full implementation in reading was +.44, nearly a half standard deviation gain. Similar results were found when using objective referenced tests. Performance in reading and math were both statistically and educationally significant, with large effect sizes of +.40 and +.52, respectively. In general, as the level of implementation and teacher integration increased, so did student performance and student and teacher effort.
We believe that underutilization is a fairly pervasive and serious problem, one that undermines the effectiveness of the ILS. No system can be expected to realize its full potential until and unless it is fully and properly implemented. Using an underimplemented ILS will have as little effect on transforming the classroom as taking a fraction of a prescribed drug dosage will have on curing an ailing patient.
How Can an ILS Transform the Classroom?
The findings of our study show that an ILS—if used appropriately—has real potential for improving student achievement. This conclusion is further supported by studies we conducted for several school districts. (Each of these studies has been funded by the school district, not the ILS vendor.) We have held focus group interviews with more than 100 teachers and principals, made more than 100 structured classroom observations to chronicle ILS use and integration with classroom curriculums, received surveys from nearly 300 ILS teachers and administrators, and obtained achievement and attitude data from nearly 5,000 students. While there is variability in all of these data, the following conclusion seems warranted: If implemented properly, ILSs do produce positive results. Indeed, they have the potential to transform the classroom into a better environment for learning.
- Increased time-on-task. In ILS classrooms, students are motivated to learn. They enjoy working on computers, and they like the animated characters, voice capabilities, and full-color graphics. Because they can progress at their own pace, and because they receive immediate feedback, students remain engaged. This is important because increased time-on-task has been shown to improve student performance (Doyle 1983). We found that time-on-task increases significantly with use of the ILS (Worthen et al. 1994). The average engaged time for ILS students was 87 percent, an increase of nearly 20 percent over the traditional classroom.
- Computerized learning resources. Integrated learning systems can facilitate research and exploration within the classroom. Because these systems include extensive tool and database software (for example, word processors, reference programs, spelling and grammar checkers), students can explore new content, solve problems, and create new concepts and associations.Writing assignments are included in many ILS language arts programs. These programs usually include feedback loops that are based on student responses. Thus, the ILS has the potential for reducing the number of teacher-scored assignments. Many ILS management systems also accommodate third-party software that will assess writing and store writing samples in a printable form for portfolio management.
- Effective assessment and reporting. These systems have the potential to provide user-friendly assessment reports. Their management systems can generate assessment files that contain a wealth of information. This includes such routine information as the correct percentage of responses for each exercise attempted, as well as less traditional information, such as the specific learning objectives covered, the amount of time required for a student to attain each objective, and the number of remediation cycles required by that student. This kind of information is vital if a teacher is to adequately plan appropriate learning activities for each student, yet it is nearly impossible to chart in the traditional classroom. In addition, most ILSs allow teachers to customize reports to meet their specific needs.Unfortunately, our research revealed that most teachers (over 80 percent) do not, at least initially, use ILS reports. Many teachers find it daunting to work with the computer to obtain the printouts, and many have difficulty interpreting the vast amount of information contained within the report. Inservice training would help to ensure that teachers can take full advantage of these reports.Integrated learning systems also have the potential for accommodating alternative assessment, such as portfolio management and scoring of student writing. They can facilitate such product, process, and performance assessments as learning logs, student journals, “think aloud” process descriptions, and self-assessment checklists. However, because the teacher must find the third-party software and the administrator must secure funds to purchase it (adding to the ILS's already considerable price tag), few schools have such supplements.
- Guaranteed individualized instruction. The ILS allows the teacher to individualize instruction to fit each student's needs. Because the system organizes content into small units, based on the learner's ability, each student can make progress. Students who learn quickly move through the curriculum at a rapid pace. Those who need additional time and extra exposure receive it.Such individualized instruction can eliminate the need for age-based grade levels. Elementary school students can simultaneously function at several different grade levels within different subjects. Students in secondary schools can be released from the strictures of particular grade-level curriculums and work toward mastery of particular curriculum strands and themes. And, ILSs are increasingly popular for use with ESL students. Several vendors offer ESL programs that allow students to continue using their native language, while being mainstreamed at a level they can handle.Learning theory suggests that an individualized program that caters to specific learning needs will lead to greater student gains, while also decreasing the gap between low- and high-ability students. There is evidence, however, that excessive individualization may actually detract from average-ability student performance (Becker 1992b). Thus, when the ILS becomes the only source for learning, some students may suffer from the lack of teacher interaction.This problem can be overcome by using the ILS in conjunction with other learning activities. For example, one 5th grade teacher (Blickhan 1992) relates that while some students work on the computers, she assigns seatwork to others, allowing her to spend time with small groups of students with similar needs. The ILS also allows the teacher to conduct one-on-one tutorials with students. This change, however, requires a fundamentally different role for the teacher.
- New roles for teachers. The most obvious difference to the observer of an ILS classroom is the role and behavior of the teacher. In traditional classrooms, teachers are the focus of most activities. Most of their time is spent in direct instruction. The balance of their time is spent performing burdensome clerical tasks, including assessment and reporting activities.
The role of the teacher changes dramatically in the well-managed ILS classroom. Teachers are still responsible for students' learning, but rather than being dispensers of information they become guides to the learning process. They act as facilitators and organizers of learning activities; they are free to focus on small groups and individuals who need more specialized attention; and they can coach their students in how to process information, helping them to make choices and validate their learning.
The importance of this new role for the teacher cannot be overemphasized. Our studies have shown that the extent to which the teacher integrates the ILS and classroom content is the most important factor in producing student learning gains. Unfortunately, most teachers see the ILS as a supplement to, rather than an integral part of, the classroom. This is, perhaps, the greatest challenge schools face in incorporating the ILS into the classroom.
Essential Ingredients in Using the ILS
- A sound implementation guide. The main reason educators flounder in their efforts to initiate change is that they lack a strategic plan for how those changes are to occur. An implementation guide that clearly identifies the objectives for using the ILS and assigns specific roles and responsibilities that must be accomplished to meet these objectives is critical to its successful adoption. Vendors typically supply this guide, but schools may wish to refine or extend it as needed. At a minimum, the implementation guide should address each of the remaining criteria.
- An appropriate hardware configuration. One of the first decisions that a school must make is whether to set the ILS up in a lab or in a distributed configuration. While both configurations can be successful in providing students with individualized learning and assessment, if classroom instruction is to be transformed and maximum teacher involvement and integration with other class activities is to be ensured, students must have easy access to the computers (Robinson 1992). Some schools we have observed have made good use of a combination of approaches: Each class has several computers in addition to a centralized lab. In this way, small groups can work on the classroom computers and periodically the entire class can work together in the lab.
- Adequate time on the ILS. In a well-run ILS class, the software becomes the primary information provider. Therefore, it is essential that students spend a substantial portion of their day working on the computers. Our studies suggest that each student must spend a minimum of 30 minutes per subject area per day on the computers to achieve significant learning gains.Ironically, it is a commitment to an egalitarian ethic that often ensures that students do not meet this minimum. When they are unable to purchase enough computers to give each student a full complement of time, educators typically shorten every student's time, thus reducing the system's effectiveness for everyone. The notion of selecting some students to receive computer instruction while others receive none is simply not palatable in most schools.
- Adequate computer equipment. It is essential that enough computers are available for each student to have sufficient time on the ILS. For example, if the ILS is to be used in four basic subject areas, each student should average 120 minutes per day on the system (30 minutes in each area). Given that there are typically 360 instructional minutes in a day, at a minimum the school would need one computer for every three students.
- Effective ILS teaching. The biggest obstacle to effective use of an ILS is the challenge teachers encounter in integrating it into the classroom. Just as its name suggests, an ILS requires integration with other classroom learning activities. An ILS curriculum is not intended to supplement the class curriculum; rather, there should be one curriculum that is presented through a combination of ILS activities, small-group instruction, one-on-one tutorials, and other activities. This means the teacher must learn how to select and schedule the most valuable learning events at the most appropriate times.Among the factors that influence the integration of the ILS and classroom activities, one of the most important is bridging between various learning activities. For example, a teacher might assign a small cooperative group to conduct a hands-on science experiment. This activity should involve the same concepts the students are being exposed to on the ILS. The coordination between ILS instruction and other classroom activities will require the teacher to become as familiar with the ILS curriculum as he or she is with the textbook. Clearly, this is not a trivial undertaking and it may initially increase the teacher's workload.
- Supportive, visionary administrators. Administrative support is a key to the success of ILS instruction. First, administrators must provide leadership in the decision to adopt an ILS. Once the decision to adopt an ILS has been made, they must support teachers in their shift from information dispenser to integrator of learning activities. Part of this support should come in the form of inservice training. Sherry (1990) suggests one to two full weeks as a minimum. Unfortunately, however, his research reveals that fewer than 10 percent of teachers in schools adopting an ILS have even five days of training in its use. Administrators should also encourage teachers to experiment with new uses of the ILS.
In Summary...
One might wonder if any fully implemented ILS classrooms exist. The answer is yes, but they are not yet abundant. Such an effort requires a major commitment of resources and professional energies, as well as an ample measure of patience. Research (Cook 1994) suggests that it will take a minimum of three years for most schools to establish a fully functioning ILS.
Our studies show that if adequate time and effort are invested, an ILS has the potential to restructure the classroom and transform teaching in ways that will result in improved student learning. But this potential can only be realized if schools are committed to adopting—and implementing fully—a well-designed system.
