Considering a Multi–level Approach for Assessment of Online Instruction
Introduction
Although research in distance education recommends a multilevel approach for assessment of online instruction, standard practice at most post–secondary institutions involves only summative forms of evaluation. At the end of an academic term, students and educators participating in the distributed learning program are given survey questionnaires or are randomly selected to participate in focus groups to generate a qualitative analysis of the existing program. The information obtained from these reviews is then used to modify the courses for next term. While these practices provide useful information to those involved in the planning and design of distance education programming, they are limited in scope because they address the concerns of participants after the courses are over. Even if modifications are made to the courses based upon the perspectives of the participants, their initial concerns may not be the concerns of the next group involved in the program. As a result, the evaluation is merely an exercise in course revision. However, in order to improve the quality of online instruction, it is necessary to develop an approach that offers a multilevel assessment.
The Value of Evaluation
Evaluation of online instruction should be a continuous process. Educators must be willing to engage in the evaluation process throughout the course in order to obtain a thorough assessment. According to Richard Clark, during the process of the course “early evaluation makes it possible to determine which aspects of a distance learning program are positive and which are negative” (Clark 66). Clark suggests that an early concern for evaluation will allow instructors to modify their course to improve the quality of instruction before the course is over. By monitoring the learning that is taking place and whether the course objectives are being met, instructors can resolve issues immediately. Although some instructors do engage in this type of assessment, its practice is often inconsistent and in some cases not well organized. Some instructors gather this information but have difficulty integrating needed changes. As a result, most instructors simply wait until the end of the semester to modify the instruction for the next run of the course. This practice has little merit for the students who are currently enrolled in the course. Furthermore, there is no guarantee that the students in the next session of the course will express the same concerns. Hence, modifications suggested during an early assessment and throughout the duration of the course need to be made as soon as possible.
Developing a Process of Evaluation
Perhaps the first step in conducting an assessment of online instruction involves really understanding all the objectives of the course or program. Educators should outline in detail course objectives and how each objective will be evaluated. Instructors may wish to consider the following equation to outline and draft a plan for assessment in their online course: “aims and objectives = content = teaching and learning activities = assessment” (Morgan and O’Reilly 47). Using this grid as a template, instructors can then examine and map out an assessment strategy for each specific objective of the course. Objectives do not only include student learning outcomes but also issues related to the delivery of the course, the media of the course, and the overall structure of the course. Instructors should also consider alternative modes of assessment in mapping their plans. Alternative approaches may need to be considered for differences in class size, course content, and time allotted for the completion of the course.
Once the course outline is complete, the instructors should then identify emerging patterns. For example, instructors may note that they rely on the same type of assessment practice for every objective of the course. Instructors should then see if they could incorporate an alternative method of evaluation to assess the given objective. Different forms of assessment should also be considered to accommodate different types of teaching and learning styles. Instructors should also determine whether the assessment tools being implemented are the most appropriate means of measuring and whether the objectives for the course are met.
Integrating Ongoing Assessment
Although there are a variety of methods to use to assess instruction in an online environment, Cucchiarelli et al. suggest that online instruction should involve all the following forms of assessment:
1. Placement assessment is aimed at determining both prerequisite skills and mastery of course content and objectives.
2. Formative assessment provides ongoing monitoring of student progress to ensure that learning is occurring.
3. Diagnostic assessment involves the identification of recurring learning difficulties unresolved by standard correction.
4. Summative assessment is the assessment that occurs at the end of the course.
5. Self–assessment engages both instructor and students to monitor the level of knowledge they have acquired during the process of the course (Cucchiarelli et al., 176–77).
It is recommended that instructors develop a practice to include each of these forms of assessment to gather a variety of information regarding the strengths and weaknesses of the course. However, many instructors do not have the time to integrate a multifaceted approach of assessment in their course and are unwilling to add additional planning and curriculum development to the already heavy workload of managing an online course. In addition, dealing with large classes in an online environment may make it difficult to engage in some of the one–on– one type of assessment practices such as self–assessment. Other instructors do not know how they might integrate these types of assessments into the structure of their course or even which tools would best serve the purposes for the specific method of assessment. Finally, there may not be a consistent university policy regarding the practice of assessment for distributed learning courses.
Yet, regardless of the reasons, it is important to consider a multilevel assessment approach in online instruction to ensure that students are receiving an equitable learning experience in comparison to their peers who are enrolled in the traditional course. There are many more variables in the online environment that affect the quality of instruction perceived by the students. For example, many Web–based students often equate technical issues in a course with the success or failure of that course. If an assessment of this issue is done during the course, an instructor may be able to address the technical problem immediately and in turn, resolve the issue with an alternative method. This response to the problem is much more proactive than waiting until the review of the summative assessment at the end of the course. Students may be less critical of the failings of technology in an online environment if given the opportunity to discuss potential solutions to these problems while they are occurring.
Implementing A Multilevel Assessment
In order to implement a multilevel approach to assessment in an online environment, it is best to begin with developing an understanding of the types of assessment strategies an instructor can use. Most instructors are familiar with the variety of methods available to conduct assessment but do not know how to integrate these methods such that assessment occurs throughout the course in a structured manner. Many instructors may be concerned about too much assessment and may even consider a multilevel approach somewhat problematic because it does not fit within the process of a course. However, in an overview of each type of assessment, strategies and ideas are explained to better facilitate this process.
Placement Assessment
Instructors should never assume that students enrolled in distributed learning courses possess fluency in information technology or information literacy. Suzanne Stokes has found that “preparing students to take online interactive courses involves more than teaching technical skills. Abilities related to time management, self–discipline, independent learning, active information seeking, and constructing must exist” (164). Since many distance education programs do not offer a consistent orientation for students who plan to enroll in distributed learning courses, instructors should provide a preliminary packet of information to each of the students that details the requirements for the course. The packet should contain contact information, the course syllabus, descriptions of hardware and software requirements for the course, and a list of remote site and technical contacts for additional assistance.
Instructors may also want to give students enrolled in their course a pretest to determine the level of competency they possess as well as a preliminary orientation on equipment that will be used in the course. Students can be asked to draft a brief essay describing the qualities they believe students need to be successful in an online course, and the assignment can be used to open up discussion in the class. Other questions an instructor may pose can relate to the software and hardware that will be used during the course. Discussion topics can also highlight some of the objectives of the course.
At least one class lesson should introduce students to the technologies (both hardware and software) they will be using during the course. During this session, the instructor should explain the procedures regarding technical difficulties. Students need to know who to contact when the delivery of the course fails and how they can obtain course materials that cannot be downloaded or retrieved from the course Web site. Instructions regarding classroom policies, remote site procedures, and overall distributed learning program policies should also be addressed at this time.
Instructors who will be using specific software in their course should also identify if their students know how to use the application. It is often best to list this information in the course description or prerequisites for the course so students can know even before they decide to take the course what technology–related competencies are expected. In addition, some students may not have immediate access to the hardware or software that is required in the course. For example, students with a text only browser will not be able to retrieve graphic rich course content. Students with disabilities who are enrolled in the class may need alternative formats of the course material. Instructors need to be prepared to make the accommodations.
Instructors should also develop activities consistent with their students’ online competencies and gradually introduce assignments that enable students to develop their skills in using new technologies. Some activities may include:
1) requiring students to send e–mail to one another and the instructor;
2) having students sign on to course–related listservs or send file attachments;
3) developing skill tests that require students to use various features of the courseware tools or applications needed in the course;
4) using electronic resources and materials for class assignments. The objective is to help students with limited experience in using such technologies find ways to increase their understanding of both the content and the technology.
It is important to take note of the students’ level of ease in using the technologies to determine which students need additional support or guidance to master these competencies. Students who are having difficulties may be paired with online classmates who demonstrate an understanding of the task. By creating partnerships based upon the assessments, peers can assist one another in a more collaborative form of teaching and learning. Such structure adds to the interactivity of an online course.
Formative Assessment
Formative assessment should be conducted during the development and delivery of the course. Barry Willis suggests that instructors should conduct formative evaluations that examine the outcomes of individual lessons. “These mini–evaluations might focus on course strengths and weaknesses, technical or delivery concerns, and content areas in need of further coverage” (par 5). Students can be asked to provide feedback on each instructional unit to determine if modifications are needed. For example, an instructor may wish to ask students whether they understood the objective of the assignment or if the timeliness of the instructor’s response was appropriate for a particular assignment. Instructors can consult students as a group or individually. If, after a discussion, instructors plan to make amendments during the course, they should alert students in a formal manner. One suggestion is to include a section on the first page of the course entitled Important Updates to list changes or important announcements such as the rescheduling of assignment or project deadlines. By addressing epistemological issues during the actual delivery of the course, instructors may be able to address immediate problems more effectively. It makes perfect sense to offer students the opportunity to modify online instruction because it engages them in their learning much in the same way as a traditional setting when we offer students the opportunity to engage in a review of the material being presented.
In addition, the practice of engaging students in peer evaluations is also valuable in formative assessment. On group projects, students can be asked to assess one another, particularly in areas when they are required to collaborate. Instructors can then use this information to form new groups and to see which students may be having difficulty with a particular task. Also, when students are asked to assess one another, they may become much more involved in the assignment. Students may be inclined to participate more actively to ensure a positive assessment from their peers.
Diagnostic Assessment
In some instances, there are some students who may have difficulties adjusting to the online learning environment. Instructors may need to assess the students’ skills in computer fluency or information literacy. Some students may be unable to complete certain class exercises because they lack the ability to locate information on the Internet for course assignments. There are many students who are unaware of the Web– based resources available to them through the campus library system. In most cases, few distance education programs include the library in the planning and development of a Web–based course. For this reason, it is important that instructors enlist the assistance of the library staff to provide an online bibliographic exercise that demonstrates to the students the Web–based library services available to them. Every online course should include links to the campus library. Instructors should find ways to show students how to locate electronic information for class assignments and introduce students to online scholarly journals, quality Web sites, and subject–specific Web resources related to the topics of discussion. The subject reference librarian can be invited as a guest to chat sessions to answer questions related to library research.
In addition to including library staff as guests in a Web–based course, the instructor should invite other individuals such as technical and academic support staff to assist the students with diagnostic assessments. If the institution has an online tutoring center for students or a technical support help desk, these links may be included as well. Many students are not aware of the campus Web–based services available to them. Including such links on the course Web site and then discussing their use is important because it provides students with a means of seeking additional information for questions related to instruction. These support services can assist in developing pretests and guides for evaluating competencies in students.
Instructors can also create their own diagnostic assessments using the testing tools available in courseware products. For example, courseware products such as WebCT and Blackboard enable instructors to restrict access to instructional modules until students have passed a post–test on the previous module. Instructors can also create skill sets of assessments to determine what type of information the students in the class have mastered. There are many software programs that enable instructors to create interactive Web–based tutorials that students can use to obtain additional review.
In addition, instructors should maintain a record of technical issues that occur throughout the semester. For example, instructors should record the number of times technical malfunctions impeded the delivery of the course. Students should inform the instructors of any difficulties with accessing or sending material, using the courseware tools, or other technical issues to determine what impact has been made on instruction. Instructors should always check if students can access the grade book, hyperlinks, and other files in the course. Such technical monitoring is important to evaluate the delivery of the course and update the technical staff of specific problems. Doing diagnostic assessments on technical issues is useful to the technical staff because they are able to identify more readily the causes of problems. All too often, technical support staff members are not informed about problems in a timely fashion. It is difficult to resolve a problem after the course is over. By working more closely with the technical support staff on tracking delivery errors and malfunctions, it may be easier to troubleshoot and prevent further instances of technical difficulties.
Summative Assessment
Although most instructors already engage in some form of summative evaluation, it is important to create two summative assessments for a course. Clark suggests that we need to separate delivery and instructional evaluation questions. In his view, media does not influence learning. Hence, when we do conduct an assessment of online instruction, it is important that students understand what it is they are asked to evaluate. Is it the manner in which the course was delivered? Or is it the quality of instruction they received? All too often students attribute problems with the media to poor instruction. Technical difficulties may contribute to problems in providing instruction but not the overall quality of the content of that instruction. Keeping a diagnostic assessment of technical problems helps create a related summative assessment for the students. Students could then respond to direct issues that actually occurred during the delivery of the course.
Focus groups, surveys, and other methods of assessment can be used to determine the overall success of a course, but instructors should also examine the factual materials available in relation to the course. How did the students score on specific assignments? How were the objectives of the course met? Were all of the objectives of the course met? What ways can orientation for the course be improved? Which support services were most beneficial to the students? These are just a few of the targeted questions instructors should ask students. Students can also be asked about specific assignments or activities. It is also important to ask students for suggestions to improve the design, development, and delivery of the course.
Self–Assessment
For various reasons, self–assessment is often overlooked in distance education. Instructors should keep a log of their e–mail interactions with their students and examine ways in which information could have been communicated better. What was the time ratio for feedback to student queries? How often did students ask questions about a specific assignment? What type of questions did students ask? Knowing this aspect will help an instructor clarify the objectives of the course. What technical difficulties prevented students from obtaining course information? How was class management? Instructors should invite other online instructors to participate in online peer assessment. All instructors should take time to determine if they have met their initial course objectives.
Students should also be encouraged to engage in self–assessment to be shared with the instructors. Willis suggests that students keep a journal of their ideas regarding the course content and technical difficulties they have experienced. This information enables instructors to determine what students have learned or what could have improved the instruction. Students should be given a post–test that addresses issues related to the course objectives. Asking students to respond to open–ended questions related to the objectives of the course in terms of their own self– assessment will provide useful information. Questions should be targeted to focus on specific class issues to direct the students to review elements of the course and not the personality of the instructor.
Instructors will know what students learned, what they understood, and what they did not. Students engaged in self–assessment recognize what areas for which they need to obtain additional skills if they continue to participate in distance education. Some students may realize that they may not be ready for the responsibilities of self–directed learning in an online environment. Instructors who review students’ self– assessments gain valuable insights from the students’ perspectives. This information will, in turn, help in modifying assignments or content in the course.
Online instructors should then conduct a meta–analysis of all the evaluation data for a final assessment of the course. There is always room for improvement in every form of instruction, and so instructors must be willing and prepared to make revisions of the course they have created. All course revisions should be documented for future reference to share with colleagues who may be considering developing Web–based courses. Major revisions in a course should always be tested prior to implementation of the course, particularly if these revisions require the use of new technologies. An instructor should evaluate the revision of a course with technical support staff to determine if the use of technology is appropriate to the instruction.
The Value of Multilevel Assessment
Many instructors may wonder why assessment of online courses should be a continuous, multilevel process. It is difficult to manage the workload of an online class, so why bother engaging in a process that will involve additional planning? Instructors interested in improving the overall quality of their online instruction must understand that various approaches of evaluation will generate a wide–range of information that can improve the overall design, development, and delivery of online instruction. Unlike a traditional classroom where instructors receive more visual cues in how a lesson or material is being received and understood by the students, the online environment is somewhat more challenging because unless students are encouraged to do so, their feedback about the success of a lesson is often difficult to attain. Some students do not even bother submitting summative assessments at the end of a Web– based course and many universities have not adequately developed a suitable assessment tool to evaluate online courses. In fact, most universities rely upon the same form of assessment used in the traditional classrooms. This practice is problematic because it does not address concerns and issues that occur primarily in an online environment. For this reason, alone, a multilevel approach for online assessment is much more critical for distance education instructors determined to improve the quality of their instruction.
Works Cited
Clark, Richard. “Assessment of Distance Learning Technology.” Eds. Eva Baker and Harold O’Neill. Technology and Assessment in Education and Training. Hillsdale, NJ: Lawrence Erlbaum and Associates, Publishers. 1994. 63–78.
Cucchiarelli, A., Panti, M. and Valenti, S. “Web–Based Assessment in Student Learning.” Ed. Anil Aggarwal. Web–Based Learning and Teaching Technologies: Opportunities and Challenges. Hershey, PA: Idea Group Publishing. 1999. 175–197.
Morgan, Chris, and O’Reilly, Meg. Assessing Open and Distance Learners. Sterling, VA: Kogan Page Publishers. 1999.
Stokes, Suzanne. “Preparing Students to Take Online Interactive Courses.” The Internet and Higher Education. 2 (2–3) (2001). 161– 169.
Willis, Barry. “Strategies for Teaching at a Distance.” Distance Education—Strategies and Tools. Updated 2002. 30 Sept. 2002
Biography
Mary Hricko is an associate professor of Library and Media Services at Kent State University and serves as the Library Director at the Kent State University—Geauga. Her recent book Design and Implementation of Web– Enabled Teaching Tools addresses Web accessibility and its applications to distance education. She may be reached at mhricko@kent.edu.