Srinivas Chappidi, IAEGS, University of Mississippi (USA)

schappid@olemiss.edu

Keyur C. Patel, IAEGS, University of Mississippi (USA)

keyur@olemiss.edu

Dr. Pamela B. Lawhead, IAEGS, University of Mississippi (USA)

lawhead@cs.olemiss.edu

 

Abstract

This research focuses on developing an innovative Evaluation Management System (EMS) for online courses in order to enhance the testing process in a distance-learning environment. The features of EMS include an automated quiz generation module, a quiz input module, a quiz edit module, an administrative module, and a report generation module. The questions in the student quiz module are randomly picked from a set of questions created by experts and presented in a non-repetitive manner. The EMS includes a framework for statistical analysis of quiz results, which are stored in an extensive database as well as  various other student performance-related information. These modules are populated from a comprehensive database of questions, answers, and explanatory material. This paper summarizes the goal of the Institute for Advanced Education in Geospatial Sciences (IAEGS) to develop an integrated and effective evaluation system for online delivery.

1. Introduction

“Distance education via online courses is becoming a standard offering by most colleges in both undergraduate and graduate courses. Students are flocking to online classes in record numbers” (Cohen & Ellis, 2002, p. T3E-8). Teaching challenges are increasing day by day with the growing enthusiasm shown by students towards online learning. One such challenge is an effective student evaluation mechanism for those taking these online courses.

The Institute for Advanced Education in Geospatial Sciences (IAEGS) (http://geoworkforce.olemiss.edu) is a NASA funded project whose purpose is to develop fifty online courses in the field of Remote Sensing (RS) and Geographic Information Systems (GIS). Course content is written by content experts who are chosen by a panel of experts in an international competition. The written material is then enhanced by technology experts who prepare it for delivery in an online environment. To prepare the material for online delivery, IAEGS has divided the course material into hierarchical modules for structured storage of the course content. In this context, each course is divided into Units, Concepts, Sub-Concepts and Objects. The Objects are the lowest level of information granularity and provide the content for the Web pages which consist of text, static images, animations, video and audio.

Figure 1:  Hierarchical structure of course content

The student navigates through the course in the hierarchical manner as shown in figure 1 (Lawhead, Darbhamulla & Das, 2003). When the student selects the Unit and navigates through the course different Concepts in that Unit are encountered. IAEGS has identified two levels of evaluation schemes namely Pre-Quiz and Post-Quiz in the lower level and Exam at the higher level. Before a student enters a Concept, optional Pre-Quiz is presented. If a perfect score is made in the Pre-Quiz the student is allowed to skip the Concept and its associated Post-Quiz. At the end of each Concept a required Post-Quiz is presented to the student and the results are used as part of student’s final grade. Exams are conducted at Unit level. The Professor Of Record (POR) is then presented with individual student’s performance and the course grade is then assigned by him (Meek, 2002).

2. Background and Motivation

The courses developed at IAEGS are licensed to Universities and various Organizations around the globe so that they may offer the courses online to their students and staff. A representative in the University or Organization who conducts this course is called the Professor Of Record (POR) and configures the course to make it appropriate for the students (like graduate and undergraduate).

As noted above the Institute selects Course Creation Fellows (CCFs) from different parts of the world annually to prepare the actual content of different courses. The CCF then sends the actual material to the Institute, which is later converted and archived to be presented in an effective interface. The CCF also sends quiz questions at Concept and Unit level through an Online Quiz Input Tool /Module developed as a part of EMS. There are many issues concerning an effective online delivery of course content to the end user. This paper addresses only an Online Evaluation Management System in a course delivery mechanism.

3. Overview of the Evaluation Engine

Every student taking an online course through IAEGS system encounters with three types of testing schemes viz., Pre-Quiz, Post-Quiz and Exam. The flowchart in the Figure 2 explains the process of such a student being evaluated while taking a course.

 

Figure 2:  Flow chart showing the evaluation process of a student

The Pre-Quiz: The Pre-Quiz is an optional quiz. If the student wants to skip a particular Concept then he or she must take the Pre-Quiz and get a perfect score. The answers are recorded but they are not presented for consideration of the final grade. If the student gets through the Pre-Quiz successfully he or she gets full points for that Concept regardless of the answers in the Post-quiz (which would not be required if a perfect score was made in the Pre-Quiz). The most important feature here is that a student will be able to take the Pre-Quiz only once and any score less than a hundred percent will prevent the skipping of a concept.

Post-Quiz: The Post-Quiz is mandatory for every student if he does not skip the Concept or does not score perfectly in the Pre-Quiz the first time. The Post-Quiz is important for every Concept and is assigned a particular weight by the POR to compute the final grade. The student has a choice of taking the Post-Quiz three times (at the most) to maximize acquired points in the quiz at Concept level but the point percentage decreases as the number of attempts increase. In the first attempt a total of 100% of obtained points is considered. In second and third attempts 90% and 80% of the total score obtained is considered respectively.

The Exam: The Exams are conducted at Unit level. The POR decides when and how (simultaneously or individually) the Exam should be taken by the student. Usually these Unit level Exams carry more weight than the Post-Quizzes for computation of grade, because they are comprehensive for each Unit.

3.1 Quiz INPUT and EDIT Modules:

These modules are for the CCFs only. The students do not have access to these modules. The main purpose of these modules is to simplify the task of quiz material submission by the CCFs. It also significantly reduces the time spent by them in the quiz creation process. This module provides options for creating different types of quiz questions including multiple-choice, true or false or any other questions in which a figure, a video or an audio are involved.

Figure 3:  Architecture of Evaluation Management System

The input module has a common starting page to all the CCFs who can select their particular course and then follow the hierarchy of Units, Concepts to enter or view quiz questions. The CCF is prompted with all the required input criteria like question text, type of question, number of answer choices for that question etc to enter a question. The number of answer choices in a multiple choice question can be chosen with an appropriate option. The CCF is then prompted to enter the corresponding number of answer fields. The Edit module provides for addition, deletion, and modification of a Quiz and its questions, edited items are immediately updated in the questions database as shown in figure 3. The “QUESTIONS” database serves as repository for all the questions and answers. 

3.2 Quiz ADMINISTRATOR Module:

The POR at the University who offers the course is the one who takes care of conducting the exam (when and how) and also administers the grades. Hence he should also be given a choice of configuring the Quiz. The Quiz administrator module is a tool used by the POR to configure the number of quiz questions and their corresponding weight (points) to be made available for Pre-Quiz, Post-Quiz and Exam. These configured Quiz questions are entered into the Quiz Configuration Database as shown in figure 3. The POR will also be able to decide when a Unit level exam has to be taken by the student.

3.3 STUDENT Quiz Module:

This module contains the actual logic for the presentation of a quiz to the student in randomized manner. Because the Pre-Quiz and Post-Quiz questions are pulled from the same database “QUESTIONS” as shown in the figure 3, question redundancy is avoided using the quiz results database. Once the Post-Quiz is answered the student can view the feedback immediately after which he/she cannot get back to the quiz again. Each student is provided with a complete list of the answered questions with the correct answers as well as alternative explanations apart from what he/she read in the course material. Also, for every question, the student is provided with a direct link to the page where this question related material was taught.

3.4 Quiz REPORT Module:

This module is used by the POR to tabulate statistical results related to quiz. He/She will be able to analyze all statistical information like the number of students who answered a particular question wrong, the difficulty level of the question, the class mean, median and mode can be determined. In this module, we also allow the CCF to access the database and get feedback from the quiz results; thereby allowing for future modification to the material (this is purely optional on the part of the CCF).

4. Implementation and Working Issues

The proposed solution for implementing the above designed Quiz system is as shown in figure 4. The DBMS used for this purpose is Sybase Adaptive Server Enterprise 12.5 where all the data resides.

 The programming language used is Java Server Pages (JSP) with Apache Tomcat 4.0.6 web server, “Tomcat is the servlet container that is used in the official Reference Implementation for the Java Servlet and Java Server Pages technologies” (http://jakarta.apache.org/tomcat). JSP is convenient to code by using conventional XML/HTML authoring tools and easy to modify because only the dynamic parts need to be changed when updating a web page.

JSP is a cross-platform alternative to traditional dynamic HTML generation technologies (Common gateway interface CGI, Active server pages ASP, etc) and works with any web server that complies to the Reference Implementation for the Java Servlet and Java Server Pages technologies as specified by Sun Microsystems.

 

Figure 4: The process flow

The added advantage with JSP is that it works seamlessly with XML making the entire application deliverable to a wide-variety of devices and browsers, with an appropriate XML publishing framework, which caters to these needs.

The World Wide Web Consortium has developed an Extensible Markup Language (XML) for applications that require functionality beyond the current Hypertext Markup Language (HTML). XML has now formed the foundation for a completely new way of communicating across the Internet. The power of XML to be applied universally to a number of areas lies in the fact that it is a clean standard and provides a very simple way for a document to carry information about itself (Das &  Lawhead, 2003,  p.72-79).

When the user requests a JSP page, Tomcat web server compiles the page and fetches the dynamic content specified  by JSP elements by interacting with the database through JavaBeans and well-formed XML is sent to  Cocoon, “Apache Cocoon is an XML publishing framework that raises the usage of XML and XSLT technologies for server applications to a new level” (http://cocoon.apache.org/2.0), where the appropriate XSL is picked up and transformed to HTML for the end user.

 5. Conclusions

Effective evaluation of a student taking a course online is a key factor involved in online course delivery. In the system presented here, the evaluation process has been combined to serve as a control mechanism as well. The evaluation is constant and ongoing. The student is not required to visit material that was mastered at a previous time. The student is also allowed to participate in a “teach/re-teach” environment that focuses on mastery and not just on grades. The concept quiz serves as a personal measure for the student and provided mechanism for the student to re-visit material not mastered in the original presentation. The integrated application developed for IAEGS consists of the system in its entirety i.e. the quiz input and edit modules, administrative module and student quiz module with a complete framework for statistical analysis through report module. Hence it is an effective tool, which controls and organizes the entire quiz system for an online learning environment.

6. References

Bosak, J. (1997).  XML, Java, and the Future of the Web. World Wide Web Journal, 2(4), p.219-
       227.

Cohen, Maxine S. & Ellis, Timothy J. (2002). Developing criteria set for an online learning   
       environment. Proceedings of 32nd ASEE/IEEE 2002: Frontiers in Education Conference,  
       Boston, MA.

Das, Manirupa & Lawhead, Pamela B. (2003). Information storage and management in large
      web-based applications using XML. The Journal of Computing in Small Colleges,  18(6),
       p.72-79.

Lawhead, Pamela B., Darbhamulla,  Ravikumar., & Das, Manirupa.(2003). Emulating the Real -
      World Expert for Online Course Delivery using Student Status Monitoring. Innovation and
      Technology in Computer Science Education (ITiCSE), Greece.

Meek, Carlysle (2002). Recreating The Expert. Retrieved July 1, 2003, from:
       http://geoworkforce.olemiss.edu/Fellows/RecreatingtheExpert.pdf

---

© Copyright 2003. The authors, Srinivas Chappidi, Keyur C. Patel, Pamela B. Lawhead  assign to the University of New Brunswick and other educational and non-profit institutions a non-exclusive license to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive license to the University of New Brunswick to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers, and for the document to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.