North American Web Developers Conference, October 1998:
This paper describes one process for evaluating the use of electronic bulletin boards, chat-rooms, e-mail, online testing and other electronic tools as vehicles for communication, information and evaluation in a traditional classroom teaching environment. Based on faculty experiences during a 3-month trial of Internet-based course management software at the Williamson College of Business Administration (http://www.cc.ysu.edu/wcba) of Youngstown State University (http://www.ysu.edu) [Ohio, USA], the paper will examine the techniques and strategies used to integrate electronic tools in the business curriculum. A student survey was administered to participating classes at the end of the quarter to assess usage, usefulness, attitudes, and achievement of pedagogical objectives. Students' reactions to the software's features varied depending upon their exposure to them and the extent to which they were used in the particular course. However, there was general agreement that the use of the course management software increased the frequency of students' contact with instructors and contributed to their overall performance in the course. Suggestions are made as to how such software may be effectively integrated into the "traditional" classroom without fundamentally altering the nature of the course.
Keywords: Internet-enhanced Education; Student Feedback; Evaluation; Online Components in Traditional Course Settings
In the U.S., the trend toward "global classrooms that harness the latest in computer and Internet technology" (1) has spurred many institutions of higher education to integrate facets of distance education into traditional teaching environments. A report by Coopers & Lybrand LLP notes that "leading colleges and universities are already investing in technology... that will transform the way educational services are packaged, delivered, accessed and measured." (2)
This paper examines the use of IntraKal (http://www.anlon.com), a class management and communication software program, by faculty in the Williamson College of Business Administration at Youngstown State University, during a three-month trial period, January - March, 1998. At the conclusion of the Winter academic quarter (March, 1998), students evaluated the frequency of access and usefulness of the software as well as their perceptions of its impact on their own learning.
Four faculty members, who taught courses in Accounting, Management and Marketing, used the features of this software to provide additional sources of information to students, as a vehicle for communication, and as a tool to evaluate students. Classes were traditional lecture sections meeting 3-4 hours weekly with required texts and class attendance required or strongly encouraged. Use of the software was required by some instructors, but offered as supplementary information by others. The software was not meant to fundamentally change the nature and/or delivery of the course.
The software was accessed by students on and off campus via the Internet, using standard web browsers. Although all four faculty members had previous Internet experience, this was the first intensive exposure to the Web for many students. Students in the business school do not live on campus (98% are commuter students), are older than traditional students, and many lack previous Internet experience. A college-wide survey conducted in September 1997 indicated that 39.1% of students report that they do not have the ability to access the Internet at home, work or school. (3)
No formal assessment of faculty motives was done prior to this trial, but increasing and/or facilitating communication, providing useful online experience for students, and managing the exam and grading process were mentioned as goals for the software. The software was installed in December, 1997 and training for faculty was provided just prior to Winter Quarter, 1998.
Each instructor incorporated different features of the software appropriate to each course. Faculty's choice to use these various aspects of the software depended upon the course level, subject matter, differences in teaching styles and individual faculty preferences. The majority of this paper will focus on the two general uses for the software which the instructors found to be the most useful: communicating with students and administering and grading exams.
All courses had some course information available online. At a minimum, this information included a course syllabus and information concerning the various ways in which the instructors could be contacted. Several of the course instructors also used other features of the software extensively. This included providing students with lecture notes, case materials, sample exams, and assignments. In addition to these static sources of information, faculty also used interactive forms of communication -- e-mail, electronic bulletin boards and chat-rooms including online office hours.
E-mail was an integral part of this trial. Students were provided with University e-mail accounts, although some students preferred to use e-mail accounts from commercial vendors. When students registered online for these courses, they were required to specify a personal e-mail address. This address was entered into faculty rosters and gradebooks. When assignments or exams were completed, confirmation e-mails were sent to students' personal accounts. Faculty could also send bulletin board postings or other notices quickly to a class section or several sections of the same course using this e-mail roster. In addition, students could contact faculty members through e-mail links at several locations within student menus.
The electronic bulletin board and chatrooms were used only slightly. Each course bulletin board allowed faculty and students to post messages for all class members to read. A "response" feature allowed students comments to be organized by topic. Through the bulletin board, students could also send anonymous messages to the faculty. These messages were not posted on the bulletin board, but sent directly to faculty e-mail with a generic system return address that was not traceable to individuals. Chatrooms were created for each course, for departments, for the college as a whole, for each level of students (i.e., freshmen, sophomores, etc.). Some instructors used the course chatroom to hold online office hours at pre-specified times.
Two faculty members used online testing with immediate feedback and all used sample exams. All students had online access to exam and course grading information. One of the more attractive features of the software is the capability it provides for administering online exams. The capacity exists for students to take the exams from any computer with an Internet connection. Without the means for monitoring students at remote locations and with no expectation of students adhering to a self-monitored honor code, the taking of exams was generally restricted to computer labs under the supervision of the instructors. However, several instructors used the "remote exam-taking feature" to give students "sample" exams that they could complete at their leisure.
In most cases, students met as a class and took the actual exams as they normally would have, except that the exam questions were accessed via the Internet. Upon completing the exams, students were able to indicate that they were ready to have their exams graded and would then receive immediate feedback as to how well they had done on the multiple choice portion of the exam. After all class members had taken a particular exam, e-mails with "missed" questions and other exam information could be sent to an entire course section by the faculty.
Some faculty members had essay questions on exams or used this feature for other assignments. When essay type responses were required, students' answers were placed in a file which instructors could access at their convenience for grading. Student responses were coded by student numbers preserving anonymity. A"corrected" response could be e-mailed back to students. Although the immediate feedback was not available for essay responses, there were still a number of advantages to be realized with such a system. Among these advantages were: students' ability to easily edit their responses before submitting them, evaluators being able to avoid the biases created by students' handwriting, and relatively quick feedback - in that students could have their scores and the evaluator's comments e-mailed to them before the next scheduled class meeting.
Once exams or essays were completed, students' grades were placed in the instructors' "online grade books." Students were able to access their grades at any time during the term using their personal user names and passwords. This kept the students' progress in the course salient in their minds, allowed the instructors to answer any questions about grades (generally via e-mail), and make any necessary changes/corrections more quickly. A particularly appealing feature of this type of feedback was that it empowered students with direct access to information typically under the strict control of the instructor in a "traditional" classroom.
A 5-page photocopied survey was administered by the instructor during the last week of classes in the academic quarter to 138 students in eight courses. In addition to Likert-type scale questions that rated usage, usefulness, and opinions about the software, students were asked to assess the frequency of their online usage and classroom attendance. The students responded on seven-point scales ranging from "very strongly agree" to "very strongly disagree." In the Results section that follows, all data have been coded so that "high" values (approaching seven) indicate more favorable reactions to the software. Some instructors gave extra credit for completion of the survey. SPSS was used to tabulate the results.
It should be noted that the differences in individual faculty usage of the software may skew data to underestimate student satisfaction. Students in courses where the faculty member did not use a particular feature or used it infrequently may be more likely to choose values on the low end of the scale, like "very strongly disagree," "very seldom," or "not at all useful." While these are valid opinions, it is also important to examine students with greater exposure to particular features of the software to determine "experienced user" opinions of usefulness and value.
In the results that follow, mean values were calculated using all data. Not applicable values were coded at 0 to indicate the least frequency or usefulness. Percentage values ignored the 0 values, and the 7-point scale was collapsed to 3 values for easy reference. For example, for item 3, "More Contact", responses 1-3 were grouped as "Disagree," response 4 remained "Neither Agree nor Disagree," and responses 5 -7 were grouped as "Agree." Instructor crosstabs also ignored the 0 values and were grouped. All data have been coded so that "high" values (approaching seven) indicate more favorable reactions to the software. A summary of data results is attached for reference.
Chatrooms, item 2d, (m=1.39) and online office hours, item 2f, (m=1.48) were not generally considered useful. Neither feature was stressed by faculty, and given the nature of student's comfort and ability to access the Internet, these values may be expected. It is unclear however, whether these features were perceived as less useful because they were not used, or whether they were not used because they were perceived as being less useful. While the results for bulletin boards, item 2e, (m=3.14) seem to indicate a "somewhat" useful attitude, examining the data for faculty with greater use of these features results in a perception of greater usefulness. For one instructor, the mean usefulness of bulletin boards is 4.92, with 66.7% of students returning a value of 5, 6, or 7.
The most useful feature was the ability of students to access grading information. While the mean value for this item, 2i, (m=4.35) shows that most students found this feature "somewhat useful" to "very useful," eliminating the not applicable values indicates that only 14.7% found this feature less than "somewhat useful," and one-third (33.3%) chose the highest level on the scale - "very useful."
The average student ranked sample exams, item 24e, (m=4.87) and exam feedback, item 24f, (m=5.05), somewhat lower than traditional learning methods, including studying the text, item 24a, (m=5.22) and attending class, item 24b, (m=6.11), "in terms of how useful they were to [the student] in working toward the grade expected in the course." This average does not fully describe students' assessment of the usefulness of the software since different aspects of the software were utilized by faculty. For example, sample exams, item 24e, were considered useful by 57.2% of all students, but those in courses with the opportunity to take sample exams found them more than "somewhat" useful (86.4%). Although the software was not intended as a replacement for traditional teaching methods, students seemed to find its features useful.
One instructor exclusively used the online software for exams, one instructor never used the exams, and the remaining two had limited online testing. Although the mean frequency reported for item 1g - "Taking exams" was 2.56, the value soars to 6.54 in the courses where online testing was the only method of evaluation, and ranges from 2.39 to 0 for the other courses. Similarly, the usefulness of taking exams in courses with online testing, item 2f, had a mean 6.54 with 100% of students rating it useful (values 5-7) and 79.2% of students choosing the "very useful" value (7).
Overall results indicate that students felt that the software was considered a valuable addition, item 14, (m = 5.18) with 70.9% of students indicating that they agreed with the statement "Overall, I feel as if the online classroom software was a valuable addition to this course" (values 5-7.) In the courses solely using online testing, the software was rated as slightly more valuable with 87.5% of students agreeing.
The performance measure, item 15, is slightly less, with over one-third of students (38%) neither or agreeing nor disagreeing with the statement "Overall, I feel as if the material that was made available on the online classroom software improved my performance in the course" which contributed to the low mean value (4.59) for this item. Again, the results for the students in the courses with exclusive online testing are significantly different, with no students disagreeing.
"The challenge is to make sure technology remains a means rather than an end." (4) For this trial, the software was not a substitute for the traditional aspects of the courses, but it is clear that online software provided students with additional opportunities for communication, 24-hour access to course and grading information, and alternative evaluation opportunities.
For faculty, the software provided a "cut and paste" method for creating online tests from existing testing materials. Less paper and more opportunities for student interaction were also benefits. Using online exams and essays significantly reduced grading time and facilitated gradebook management. The ability to grade essays anonymously and the ease of communicating with students "between class sessions" were also considered helpful.
Students, even those without prior Internet experience, reported that the software was easy to use and understand. Good menus and clear instructions contributed to this perception. The ability to get questions answered quickly and easily "in real time" was also a benefit. Expanding discussions "out of the classroom" through bulletin boards and chatrooms may also benefit students with more opportunities to interact, not only with faculty, but with other students in a commuting-student environment.
All faculty would agree that this process involved more preparation time (at least initially). Learning to use the software was also time-consuming. The increased time spent in communication with students was balanced by the time saved from "not having to repeat the same answer 50 times" (through use of the bulletin board) and no longer having to "play telephone tag." Since the faculty in this trial were all comfortable with computers and had previous Internet experience, learning time (and frustration levels) would increase if faculty with little or no computer and/or Internet experience were required to implement this software.
Some students reported difficulty accessing the software, ranging from inability to connect to the Internet from commercial sources at peak hours to inability to use web browsers effectively. Some of these problems may have been due to students' lack of prior exposure to the Internet, or lack of access at home (for commuting students). As student and faculty Internet exposure and access increases, most of these challenges will disappear.
Organizational computing infrastructure was also a challenge. A server and system administrator is required to store and organize information. The availability of hardware was also a factor that prevented all faculty from utilizing online testing.
While the authors agree with our colleagues at the University of Washington that "Education is not reducible to the downloading of information," (5) it is clear that students in traditionally-taught courses gain significant benefits by using online software as a supplement to other teaching methods.
1. David Wilson qtd. in Peggy Frank Freeland, "Distance learning: Miracle tool or rote teaching?" Columbus Business First 24 Aug. 1998. http://www.amcity.com/columbus/stories/082498/focus2.html (return to text)
2. Tom Diederich, "Tochnology may spark education revolution," Computerworld. Online edition. 21 May 1998. http://cnn.com/TECH/computing/9805/21/education.idg/index.html (return to text)
3. Internal Memoranda, Williamson College of Business Administration, Youngstown State University. Youngstown, OH. November, 1997. (return to text)
4. Wilson, ibid.(return to text)
5. Todd Woody, "Academics rebel against an online future," The Industry Standard. Online edition. 12 June 1998. http://www.cnn.com/TECH/computing/9806/15/academics.idg/index.html. (return to text)
Diane Ciba, MBA
Research Assistant/Marketing PhD Student
University of Connecticut
12 Homestead Avenue
Waterbury, CT 06705 USA
diciba@aol.com
Thomas Rakestraw, PhD
Associate Professor/Management
Williamson College of Business Administration
Youngstown State University
1 University Plaza
Youngstown, OH 44555 USA
rakestra@usaor.net
© 1998. The authors, Diane Ciba and Thomas Rakestraw, assign to the University of New Brunswick and other educational and non-profit institutions a non-exclusive license to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive license to the University of New Brunswick to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers, and for the document to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.