The Validation of a Measurement Instrument Relationships Among Student Demographic Characteristics, Student Academic Achievement, Student Satisfaction, and Online Business - Course Quality Factors Kuan-Yao Chiu, Richard Stockton College of NJ (USA) chiuk@stockton.edu Bob Stewart, University of Missouri-Columbia (USA) Mark Ehlert, University of Missouri-Columbia (USA)

Abstract

A survey instrument developed to understand the relationships among student academic achievement, student satisfaction, course quality factors, and student demographic characteristics is described. The instrument was determined to be reliable and included 45 items related to perceived course quality. Four additional items focused on student satisfaction and 11 items pertained to student demographic parameters. The relatively high Alpha Coefficients for five factors (course design, α = .9593; interaction with instructor, α = .9209; technological feature, α = .7588; interaction with students, α = .7536; student satisfaction, α = .7788) indicate that the factors were internally consistent.  Instructors could use the instrument to obtain information on all facets of their teaching and to improve their online teaching. Educational institutions can improve their online courses by investigating the characteristics, and the needs and wants of their new target market, online students, by using data collected with the questionnaire.

Introduction

Rapid technological developments and structural changes in the economy have expanded the need for adults to continue learning. Non-traditional students are often unable to attend classes offered during daytime hours. Educators and administrators are challenged to provide continuing educational opportunities that are tailored to the needs of today’s citizens in order to accommodate both traditional and non-traditional students (Baird & Monson, 1992; Fradkin, 1993; Glaser, 1993; Martinsen & Adams, 1993; Riddle, 1994).

There are various delivery methods for distance education such as print-based, audio/radio-based, video/television-based, computer-based, or satellite-based (Stewart, Keegan, & Holmberg, 1983). Technological advancements applicable to distance education methodologies such as cable television, fiber optics, microwave, slow scan television, satellites, and microcomputer networking have opened new opportunities for students (Barker 1987; Kitchen & Russel, 1987). 

Several years ago the Internet was a computer-based, text-driven communications system for scientists and academics. The World Wide Web (WWW) has emerged with the use of graphic devices and point-and-click navigation which provide the mechanism for learners to search, select, and synthesize information, to discover how and where to find answers and solutions, and to understand, transform, and present ideas. Forsyth (1998) stated that, “There is a great deal of hype about the Internet and the ability of this service to open up access to information for all. One view of the Internet is that it is the alternative method of delivering existing course material” (p. 13).

According to a review of consumer satisfaction conducted by Yi (1990, p. 68-70), customer satisfaction has been defined in two ways, either as an outcome or as a process. The outcome definition characterizes satisfaction as the end-state resulting from the consumption experience. Satisfaction also has been considered as a process, emphasizing the perceptual, evaluative, and psychological processes that contribute to satisfaction.

The foregoing suggests one may extrapolate consumer satisfaction to situations where the student is considered a consumer. The approach to evaluating quality of education used by others is not contrary to established educational psychology theory and research. Guolla (1999) used this approach in which he considered students as customers, reflecting the fact that they have experienced a highly valued service. Guolla stated “students would consider themselves expert consumers of the service experience since they have taken numerous courses previously” (p. 3).

Several studies have investigated relationships between student academic achievement and student satisfaction for a number of delivery methods. No study has extrapolated established customer satisfaction theory to online courses. The purpose of the study was to establish an instrument to assess student achievement, student satisfaction, demographic factors, and student perceptions of online course quality.

Instrument

A modification of an established instrument was used to assess student achievement, student satisfaction, student perceptions of course quality, and demographic factors. A survey instrument originally created by Virginia Polytechnic Institute and State University with the aid of the FlashlightTM Current Student Inventory (TLT Group, 1997) was used. These combined efforts produced an instrument with 71 multiple-choice items and 4 opportunities to enter free-form comments.

The Virginia Polytechnic Institute and State University instrument was modified to achieve the specific research goals of the study. The University of Missouri – Columbia doctoral committee contributed to the modification of various items. The final questionnaire product contained 50 multiple-choice items proceeded by Likert-style questions to gain students’ comments about the online courses and another 11 open-end questions to obtain students’ demographic data (Chiu, 2001).

A pilot study established the reliability of the survey instrument by employing it in two graduate education courses for a total pilot test of 15 participants. The 50 items of the survey instrument were deemed reliable as measured by Cronbach Coefficient Alpha = .8830, a coefficient indicating the reliability of any measurement of the underlying construct of a set of items being evaluated.

For the main research evaluation, three hundred and ninety-two survey packets were sent to the college students, and four were returned as being wrongly addressed. One hundred and eighty-eight were completed and returned yielding a response rate of 48%.

Results

Table 1 presents the results of the Kaiser-Meyer-Olkin (KMO) and Bartlett’s Test. Tabachnick and Fidell (1996) stated that “Bartlett’s (1954) test of sphericity is a notoriously sensitive test of the hypothesis that the correlations in a correlation matrix are zero.” They continued “…because of its sensitivity and its dependence on N, the test is likely to be significant with samples of substantial size even if correlations are very low. Therefore, the test is recommended only if there are fewer than, say, five cases per variable” (p. 641). “The KMO measures the sampling adequacy which should be greater than 0.5 for a satisfactory factor analysis to proceed” (University of New Castle upon Tyne, 2002, ¶ 12). Bartlett’s Test of Sphericity was significant. The KMO measure was .923, exceeding the .50 required value for factor analysis.

Table 1
KMO and Bartlett’s Test
Kaiser-Meyer-Olkin Measure of Sampling Adequacy	.923
Bartlett’s Test of Sphericity
Approx. Chi-Square	5396.914
df	903
Sig.	.000

Principle components extraction through SPSS on 45 items for a sample of 188 college students was used in an initial run to estimate the number of factors with forced eigenvalues that exceed one. Eight components were extracted (Chiu, 2002). However, eigenvalues for the first four factors were all larger than two and, after the sixth factor, changes in successive eigenvalues were small. This was taken as evidence that there were probably between four and six factors. The solution that was evaluated, interpreted, and reported was the run with Principal Factors Extraction, Varimax Rotation, which yielded five factors (Chiu, 2002).

Tabachnick and Fidell (1989) recommended that, “as a rule of thumb, only variables with loading of .30 and above are interpreted” (p. 640). However, they continue, “Choice of the cutoff size of loading to be interpreted is a matter of researcher preference” (p. 640). With a cut of .30 for inclusion of a variable in interpretation of a factor, all 45 variables load on these 5 factors. Two of the variables in the solution, variable 29 and variable 45 were complex with low factor loadings and were deleted.

Table 2 shows the mean value of students’ ranking of each item (strongly agree = 5, agree = 4, neutral = 3, disagree = 2, and strongly disagree = 1). Also tabulated are the item standard deviations and the reliability coefficients for each perceived course quality factor and the items that compose them. The relatively high Alpha Coefficients for four factors (course design, α = .9593; interaction with instructor, α = .9209; technological feature, α = .7588; and interaction with students, α = .7536) indicate that the factors were internally consistent. Factor 5, benefits of online course, with its low Alpha Coefficient (α = .4894) was purposely excluded to reduce the relevant factors to four. This approach was chosen as the best way to examine the influence of the independent variables in this study.

An additional grouping of items 46, 47, 49, and 50 was summed to yield a measure that denoted student satisfaction. Table 3 shows item means and item standard deviations for student satisfaction. The relatively high Alpha Coefficient (α = .7788) indicates that the scale was internally consistent.

Table 2
Item Means, Item Standard Deviations, Factor Scores Reliability, and Factor Loading
Item number	Item			M	SD			Factor loading
Factor 1: Course Design (α = .9593)				4.0777		.6960	--
12	The course helped me to understand the ideas and concepts			4.1915	.77772			.817
13	I learned how to apply techniques to complete activities in the course			4.0053	.85592			.793
9	Course assignments were relevant and helped me learn the course material			4.1702	.91498			.780
14	The course developed my ability to solve problems			3.7807	1.00270			.778
11	I learned about principles and theories in the course			4.1223	.85339			.769
15	The course materials prepared me for assessments and examinations			3.9305	.98680			.763
5	The course materials helped me learn			4.1915	.91074			.723
17	The tests and assignments were suited to the objectives of the course			4.0851	.87337			.708
4	The written objectives agreed with those actually taught in the course			4.2995	.77814			.703
18	The tests and assignments enabled me to demonstrate what I had learned			3.9468	1.01715			.682
19	The technology used in the course was appropriate for performing the tasks required			4.1011	.90477			.670
8	I was challenged to solve problems in the course			4.3191	.79049			.662
2	The course aims and objectives were made clear			4.3351	.80735			.635
26	I acquired skills that will be useful in my profession			3.7473	1.01624			.635
1	The course was well organized			4.3191	.79722			.617
40	Methods of evaluating my work were fair and appropriate			3.9358	1.00594			.608
6	All materials and resources needed to complete the class were accessible			4.2447	.90968			.580
3	Grades were assigned fairly			4.1444	.94530			.578
16	The time and effort required for completing assignments was what I expected for the course			3.6064	1.15826			.470

Factor 2: Interaction with Instructor (α = .9209)		3.6368			.8961		--
36	I received detailed comments on assignments from the instructor	3.5851		1.28262				.782
37	I received comments from the instructor on assignments quickly	3.6684		1.27804				.781
39	Feedback on examinations and other graded materials was valuable	3.7189		1.16404				.728
38	I was encouraged to ask for clarification when I didn’t understand something	3.8777		1.14272				.683
44*	I felt isolated from the instructor	3.6915		1.23692				.673
41	I felt comfortable asking the instructor an awkward question	3.75		1.15913				.662
42	I felt comfortable disagreeing with the instructor	3.4677		1.143				.614
35	I was encouraged to interact with the instructor about the ideas and concepts taught in the course	3.9362		1.08276				.593
43	I was encouraged to discuss my academic goals and/or career plans with the instructor	3.0919		1.18691				.582
22	The procedures for taking exams and/or quizzes in the course caused problems	3.5806		1.02193				.518
Factor 3: Technological Features (α = .7588)		3.7292	.7442			--
23*	I spent too much time trying to access the course site on the WWW	3.7606		1.17966				.764
20*	The technology used in the course did not work properly	3.9568		.96919				.725
10*	There are other features and functions that should be added to the main web site for the course	2.828		1.06599				.619
7	The main web site for the course was easy to use	4.1649		.86491				.616
21*	The procedures for taking exams and/or quizzes in the course caused problems	3.9355		1.10716				.615
Factor 4: Interaction with Students (α = .7536)		3.6008	.7584			--
30	I interacted with other students about the ideas and concepts taught in the course	3.4947		1.12577				.779
33	The course allowed me to work on assignments with other students	3.1277		1.16760				.703
32	The course allowed me to communicate with people from outside campus	3.7754		.86081				.674
34	I actively participated in scheduled interactions about the course material	3.8777		1.05513				.632
31*	The course made me feel isolated from other students	3.7287		1.10701				.484

Factor 5: Benefits of Online Courses (α = .4894)		4.1711		.5440	--
27	Because the course was online, it was easier to juggle my course work with my work and/or home responsibilities	4.4628	.97766			.620
24	My computer skills were adequate to complete the course	4.5266	.64113			.615
28	Because the course was online, I put in less time traveling to and from class	4.6310	.78604			.531
25	Because the course was online, I was more confident that I would do well	3.0638	1.00596			.325

* Indicates that the numerical coding order was reversed.

Table 3
Item Means, and Item Standard Deviations of Student Satisfaction
Item number						M		SD
Student Satisfaction (α = .7788)					4.0325		.87884
46	I will take another online course				4.5784		.80478 1.3126 1.15824 1.1949
47	I prefer to take all course online				3.5189
49	I would recommend the course to others				4.0270
50	Overall, I have been very satisfied with the course				4.0054

Conclusion

Technological advancements have led to new developments in distance education methodologies. This survey instrument was determined to be reliable and included 45 items related to perceived course quality. Four additional items focused on student satisfaction and 11 items pertained to student demographic parameters. Instructors could use this survey instrument to obtain data and perform appropriate analysis to obtain feedback that should be useful in making changes to improve their online teaching. The instrument described and evaluated in this paper is offered as a valid tool for finding the most appropriate approaches and tactics to ensure that online courses are effective.

References:

Baird, M., & Monson, M. (1992, Fall). Distance education: Meeting diverse learners’ needs in a changing world. In M. J. Albright & D. L. Graf (Eds.),  New Directions for Teaching and Learning (pp. 65-76). San Francisco, CA: Jossey-Bass.

Barker, B. (1987). Interactive distance learning technologies for rural and small school: A resource guide. ERIC Mini-Review. New Mexico State University, ERIC Clearinghouse for Rural Education and Small Schools. Las Cruces, New Mexico.

Chiu, K. Y. (2002). Relationships among student demographic characteristics, student academic achievement, student satisfaction, and online business – course quality (Doctoral dissertation, University of Missouri, Columbia, 2002).

Fradkin, B. & Hisle, W. (1993). Harnessing the future: Administrative support for learning resources. Community College Journal, 63(5), 24-29.

Forsyth, I.(1998). Teaching and learning materials and the internet. London: Kogan Page Limited

Glaser, R. (1993). Is distance education something you can afford to detour? School Business Affairs, 59(5), 19-22.

Guolla, M. (1999). Assessing the teaching quality to student satisfaction relationship: applied customer satisfaction research in the classroom. Journal of Marketing Theory & Practice, 7 (3), 87-97.

Kitchen, K., & Russell, J. T. (1987). Educational telecommunications: An overview of Minnesota’s state founded projects. T.H.E. Journal, 14(8), 55-56.

Martinsen, K. & Adams, H. (1993). Creating a district plan for technology. Book Report, 12 (2), 25-27.

Riddle, J. F. (1994). Factors which contribute to grade achievement and course satisfaction of distance-education students. Unpublished doctoral dissertation, University of Northern Colorado.

Stewart, D., Keegan, D., & Holmberg, B. (1983) Distance education: International perspectives. New York: St. Martin’s Press Inc.

Tabachnick, B.G. & Fidell, L. S. (1989). Using multivariate statistics. New York: HarperCollins Publishers Inc.

Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics. (3^rd. Ed). New York: HarperCollins College Publishers.

University of New Castle upon Tyne. (2002, March). Statistics support: How to perform and interpret factor analysis using SPSS. Retrieved March 12, 2002, from http://www.ncl.ac.uk/ucs/statistics/common/specialisttopics/factor_analysis/factoranalysis.html

Yi, Yougae (1990). A critical review of customer satisfaction. In V Zeithmal (Eds.), Review of marketing  (pp.68-123). Chicago: American Marketing Association.