North American Web Developers Conference, October 1998:

What Do Web-Based Courseware Developers Discuss? A Qualitative Analysis of the WWWdev Listserv

Rik Hall
University of New Brunswick

Allan Ellis
Southern Cross University

ABSTRACT

Listservs with large, global subscriber bases and a focus on professional issues provide excellent opportunities for informal professional development. The WWWdev listserv was set up in 1995 to provide a forum for the discussion of issues arising from activities related to the development of Web-based courseware. In its three and a half years of operation the list has attracted a subscriber base of over 1400 people from around the world and has received over 5000 postings.

Qualitative analysis techniques provided the framework to explore the dialogue and exchanges embodied in the list postings. Analysis by category at the level of the individual postings revealed that the major issue/subject of discussion was clearly people looking for products and course information (24.5% of all postings). Not surprisingly the next highest category is information about products and courses (14.6%). Clearly a major function of the listserv is exchange of information. Two other categories that rated a significant percentage of responses were questions/comments about Distance Education (12.8%) and software (14.4%). These postings involved subscribers probing for specific information in particular subject areas.

A preliminary study of only the 1998 data using the Australian developed qualitative analysis software package NUD.IST, provided a more fine grained analysis involving the development of a 'tree structure' and the ability to generate matrix intersection searches. While detailed computer coding is initially more time consuming than the generation of a simple categorical listing, the former does provide a powerful search and retrieval facility that cannot be matched by simpler analysis tools.

Keywords: professional development, Web-based courseware development, WWWdev listserv, qualitative analysis techniques

The setting up of the WWWdev listserv

The WWWDEV listserv was set up by Rik Hall at the University of New Brunswick on March 30,1995. As the field of Web-based courseware development is relatively new and still rapidly evolving, educators and trainers, instructional designers and programmers have no substantial, traditional publications base or professional association to meet their staff development needs.

The listserv was started as a result of a dialog on a national teleconference sponsored by the Canadian Association for Distance Education (CADE). During the teleconfernece Rik Hall asked if anyone else was involved in the development of Web-based courses and was there a source of information for those developers. When the latter question was not answered, Rik Hall offered to start such a forum. That weeked the listserv was started, and by Monday morning more than two hundred people had joined the WWWDEV listserv.

Since the list was created it has continued to grow each year. At the end of 1995 there were about seven hundred members, at the end of 1996 a thousand, at the end of 1997 1200, by mid 1998 1400. These subscriptions now come from over 25 countries and the membership remains around 1400 people.

Qualitative Analysis

Qualitative analysis techniques are now well established as research tools and their history can be traced back well over one hundred years (Tesch, 1990). While today there are a range of well accepted approaches to qualitative analysis, the application of classical 'content analysis' (Berelson, 1952) and 'grounded theory' (Glasser and Strauss, 1967) are the two approaches used in this project.

The data from all years was subject to a coarse grained analysis using the basic principles of categorizational analysis outlined by Tesch (1990). This approach involved using a stepwise process that proceeds from an overview of the data to identifying topics and finally recognising categories.

The final categories were determined from the actual data. A sample of the data identified initial topics. Topics that recurred in more data were elevated to the status of categories. This is a standard qualitative research technique.

While the traditional approach of categorization provides a basic level of data reduction it allows little scope for reworking or more detailed analysis. It is the 'grounded theory' approach of Glasser and Strauss (1967), combined with the rapid processing capability of today's desktop computers, that provides the opportunity for detailed analysis. By adopting a grounded theory approach, researchers try to ensure their understanding emerges from the data rather than being a product of imposing a pre-defined structure on the analysis.

Consistent with qualitative research, data is analysed inductively through categorization and comparison and the search for relationships within the data. The analysis process is systematic and comprehensive, but not rigid, and usually entails three concurrent flows of activity: data reduction, data display and conclusion drawing and verification (Miles and Huberman, 1994).

The Australian developed software package NUD.IST 3.0 (Non numerical Unstructured Data Indexing Searching and Theory-building) developed by Qualitative Solutions Research Pty Ltd (http://www.qsr.com.au) provides a powerful tool for what Tesch (1990) has described as the "de-contextualisation" and "re-contextualiation" of data. This she believes leads to theory building by allowing "the search for co-occurring or overlapping coding within the data... plus the corresponding search for counter evidence".

Analysis by category and results

This involved categorising some 5228 postings made to the listserv in the three and half year period from 1995 to August 1998. Twelve categories were developed from an initial set of topics based on the researchers overall knowledge of the total data set. Figure 1 shows the number of postings to the twelve identified categories by year and total postings since inception of the list.

Year	1995		1996		1997		1998 (to 26/8)		Since inception
Category	no.	%	no.	%	no.	%	no	%	no.	%
1.Distance Ed Questions/comments	153	14.2	195	13.8	206	12.3	117	11.0	671	12.8
2. General Computer Questions/Comments	60	5.6	31	2.2	48	2.9	6.5	6.1	204	3.9
3. Hardware Questions/Comments	9	0.8	3	0.2	15	0.9	12	1.1	39	0.7
4. HTML Questions/Comments	211	19.6	99	7.0	6.3	3.8	47	4.4	420	8.0
5. Listserv Questions/Comments	109	10.1	74	5.3	112	6.7	4.6	4.3	341	6.5
6. Looking for Products/Info/courses etc	119	11.0	318	22.6	428	25.6	414	38.8	1279	24.5
7. Off Topic	43	4.0	66	4.7	6.4	3.8	2.3	2.2	196	3.7
8. Products/Info/Courses Available	127	11.8	211	15	300	17.9	125	11.7	763	14.6
9. Software Questions/Comments	122	11.3	239	17.0	270	16.1	120	11.3	751	14.4
10. Web Design Questions/Comments	92	8.5	55	3.9	101	6	74	6.9	322	6.2
11. Web Development Questions/Comments	27	2.5	91	6.5	6.3	3.8	21	2	202	3.9
12. Web Maintenance Questions/Comments	6	0.6	27	1.9	5	0.3	2	0.2	40	0.8
Totals	1078	100.0	1409	100	1675	100	1066	100	5228	100

Figure 1 Number of postings to the 12 identified categories by year and total postings since list inception.

The categories and pattern of postings reveal something of the history of the Web and its development. In 1995 the category attracting most interest was HTML (19.6%) which is not surprising since at that time most developers were hand coding their tags! The next most active category is questions /comments about Distance Education (14.2%) which suggests that the pedagogical issues involved in online delivery is a key concern of developers. Indeed this category has remained above 10% in every year and averages 12.8% overall.

The category which has experienced the most growth over the years (11%, 22.6%, 25.6%, 38%) and which accounts for the single biggest overall percentage (24.5%) is the 'Looking for Products/Info/Courses'. Clearly most subscribers use the listserv to request information. This category combined with 'Product/Courses/Info Available' (14.6%) indicates that some 40% of listserv traffic is directly about the exchange of information.

A more detailed analysis of the 'Looking for Products/Info/Courses category' is provided in Figure 2 where some 9 topics (sub-categories) are identified. It is worth noting that topics like 'design advice' and 'publishing information' have received a steady increase in postings each year.

Year 1995 1996 1997 1998 (half year) Since Inception

Topic no. % no. % no. % no. % no. %

1. Courses/Conferences 11 9.2 35 11 46 10.7 34 8.2 126 9.9

2. Design Advice 0 0 3 0.9 27 6.3 46 11.1 76 5.9

3. Help with survey 4 3.4 13 4.1 10 2.3 14 3.4 41 3.2

4. Methods/shortcuts 17 14.3 31 9.7 70 16.4 28 6.8 146 11.4

5. Other 33 27.7 62 19.5 62 14.5 97 23.4 254 19.9

6. Product/Software 28 23.5 55 17.3 99 23.1 69 16.7 251 19.6

7.Published Information 0 0 54 16.7 33 7.7 68 16.4 155 12.1

8. Service 4 3.4 0 0 10 2.3 6 1.4 20 1.6

9. Web address 22 18.5 65 20.5 71 16.6 52 12.6 210 16.4

Total 119 100 318 100 428 100 414 100 1279 100

Figure 2 Postings to 'Looking for products/Info/Courses' category based on topic (sub-category) and year.

Postings to the 'software' category based on seven topics (sub-categories) reveal that issues concerning general Web software are dominant in every year.

Figure 3 Postings to the software category based upon 7 topics (sub-categories) and year.

Analysis using NUD.IST and sample results

As an example of the potential for fine grained analysis using specialised software, a block of postings from January to June 1998 were selected. These postings then underwent a code and retrieve process that consisted of labeling passages of the data then collecting identically labeled passages. The process involves noting regularities, patterns, explanations, casual flows and propositions. Theorising and concept definition occurs concurrently with the data analysis and shapes the analysis process.

As the process progresses the search and retrieve facilities of NUD.IST facilitate faster coding of data, for instance, to code all data related to Java, a simple search facility retrieves all text units containing the term then a "spread" function allows the researcher to re-contextualise these statements. The indexing system used by NUD.IST also allows for a more advanced interrogation of the data, allowing more complex responses to questions than would be possible through a purely categorizational summary.

For example, the following types of questions could be asked:

Whether one subscriber dominates the conversations about a particular issue.
Whether there are any gender differences; e.g., are males more likely to discuss one issue more than females.
Whether conversational relationships develop between two or more subscribers on a set issue.
Whether issues related to, say, course development are being discussed more by subscribers from one setting than another.

Of course the unstructured nature of the data retrieved from the postings has meant that not all information is available on the nature of all participants (e.g., gender) and this is a limiting factor in relation to some lines of questioning. The following Figures illustrate some of the features of NUD.IST which clearly needs to be demonstrated live to show its full analytical capacity.

Figure 4 Screen grab of coding screen showing a number of windows: Node Explorer Window at top left, Document Explorer Window at bottom left and Document Browser Window on the right with coding toolbar in the top right hand corner.

Figure 5 Shows an export table generated from a previously run index search; this particular search shows the amount of text (lines) coded by gender and country of origin.

Figure 6 Choosing the 'Make Report' option in the Node Explorer Window and selecting a node, in this case (3,3) will show all comments made about management systems.

Figure 7 shows how a matrix intersect search on time period and languages could be used to show when a subject was topical. Of course this would only be meaningful if data covered an extended time period.

Figure 8 shows search results of an intersect search for node (3 5) chat and (18 2) student involvement showing the discussion about chat software playing a part in encouraging student involvement.

Conclusions

While the WWWdev listserv provides a daily accessible professional development resource for subscribers its major use remains the basic exchange of information on products and courses. The tabulated data presented in the tables and graph provides a very basic summary of the nature of the subscriber exchanges when compared with the detail and the intricate relationships that can be explored by using the coded data that can be manipulated by the NUD.IST program.

Acknowledgements

The authors which to thank the research assistants who contributed to this project: Jason Bourque (UNB) and Renata Phelps and Toni Ledgerwood (SCU).

References

Berelson, B. (1952) Content Analysis in Communications Research, New York, Free Press.

Glasser, B. and Strauss, A. (1967) The Discovery of Grounded Theory, Chicago, Adeline.

Miles, M.B. and Hubeerman, A.M.(1994) Qualitative Data Analysis: an Expanded Sourcebook, 2nd ed, Thousand Oaks, Sage.

Qualitiative Solutions and Research (1997) Q.S.R. NUD.IST User Guide, 2nd Edt, Melbourne, Qualitative Solutions Research Pty Ltd

Tesch, R.(1990) Qualitative Research: Analysis Types and Software Tools, New York, Falmer.

Rik Hall
Progam Director, Distance Education and Off-Campus Services
University of New Brunswick
http://www.unb.ca
hall@unb.ca

Allan Ellis
Senior Lecturer, Director of Postgraduate Studies and Research
Southern Cross University
http://www.scu.edu.au
aellis@scu.edu.au

© 1998. The authors, Rik Hall and Allan Ellis, assign to the University of New Brunswick and other educational and non-profit institutions a non-exclusive license to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive license to the University of New Brunswick to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers, and for the document to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.