This article presents some results from a number of research and evaluation studies on courses on the WWW. The research was carried out in three different organisations as contracted activities of a collaborative project partly financed through the EU Leonardo funds. The main aim of the project was to develop a "multi media WWW kernel for distance education" and to test the product on real courses under different conditions. The project partners were Ericsson Systems Expertise in Ireland, University of Rome 3 in Italy and NKI Distance Education in Norway. The empirical part of this article is a summary of results from different evaluation studies carried out during the project.

The main areas addressed by the evaluations were:

• The students’ attitudes towards WWW-based learning - What is the value of WWW courses as a form of educational or training provision measured by student views?
• Who are attracted to WWW courses? The students’ situation concerning use of computers and computer technology, i.e. what is the market for WWW courses and what kinds of students enrol?

Our focus in this article is specifically on the students’ views on the value, appeal and learning efficiency of courses on the WWW.

In 1968 Dubin and Taveggia from the University of Oregon published their "The teaching-learning paradox: a comparative analysis of college teaching methods". This research provides the foundations for the achievement of academic excellence in distance education programmes, whether electronic or not.

Dubin & Taveggia (Ibid.) stated that no particular method of college instruction is measurably to be preferred over another, when evaluated by student examination performances. Methods of college instruction analysed included lectures, seminars, instructional TV, correspondence courses or any combination of these and their findings can confidently be applied to the electronic distance education courses of today.

They examined data on 7.000.000 adult or near-adult students attending two- and four-year American universities to find the relations between the various methods of instruction and the outcomes produced when measured by final examinations on their courses.

Their conclusions were decisive: the method of instruction chosen did not affect student performance. They also concluded that replication of their work would not produce conclusions different from theirs. One can agree with them. So vast were their database and their meta-analyses that there is of little value in repeating them. Their conclusions have not been challenged since publication 30 years ago. This line of research is best regarded as closed.

Dubin & Taveggia (Ibid.) produced an educational "law"; the achievement of academic excellence does not depend on the method on instruction whether face-to-face or at a distance or on the web. The variables for student success or failure lie elsewhere.

A decade later confirmation for Dubin and Taveggia’s findings was provided by the publication of Schramm’s (1979) "Big media, little media : Tools and Technologies for Instruction". Schramm worked at the East/West Centre in Hawaii mainly in the comparison of lectures with textbooks, lectures with motion pictures, pictures with text, instructional (educational) television, teaching machines and computerised instruction.

He concluded that:

• No single medium is likely to have properties that make it best for all purposes
• Most instructional functions can be performed by most media
• Big media have no inherent advantages over little media
• Expensive media have no pedagogical advantage over cheaper media.

Thus the technique used is not per se a major element in the success or failure of a course of instruction. Bigger and more expensive media cannot per se contribute to successful learning any more than less expensive media.

Further evidence has been provided by Clark (1983) from the University of Southern California in the early 1980s. (See also Clark (1994).) He showed that media do not influence learning under any conditions and that "media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition".

According to Clark (1983:450) fifty years of research have shown that there are no learning benefits to be gained from employing different media in instruction, regardless of their obviously attractive features or advertised superiority.

Clark focused on the high expectation that people have from technology of all kinds, an expectation even higher today. Machine-based technologies, he says, have revolutionised industry and people have had understandable hopes that machine-based technologies would also benefit education.

At http://cuda.teleeducation.nb.ca/nosignificantdifference/ (visited 28/6/2000) one gets the ‘No significant difference phenomenon’ website on a TeleEducation, New Brunswick server.

This is a compilation of 355 research reports, analyses and papers compiled by Thomas L. Russell, until recently Director of the Office of Instructional Telecommunications of North Carolina State University, on the theme that there is no significant difference between face-to-face teaching and distance education.

This compilation has been built up on the web for a number of years as Russell collected and summarised on his site books, articles and statistical reports dealing with the theme.

It is an excellent archive of references to all major studies of the evaluation of academic achievement using any instructional methodology. The site has recently been revised and published in book form with a foreword by Clark.

For over 50 years studies have been conducted on students taught in face-to-face classes, seminars and lectures and on those taught at a distance. The almost invariable result of these studies is that there is no significant difference between distance education and on-campus provision.

The variables of successful learning, of the quality of learning, of the retention of learning, of the transfer of learning to the workplace lie elsewhere.

Dubin and Taveggia (1968), Schramm (1977), Clark (1983, 1994), Russell and others have proved that academic excellence can be achieved by distance education, whether in non-electronic systems or in virtual systems on the web. This is excellent news for those working in the provision of training at a distance or on the web or using some other non-traditional methodology: one does not have to be rich to learn; one does not have to enrol in an expensive system or use expensive technologies to learn; certificates, diplomas and degrees won at a distance can be as good as those won in training centres or universities; academic excellence is achievable at a distance.

It is best, however, to ask what happens in fact, in practice. Analysis is provided from the University of New England at Armidale in Australia, the Centre National d’Enseignement à Distance in France and the Open University of the United Kingdom at Milton Keynes.

One of the earlier attempts seriously to address the question of parity of achievement between distance education students and students attending universities was undertaken in 1979 by K. Smith, the then Director of External Studies at UNE. In the silver jubilee report on external studies at his university, Smith (1979) compared data on off-campus and on-campus students over 25 years from 1954 to 1979.

What made the study significant was that Smith’s university followed what was sometimes known as the ‘Australian integrated mode’ form of provision. This meant that professors and lecturers at the university had equal responsibility for teaching off-campus and on-campus students and, ceteris paribus, contractual responsibility for equal groupings of external and on-campus students. It also meant that the external students had the same curricula, the same textbooks, the same tutors, the same assignments, the same examinations and the same evaluators as those who were in residence at the university.

Many of the on-campus students were recent school leavers with excellent result in their HSCs; some of the external students were farmers on their properties who had not studied since primary school.

Smith was able to show not only that there was no significant difference between the external and the on-campus students, but that university medals for academic excellence in particular faculties or departments were won by external students.

At the time of writing the CNED is celebrating its 60^th anniversary and is by far Europe’s largest government training provider and largest distance education provider with 405.000 students enrolled: 200.000 at higher education level; nearly 200.000 at further education level and a considerable enrolment of children at primary and secondary level as well. It teaches totally globally with 40.000 students in 200 countries outside France in the year 2000.

A feature of the French educational system is that certification is characterised by government awards and examinations or competitions at every level from children’s schooling to post-graduate university degrees.

This may not be a system that appeals to some commentators from anglo-saxon backgrounds but it can work in favour of distance education. Over many years one can analyse the results of the students who studied with the CNED, and the students who went to schools, training centres and universities and show that parity of achievement can be realised.

The Open University of the United Kingdom lives in the highly competitive British higher education market: there are the universities, the Open University and perhaps 100 of the universities also competing for the distance education market. All of these are funded, at least in part, by taxpayers’ monies.

The most recent issue of The UCAS Guide, the government listing of higher education institutions issued to high school leavers to assist them in their application for university study, details 199 higher education providers, 24 of which are structures within the University of London.

In the mid-1990s the British government set about measuring the academic excellence of all the universities and publishing its results. Although some commentators from non-anglo-saxon countries look askance at the measurement of academic excellence, it is a system that can work in favour of distance education.

The first published results showed the Open University in the top 10 in the United Kingdom, with two tutorial universities (Oxford and Cambridge) and seven conventional universities.

To reach these rankings the UK government Quality Assurance Agency puts together panels each year to go and look at the quality of teaching in particular disciplines in each university that teaches them.

The teams have to assess the quality of teaching in the discipline in each university on six dimensions:

• Curriculum design, content and organisation;
• Teaching, learning and assessment;
• Student progression and achievement;
• Student support and guidance;
• Learning resources; and
• Quality assurance and enhancement.

The university’s teaching is rated on a scale of four on each of these six dimensions, so the maximum score a university can get for the teaching of a particular discipline is 24/24.

Among the subjects in which the Open University has been rated excellent are Chemistry, Geology, Music and Business, in addition to Engineering. These are all subjects where one would not think that a distance teaching university had a natural advantage.

It follows that over one hundred British universities and institutes of higher education have been rated as inferior to the Open University in quality. Although there is no direct relationship between the quality assessment results and student learning, there is no doubt that the Open University (and a great many other similar institutions world wide) has proved that the students’ results are at least at par with conventional institutions. When one realises that conventional universities use their full-time professors, readers, senior lecturers and lecturers to teach full-time students who have qualified with outstanding ‘A’ levels, while the Open University teaches students, nearly 40% of whom would not have academic qualifications acceptable to face-to-face universities, one can see the extent of the distance university’s achievement.

It can be considered as somewhat of a paradox that up to 100 of these face-to-face universities have chosen to challenge the Open University in the distance education market, often with small departments staffed by part-time consultants, when their full-time professorial staff have failed to match the academic excellence of the Open University with their ordinary students.

Both from research and practical evidence it seems clear that academic excellence can be achieved by distance systems, whether they teach by ‘traditional’ distance education or on the web. A number of evaluation and comparative studies at one of the institutions of this study, NKI, have also shown that distance students achieve as least as good as conventional students at exams in similar courses (non-significant differences in favour of the distance students) (Paulsen 1992).

The following data is collected from 7 different samples of students who had taken different courses on the WWW. The courses (form, content, level, duration) and institutional context and organisation were quite different.

The Ericsson first test course was an Intranet adaptation of one module of an existing face-to-face course called the Capability Maturity Model (CMM), which is a framework that demonstrates the key elements of an effective software process. The WWW version of the course typically takes one day to complete. The evaluation covered three runs of the course in 1997 and 1998, the two first were offered to Ericsson staff at two locations in Ireland and the third to a group of employees at an Ericsson design centre in Mexico. The three groupings comprised a sample of 16 students who all completed the questionnaire. The second Ericsson test course, Data Communications (DC) was administered to 12 staff members and trainees early 1999. The context of course administration was similar to the first course.

Three different University Rome 3 evaluation studies were carried out:

A distance course for schoolteachers in Museum Didactics. The students who took the course were Italian schoolteachers specialising in applying museums for teaching during spring 1998. The course has been a popular distance teaching course for some years. The course typically takes one year to complete. In includes a number of assignments for submission, which are automatically assessed and commented on by a computer. The students in the test group enrolled to take the course on the WWW including assignments for submission answered on the WWW with automated assessment and feedback.

The two additional University of Rome 3 courses were one Informatics Engineering course and one Electronic Engineering course. Both courses were courses organised as ordinary face-to-face courses at the university through a period of four months autumn 1998 and spring 1999, with WWW learning material as auxiliary support. The majority of the students spent most of their time on the WWW material from computers outside the university, mainly trough modems from home.

Two different evaluation studies were carried out at NKI, the first covered students who had completed one of four different WWW courses during 1997. The second study sampled all students who had been registered in at least one course in a specialisation programme in Information Technology autumn 1998. All the courses covered by the two evaluations were distance learning courses on college level, each course equivalent to approximately 1/10 of a full-time year of studies. The main difference between the two evaluations was that the first included only completers, while the second included all registered students. The first study applied a postal questionnaire, while the second applied a questionnaire to be answered on the WWW and returned as e-mail. In the NKI case we also had the possibility of comparing examination results between the part-time distance students and another sample of part-time face-to-face students taking the same courses with identical exams.

Typically, the Ericsson students were internal technical staff taking their course as in-service training. The majority were young males, nearly half being below 25 years old. Very few were aged over 30. The company paid the course. The Rome 3 Museum Course students consisted mainly of female students, very few below 25. The large majority paid for the course themselves. The students who took the two Rome 3 Engineering courses were mainly young men below 25 years of age. Concerning the NKI surveys, the first one included Information Technology for Teachers in addition to the Information Technology courses of the second survey. Thus, the first survey sampled approximately 50 percent men and 50 percent women, while in the second there 60 percent men and 40 percent women. 60 percent of the NKI students pay their courses themselves.

The majority of the students in the courses examined have completed at least 4 years of post secondary education, with an exception for the two Rome 3 engineering courses, which were introductory level courses for ordinary students.

Although some questions could be more relevant for one partner than for another, in constructing the questionnaires we emphasised that the evaluations conducted by the three project partners should have a large degree of similarity to make the ground for possible comparisons. A few of the questions were adapted from a previous survey, ‘Evaluation virtuelle Seminare’ carried out by Fritsch (1997) at the FernUniversität.

The questions also included information on speed of access to the course pages, on how much time the students generally spent "at a computer screen" and knowledge of HTML. It seemed clear that the student groups differed concerning their use of computers. For instance, all the Ericsson CMM students answered that they spent more than 10 hours and 75 percent over 20 hours in front of a computer screen during the last week. Among the NKI students these figures were 70 and 40 percent respectively and among the Italian Museum course students the figures were 40 and 17 percent. It is not surprising that the Ericsson students, who took their courses on an Intranet, found that access time to reach the course pages was "very quick" or "quick". However, it is interesting to note that among the WWW distance students at NKI more than 80 percent answered "very quick" or "quick". It seems that access at the time when these surveys were carried out were slower in Italy, as only 40 percent of the Museum course students gave answers in these categories, while 55 percent answered "slow" and 5 percent "very slow". Similar negative experiences were reported by the Italian engineering students. It is reason to believe that access time does influence how students value learning on the WWW.

One important aim of the evaluations in this research was to examine whether offering the courses on the WWW facilitated enrolment, whether the WWW was seen as an efficient way of learning and whether their general experiences had been positive, measured by possible interest in enrolling in another course themselves or if they would consider recommending others to take courses offered on the WWW. To approach these problems from different angles we asked six different questions measured by a 5-graded ‘Likert scale’. The results are presented in the following tables as percentages of respondents who have ticked the alternative and the median (Md) of the answers, where the extreme positive end of the scale counts 1 and the negative end counts 5 (Note that in Table 2 were the statement is put in the negative form the values of the scale are changed.) Response rates varied between the different evaluation studies: (Ericsson studies: 100%), (Rome 3, Informatics: appr. 75%), (Rome 3 Electronics: appr. 50%), (NKI 1997 Survey: 61%).

Courses can be presented in many ways, such as full-time or part-time classes, distance education based on printed material and other types of pre-produced material, correspondence education or e-mail communication with or without conferencing facilities.

From the information presented in table 1 we can clearly conclude that the students in all surveys agreed and agreed strongly that efficient learning can result from distance courses based on the WWW. Of all the students in all 6 surveys, only 8 of the 286 students answered the question negatively.

As mentioned the NKI surveys comprised students who study completely at a distance on the Internet/WWW. The exception was 3 students in the 1997 survey, who actually were studying full time at the NKI Polytechnic and took one elective subject (JAVA course) at as registered distance students. It seemed that these students were somewhat negative to the distance learning experience. These three students were actually among the 4 students who did not answer positively. The reason might be that these full time face-to-face students were actually not prepared for the demands and learning styles necessary to succeed in distance study.

As this question was given in a negative form, the value assigned to the alternatives when calculating the median is opposite to the other questions. In all surveys some very few students agree to the statement that learning was damaged because the course was presented and taught on the WWW. Some of the museum students obviously have met some difficulties. It is reason to believe that the difficulties have resulted from access time and communication difficulties from modems in Italy at the time.

It is perhaps not surprising that a large number of the students find it difficult to make any comparisons between an assessment of learning on the www with a hypothetical experience of learning the same material face-to-face – as their experience in the course concerns learning on the www. Thus, generally the answers seem to cluster in the middle category, "uncertain". It is not surprising either that they are reticent about the question as they hesitate to forecast the result of their study – as they have not received any test results when answering the questionnaires. It could be noted, though, that except for the to Italian student groups who used the www learning material as auxiliary support, it seems to be more students who agree than disagree with the statement.

From the responses to the three statements above in the seven evaluation studies we may conclude with quite some certainty that the students find learning on the WWW as an efficient form of learning. It might be of interest to note the differences between the NKI 1998 survey, which included all registered students, and the 1997 survey including only completers. This difference does not seem to have influenced the results significantly. We would expect that the 1998 survey included some more students who had not found WWW study satisfactory, as many of them (41%) had not completed any courses.

As the students in the different samples took their courses under different conditions, comparisons are difficult. E.g. for some of the Ericsson students the course was a compulsory induction exercise which makes it easy to understand a degree of uncertainness about whether or not the offering of the course on the WWW facilitated access. It is worth noting that the "real distance students" of the Rome 3 Museum Course and the NKI students agree strongly that the fact that the course was offered on the www facilitated access. These figures might be compared with results from a previous evaluation study at NKI, where Rekkedal and Paulsen (1997) asked distance students taking courses on the Internet whether they would have considered enrolling if the course was offered as "correspondence study" (41% positive), part-time face-to-face (23% positive) or full-time face-to-face (3% positive). In our studies here it might be difficult to say what form of study the students actually do compare with. However, specifically it seems that the distance students have the opinion that their enrolment was facilitated because the course was on the WWW. The Rekkedal and Paulsen (1997) report indicates that the Internett/WWW opens the market for new groups of students as less than half said that they would consider enrolling for a "coorespondence course" (In fact, the NKI programme was at the time offered both as "correspondence based" and "Internett/WWW based" distance study.

Attitudes can be measured in many ways. We decided that the answer to whether the student would consider participating in the activity again or recommend the activity to another person, friend or colleague, would be reasonably valid indicators of how the activity or experience is valued. (See e.g. Boshier et al. (1997) who used the variable "Recommend this course to friends interested in the subject" as a main indicator of the researchers’ subjective quality assessment of WWW courses.)

Again the conditions and frames of reference differ. E.g. it is probable that both Ericsson and Rome 3 students do have a feeling of taking part in some experimental situation, while the NKI context is more characterised by students studying courses offered as an ordinary part of the NKI portfolio of distance teaching programmes. The results can be characterised as remarkable as the challenge of enrolling again in an experimental medium great – one can do a non-traditional course as a one-off experiment or a novelty, but to enrol again may indicate that the validity of the medium has already been accepted. Concerning the possibility of enrolling again in a WWW based course and overwhelming majority in all the six evaluations is very positive. Specifically, it could be noted than there is not a single one of the distance students in the Rome 3 Museum Course or in the NKI 1998 Survey who answers negatively to the statement.

The attitudes expressed above are largely supported by the similarly positive responses to the statement of encouraging a friend or colleague to enrol in a WWW based course.

The figures presented in table 6 are quite promising for the future market and the possibility of attracting new groups of students to learning on the WWW. A great majority of the students agree to the statement of encouraging a colleague or friend to enrol in a course presented and taught on the WWW. It is perhaps specifically interesting to note that the "real distance students" (Rome 3 Museum Course students and NKI students) gave the most positive replies.

The evaluations presented here based on students studying on the Leonardo MMWWWK kernel in different contexts and under different conditions are highly positive. The majority of the respondents in all six groups agree that:

• It is possible to achieve excellence in courses that are taught on the WWW
• Their enrolment was made easier because the course was offered on the WWW
• They would enrol again in a WWW based course
• They would encourage a colleague or friend to enrol in a WWW based course
• The students generally disagree that their learning was damaged because the course was on the WWW

What about learning results? We have in this article clearly expressed our view that that form, method or medium not in themselves influence the quality of teaching or learning significantly. Still, we would like to present some statistics from one sub study of the evaluations above on examination results among the NKI 1998 students.

The students in the 1998 NKI survey were registered in one or more distance education courses in the Information Technology Programme. This programme is also offered face-to-face to part time students and the students take the same exams at the same time. Autumn 1998 we collected exam results achieved by the two groups of students presented in table 7.

The table above shows that 7 identical exams have been administered to face-to-face students and to the WWW students. The total average pass rate was 84 and 87, respectively. The average grades also are also somewhat better among the WWW distance students (the lower grades are the better, best possible grade 1.0, over 4.0 means fail).

The achievement of training excellence is a challenging goal. The NKI statistics clearly demonstrate that the WWW students achieve at least as good as the face-to-face students taking the same courses. According to the respondents in the evaluation studies presented here the courses offered on the WWW have been well accepted. It should be noted that at least in the Ericsson study, the Rome 3 Museum Course study and in the NKI surveys, the respondents are comprised of very experienced students, the majority having at least 4 years of university experience. In the Ericsson context some of them were, themselves, trainers.

Boshier, R., Mohapi, M., Moulton, G., Qayyum, A., Sadownik, L. & Wilson, M. (1997): Best and worst dressed web courses: Strutting into the 21^st century in comfort and style. Dist. Ed. 1997, 2, pp. 327-349. Toowoomba, University of Southern Queensland.

Carli, M. (1998): Analisi del Questionario di Valutazione per il Progetto Leonardo. http://www.nki.no/eeileo/research/italian4may99.html

Clark, R.E. (1983). "Reconsidering Research on Learning from Media." Review of Educational Research, volume 53, number 4, p. 445.

Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42, 21-29.

Dubin, R. & Taveggia, T. C. (1968). The Teaching-Learning Paradox. Eugene, Oregon: Center for the Advanced Study of Education Administration. ERIC Documant ED 026 966 (1968)

Fritsch, H. (1997): Host Contacted, Waiting for Reply. Hagen: FernUniversität. http://www.fernuni-hagen.de/ZIFF/EVIRTXT.htm

Paulsen, M. F. (1992): The NKI Electronic College: Five years of computer conferencing in distance education. In: Paulsen, M. F.: From Bulletin Boards to Electronic Universities: Distance Education, Computer-Mediated Communication, and Online Education. Pp. 2-17. University Park, Pennsylvania: The American Center for the Study of Distance Education.

Rekkedal, T. & Paulsen, M. F. (1997): The Third Generation NKI Electronic College. A Survey of Student Experiences and Attitudes. An Evaluation Report Written for The Leonardo Online Training Project. http://www.nettskolen.com/alle/forskning/33/evaluati.htm

Rekkedal, T. (1999): Courses on the WWW – Student Experiences and Attitudes Towards WWW Courses – II. http://www.nki.no/eeileo/research/Rekkedalcorrected.html

Russel, Th. L. (2000): ‘No significant difference phenomenon’ website. http://cuda.teleeducation.nb.ca/nosignificantdifference/

Keegan, D. (1998): Ericsson students studying on the WWW. http://www.nki.no/eeileo/research/eei/Brusseev.htm

Keegan, D. (1998): Ericsson students studying on the WWW – Part II

http://www.nki.no/eeileo/research/eei/Brusseev2.htm

Nardi, E. (1998): Analysis of Università di Roma III students' attitudes to studying using the Leonardo MMWWW kernel for distance learning. http://www.nki.no/eeileo/research/roma/emmaev.htm

Schramm, W. 1979. Big Media, Little Media: Tools and Technologies for Instruction. Beverly Hills, Calif.: Sage Publications.

Smith, K. (1979): Extermal Studies at New England: A Silver Jubilee Report. Armidale: University of New England