An Evaluation Report Written for the Leonardo Online Training Project http://www.nki.no/eeileo/
October 1997
Table of Contents
The Focus of this Evaluation
Research Methodology
Concluding Remarks
Literature
Appendix. Open answers and comments on the questionnaire
This report presents an evaluation study of experiences and attitudes within a group of students enrolled in NKI's Internet based courses or study programmes during one year from April 1st 1996 to March 31st 1997. The students have studied courses and programmes with large differences in course content, use of WWW resources, emphasis on group communication, and individual flexibility such as free starting time and individual pace of study. A questionnaire was mailed to all students registered as active students July 1st 1997.
The study was carried out as a pilot evaluation, related to an European Leonardo da Vinci project, concerning distance education based on the World Wide Web. The questionnaire focused on the students' background and their access to the Internet, the students' views on studying via the net including aspects such as communication, collaborative learning, use of WWW and online learning resources, and aspects related to distance study in general, such as individual flexibility. Furthermore, the study analyses technical difficulties, gives views and recommendations for further development of NKI Internet courses and specific advice concerning content and teaching of individual courses.
The majority of the students has access to Internet both at home and at work. A large majority of the students has their studies paid fully or partly by their employers. It seems clear that the fact that the studies are offered on the net is an important reason for enrolling. Few of the students would have enrolled for part-time or full-time study, while quite a large group indicate that correspondence study could have been a possible choice.
There is an overwhelming majority preferring studies with free starting time and un-paced progression; the preference is (with some few exceptions) independent of their experience as students. Concerning the learning value of different didactic aspects, the students emphasise the importance of aspects well-known from correspondence programmes, such as the quality of the study guide, reading and studying, working with assignments for submission, tutors' comments and individual flexibility, while specific aspects related to group communication seem to be less valued. Except for writing to the tutor, the students have taken little advantage of the possibility of taking part in academic and social conferences by either reading or writing.
Some courses in the programme have included material presented on the WWW. It did not seem that course material presented on the WWW was the aspect of study valued most highly. The availability of external resources including structure and links to these resources seems to be the aspect most appreciated. The students are neither specifically positive nor negative to the conferencing system used (Listserv distribution list). This might be a result of the relatively little use the students have made of the conferences.
The students have experienced some technical and communication related problems. As far as we can judge, these have generally not been very significant. Most students have coped rather well with the technical challenges of Internet studies. There have been sporadic problems, because of specific faults in software and administrative routines, which have been solved. The study shows clearly that the turn-around time is at least as important for Internet students as for correspondence students. Average turn-around time is approximately 3 days. Some students report that they have waited longer for their assignments than we would see as acceptable in an Internet based teaching/learning system.
It is specifically encouraging that a large majority of the students say that they would recommend Internet studies to other people.
NKI is one of the largest non-governmental educational institutions in Norway. NKI comprises The Polytechnic College, NKI Distance Education, the Business Training Centre (NA) and The NKI Publishing House. A further presentation of NKI is available at URL: http://www.nki.no
NKI Distance Education offers more than 60 distance education programmes and more than 300 courses, mainly in technical and vocational subjects. It has about 15,000 students. An extensive network of local partners around the country is established to offer face-to-face classes to support the distance students. The distance education tutors are part-time employees recruited from business, industry, schools and colleges around the country.
The College of Computer Science (now incorporated in the Polytechnic College) was established in 1983. In addition to its ordinary two- and three-year full time programs, it offers a two-year, part-time program in information processing, emphasising programming and system analysis. This part-time program is offered in three modes: face-to-face classes, correspondence courses, and the Electronic College
.The Electronic College is designed so that distance students can do all their communication online. In contrast, many of the internationally renowned implementations of 'electronic colleges' are actually using computer conferencing as a supplement to correspondence courses or on-campus teaching. The philosophy of the NKI Electronic College has always been to offer programs that are independent of time and space through a system that facilitates flexible communication for administrative, social, and teaching purposes.
Since the first courses were delivered in 1987, one may distinguish between three technological generations of the NKI Electronic College. The first generation system, which was based on the 'EKKO computer conferencing' system developed at NKI, was in operation from 1987 to 1993. The second generation, which was Internet based, started in January 1994. The first and second generation experiences are documented in an OECD report (Paulsen and Rekkedal 1997) and in several articles available at http://www.nki.no/ekko/for_alle/fagartikler/.
This report will focus on the third generation that was introduced with the first web-based courses in 1996.
The first generation of the NKI Electronic College proved its value through seven years of operation and 1067 delivered course packages. Of these 1067, 868 course packages have later been completed. The completion rate is hence 81.3%. Furthermore, some 45 of the 1st generation students have graduated from the Information Technology Programme. These graduates have completed a ten course programme - equivalent to one year's full-time study -via computer conferencing. In addition, the students have generally obtained better grades than either correspondence students or on-campus students (Paulsen 1992, 15). Surveys among the students show that they have a positive perception of the curriculum and the Electronic College as such. Student feedback also indicates that a teacher's way of handling communication is of crucial importance to students' perception of a course.
On January 1st, 1994, NKI introduced a new generation of the 'open electronic college' as a further development of the EKKO system. This second generation system had a philosophy amendment stating that it should be as open as possible to other networks and services. The second generation system was accordingly based on Internet, e-mail, and the Listserv conferencing system. Initially, the second generation user interface for modem users was text based. In January 1995, NKI started to offer modem users graphic Internet software developed for MS-Windows and the Serial Line Internet Protocol (SLIP). In 1995, the number of online courses was also increased to more than twenty.
The third generation was introduced with the first web-based courses in 1996. The web-service could be regarded as a two-level system. The top-level, the NKI Electronic College homepage http://www.nki.no/ekko/ as shown in figure 1, provides general information about the college such as course descriptions, prices, contract form, contact information, support information, and an article library on online and distance education. The graphical and user-friendly web-interface introduced opportunities such as hyper linking and multimedia presentations. However, there were also new challenges such as access control and copyright issues.
Figure 1. The NKI Electronic College homepage.
The second level, the course homepages, is passworded and can only be accessed by NKI employees and paying students. The course homepages are designed with a set of templates to secure some course conformity. A typical course homepage provides links to each of the study guide units, to the tutor's e-mail address, to the class discussion forum, to external Internet services, to a course evaluation form, and possibly to multiple choice assignments. The study units are also designed so that students can benefit from printing out the material.
Figure 2. A typical course homepage.
NKI has decided to develop course homepages for all of the courses delivered via the electronic college. By September 1997, all courses provided access to the college homepage as well as to computer-mediated communication such as e-mail and computer conferencing on the Internet. However, only 10 of the 30 courses offered had course homepages as shown in table 1.
Table 1. Programs and courses offered during the survey period.
| Programmes and courses | |||
| Information Network Programme | |||
| Internet | |||
| Data communication | |||
| Network administration | |||
| Multimedia network | |||
| Information Technology Programme | |||
| Introduction to information technology | |||
| Project management and development of information systems | |||
| Introduction to programming | |||
| Systems analysis | |||
| Object oriented programming | |||
| Project economy | |||
| Databases | |||
| Management of local area networks | |||
| Strategic use of information technology | |||
| Project work | |||
| Information Technology for Teachers programme | |||
| Internet for teachers | |||
| Information technology | |||
| Courses not included in a programme | |||
| Java-programming | |||
| Tutor in distance education |
The Information Network Programme is designed specially for distance education on the Internet. The text books are in print form, but the study guides are only available as web-material. E-mail for one-to-one communication and Listserv for many-to-many communication are the two main channels for communication. All courses have individual start-up times and unpaced progression. Exams are offered twice a year for each of the courses.
The Information Technology Programme is available as evening classes, correspondence courses, and courses on the Internet. Both textbooks and study guides are in print form. The courses have no course homepages, but the students have access to the college homepage and e-mail for one-to-one communication and Listserv for many-to-many communication. All courses follow a semester plan with start in September and February and have paced progression. Exams are offered twice a year for each of the courses.
The Information Technology for Teachers Programme is designed specially for distance education on the Internet. The text books are in print form, but the study guides and some of the readings are only available as web-material. E-mail for one-to-one communication and Listserv for many-to-many communication are the two main channels for communication. A pilot group of 21 students working as primary and secondary school teachers in the Oslo region took part in the programme. The courses followed a fixed semester plan, had paced progression and two scheduled face-to-face meetings. Internet access and support was provided by the Oslo School Authority and the pilot project was funded by the Ministry of Education.
This study is part of the continuous evaluation process of NKI distance education courses and programmes. The immediate background to the survey was, as described above, the participation in the EU project on developing a 'Multi Media WWW Kernel for Distance Education' and NKI's strategy on developing infrastructures, didactics and courses/programmes for teaching via the Internet.
From experiences during the last ten years, we have changed views and strategies concerning some important aspects of the delivery of courses on the net, e. g. based on some expectations concerning student backgrounds, access to equipment and answers to questions concerning attitudes and preferences of how, when and where to learn. Our strategies for future developments have partly been based on subjective experiences of personnel with varying responsibilities within the organisation; such as developers, academics, tutors, counsellors, IT support staff and administration, as well as on formal evaluation studies, discussions with other institutions, students and tutors. Thus, this study is an attempt to cover some issues which we felt needed more coherent and updated analysis before completing the structure and design of the kernel for distance learning in co-operation with our partners in the project and the didactic structure and design of courses to be developed in the near future. The issues are concerned with:
This evaluation study is seen as a 'pilot study' related to the EU project. In connection with the project similar co-operative evaluation studies of specific trial courses will be carried out. This survey was meant to supply information both for the development of the trial courses (specifically at NKI) and to contribute to the development and refinement of research questions within the project. Thus, time for both the development and distribution of the questionnaire and for receiving replies was restricted. However, with reference to the needs of the total project it was more important to collect these pilot data in time, rather than planning a more complete and ideal survey.
We decided to seek information from all students enrolled in one or more courses offered via the Internet during the period April 1st 1996 to March 31st 1997. The reason for the choice of time period was the following:
Because of a malfunction in the selection programme, only students who were active at the time of selection (July 1st 1997) were included. The mistake was not discovered before the data were processed and the report nearly completed. The difficulties in the selection process have at least the following consequences for the representativeness of the sample:
There is reason to expect that the two latter groups could represent 'extreme' views of being positive or negative concerning their study experiences.
We have over the years experienced that it becomes more and more difficult to receive acceptable response rates to postal questionnaire surveys. This seems to be a result of the fact that people in general receive so many questionnaires and these vary in quality and are of varying interest to them. Thus, we have found that telephone interviews often achieve better results - both quantitatively and qualitatively. Again, because of restrictions in time, we chose to construct a quite extensive questionnaire. We hoped that the considerable media and public interest in questions concerning the Internet would lead to a relatively high response rate.
We chose to construct a questionnaire including mainly fixed answers to tick. We aimed at constructing questions which we considered easy to answer. Some open-ended questions were also included. It seems clear that fixed answers produce information that does not go deeply under the surface of the problems that we wished to examine. On the other hand, we hoped that the format would motivate answering and give a higher response rate, and we would have other opportunities to approach the same or other students using other methods, e.g. telephone interviews and evaluation of specific courses or aspects of different courses at a later stage.
The questionnaire was distributed to 148 students during the first week of July 1997, regrettably not the best period for a survey, as most Norwegians start their summer holidays at just that time. An introductory letter explaining the aims of the survey and a free postage return envelope was enclosed with the questionnaire.
After 4 weeks we had received 43 completed questionnaires (and 2 returned with unknown address). We decided to send a reminder including a extra copy of the questionnaire. The reminder resulted in another 21 answers during the next 4 weeks. These 64 respondents are included in the analyses which follow.
Table 3. Questionnaires distributed and returned (N=146).
| Percent | |
| Returned without reminder | 29 |
| Returned after reminder | 14 |
| No response | 57 |
| Total | 100 |
We have found that the response rate of 43 percent is acceptable for our analyses in this survey as a pilot for following surveys - both in NKI and with our collaborators in the project. Some few forms have been received after the data were processed for computer analysis. We have no specific assumptions whether there are any systematic relationships between responding to this survey and attitudes and/or experiences related to the NKI Internet courses or programmes.
The students are enrolled either for one or more single courses or to a total study programme. They represent different courses and programmes and are studying under different conditions. We will try to account for different categories below:
Table 4. Courses and programmes represented in the study (N=56).
| Percent | |
| Information Networks courses | 18 |
| Information Technology courses | 13 |
| Tutor in Distance Education | 9 |
| Information Networks Programme | 11 |
| Information Technology Programme | 30 |
| Information Technology for Teachers | 20 |
| Total | 101 |
The programmes and courses have been described above. The 'Information Networks courses' cover single courses included in the 'Information Networks Programme' (½ year of full-time study) plus Java-programming. 'Information Technology Programme' covers courses on programming, systems analysis and computer technology. The course 'Tutor in Distance Education' (Norwegian version) is delivered free of charge to prospective tutors seeking employment within the NKI distance teaching system. 'Information technology for teachers', ½ year of full time study, is offered in co-operation with the Bodø College of higher education. The students in the survey all come from a trial group of Oslo teachers. One or two of the single course students take the Internet course as a special interest subject within the Technical College distance learning programme. All courses included in the study represent tertiary level studies.
In addition to types of courses and programmes they study, the students' background for answering will vary according to how long they have been active (or in fact for some inactive) in the system and will also depend on how many study units or courses they may have completed.
58 respondents have stated how long they have studied the course or programme in question. 8 respondents (14%) had started less than 4 months ago, 38 (66%) had studied for between 5 and 10 months, while 12 (21%) had been studying for more than 10 months.
62 gave information concerning completion of courses. 32 of these (52%) had yet not completed any courses at all, 11 (18%) state that they have completed one course, while 19 (31%) have completed two or more courses. The question concerning completion of study units (or formal assignments submitted) was answered by 48 students, of whom 11 (23%) had not completed any study units. Whether these students actually had started their studies by studying the material, communicating with their tutor or fellow students, or accessed WWW learning material or resources is not really clear. Some few state that they are not able to answer some of the questions because of lack of experience. 21 (44%) had completed 1 to 3 study units, 9 (19%) students had completed between 4 and 6 units, while 7 (15%) had completed 7 study units or more.
The courses included in the study comprise courses with learning material and learning resources presented on the WWW as well as courses based exclusively on printed material, courses based on printed material and/or computer software.
According to the respondents' answers, 23 (41%) study courses that do not include material presented on WWW, 25 (45%) study courses with WWW material, while 8 (14%) study both types of courses. The question is unanswered by 8 students.
Some courses are designed to encourage group communication more than others, e. g. by fixed starting points and paced progression during one semester (Information technology programme and Information technology for teachers).
There is internationally and nationally research data which illustrate that distance learning is not specifically chosen because students are living in remote areas, far from other part-time or full-time educational opportunities. A majority of distance learners choose their form of learning out of a need for flexibility and for adapting study requirements to demands from family, job or social life. What about Internet studies based on a rather new technology, does the new technology change this situation? Concerning adaptation of new technologies, are early use and interest for applying this technology for learning related to urban or rural living?
Table 5. The distribution of respondents by county in Norway (N=62).
| Percent | |
| Østfold | 3 |
| Akershus | 11 |
| Oslo | 34 |
| Oppland | 7 |
| Hedmark | 10 |
| Buskerud | - |
| Vestfold | 3 |
| Telemark | - |
| Aust-Agder | - |
| Vest-Agder | 3 |
| Rogaland | 7 |
| Hordaland | 3 |
| Sogn og Fjordane | - |
| Møre og Romsdal | - |
| Sør-Trøndelag | 2 |
| Nord-Trøndelag | 3 |
| Nordland | 7 |
| Troms | 5 |
| Finnmark | - |
| Abroad | 2 |
| Total | 100 |
The respondents represent 13 of 19 counties plus one student communicating from abroad. We have an overrepresentation from Oslo. To a certain extent this is a result of the group of teachers taking credit studies as part of a collaboration between the school authorities of Oslo and NKI.
Table 6. The distribution of respondents by type of domicile (N=64).
| Percent | |
| Large city (Oslo, Bergen, Trondheim, Stavanger) | 44 |
| City | 23 |
| Suburb | 11 |
| Rural areas | 22 |
| Total | 100 |
This categorisation is based on the students' own ticks of alternatives and are not related to official categories of the national statistics. Again the question was included to give some impression of whether the Internet students were recruited from rural or urban areas. Although we cannot refer this distribution to demographic statistics, it seems clear that the cities and suburban areas are over-represented among the NKI Internet students. One source of error in this conclusion is, of course, the group of 11 teachers recruited to the specific Oslo project.
We were interested in looking at types of occupations represented in the study, with specific reference to whether Internet studies are chosen by persons working in IT related occupations. 'Occupation' was included as an open question and the categorisation undertaken by the researchers before coding of the data. This kind of categorisation involves many difficulties and possible sources of errors, and the question could have been clearer concerning computer or IT related work. We also tried to differentiate between high and low level occupations defined by jobs normally requiring higher education or not.
Table 7. Occupation (N=59).
| Percent | |
| Teacher | 20 |
| High Level Administrative Not IT related | 24 |
| High Level Technical Not IT related | 9 |
| High Level Administrative IT Related | 5 |
| High Level Technical IT Related | 12 |
| Low Level Administrative Not IT Related | 10 |
| Low Level Technical Not IT Related | 9 |
| Low Level Administrative IT Related | - |
| Low Level Technical IT Related | 3 |
| Students | 3 |
| Unemployed | 3 |
| Retired | 2 |
| Total | 100 |
If we to some extent can trust the categorisation above, we may conclude that the majority (70%) have high level occupations (including the teachers), dominated by not IT related occupations (53%). Only 22% are categorised as working in low level jobs, and again only a couple of these are working in IT related jobs. Thus, it seems, with a reservation for possible errors, that most of the students do not actually work in IT related jobs. This does not at all mean that they do not need the competence gained in their present work - look e.g. on the pressure on school teachers to apply computers for the benefit of their students' learning.
Among the respondents there are 50 (79%) men and 13 (21%) women (one unanswered). 6 of the women were in the group of Oslo teachers studying Information technology for teachers.
The age distribution is given below:
Table 8. Age (N=62).
| Percent | |
| 20-24 years | 5 |
| 25-29 years | 21 |
| 30-34 years | 31 |
| 35-39 years | 16 |
| 40 years and over | 27 |
| Total | 100 |
The students were quite evenly distributed on the three levels of education specified in the questionnaire. As the study programmes are defined as tertiary level courses, the participants are generally expected to be qualified for entrance to university and colleges - with the exception that one or two students took part in the Internet course as part of their Technical College distance study, and in principle students are allowed to take a few of the single courses without examination even if they are not formally qualified for entrance to higher education.
Table 9. Previous level of education (N=63).
| Percent | |
| Secondary School 'A' Level | 32 |
| 1-3 years of Higher Education | 32 |
| 4 years or more of Higher Education | 37 |
| Total | 101 |
It is clear that the students in general have attained a high level of education before starting their Internet studies at NKI. In the early start of NKI CMC courses, we found that students choosing electronic communication had better qualifications than students choosing correspondence education where both alternatives were offered for similar courses (Rekkedal 1990). The educational background of the students in this study is very similar to that found among the CMC students 8 years ago, and clearly much higher than the average NKI student in other courses.
17 students (27%) state that they previously have taken correspondence courses at NKI; 6 (10%) have participated in correspondence education at other institutions. 1 student has previously taken 'electronic' distance education while 1 student has participated in 'distance education' at another institution. 37 (60%) have not been involved with any form of distance education before.
Different reasons for enrolling were presented in the questionnaire with possibilities for ticking one or more. In addition we have presented what we have called 'most important reason' for respondents who ticked only one reason (plus a combination of defined reasons stated).
Table 10. Stated reasons for enrolling in the course/programme.
| Percent | |
| (Percentages below based on 60 respondents) | |
| 1 Needs the competence in present job | 50 |
| 2 Increase formal competence | 52 |
| 3 To increase possibility of changing job | 32 |
| 4 Interest in the subject | 57 |
| 5 Interest in this form of teaching | 15 |
| 6 Other | 9 |
| (Percentages below based on 58 respondents) | |
| Most important reason: | |
| 1 Needs the competence in present job | 9 |
| 2 Increase formal competence | 13 |
| 3 Increase possibility of changing job | 8 |
| 4 Interest in the subject | 8 |
| 5 Interest in this form of teaching | - |
| Two or all 3 of reasons 1, 2, 3 | 13 |
| (4 and 5) | 5 |
| (1 and/or 2 and/or 3 + 4) | 30 |
| (1 and/or 2 and/or 3 + 5) | - |
| (1 and/or 2 and/or 3 + 4 and 5) | 9 |
| Other | 3 |
| Total | 99 |
Some comments on the table above on 'the most important reason': It seems clear that reasons related to needs in present job and formal competence and interest in the subject are dominating when deciding to enrol for distance education based on computer communication technologies. We should note that specific interest in 'this form of teaching' is ticked only by 9 students (15%) as relevant and never as the only reason. Among those who have ticked only one alternative, the most frequent is 'formal competence'.
Access to computers, modems and networks is one important requisite for opening up the markets for computer based distance education. During the ten years since 1987 when the NKI 'Electronic College' was launched, access has been considered one important barrier for recruitment, specifically to courses where the academic content or target groups have been related to computer science.
Table 10. Access to computers and networks (N=64).
| N | Percent | |
| PC (only) at home | 16 | 25 |
| PC plus modem at home | 41 | 64 |
| PC at work (no modem or network) | 2 | 3 |
| PC and modem at work | 16 | 25 |
| PC in network at work | 30 | 47 |
All students, who had no PC at home, had access to a PC either with modem or in a network at their workplace. Of those 16 who had a PC only without a modem at home, 13 had access to a modem or network at their workplace. This means that all the students had access to computers either at home or at their workplace, and that only three students had no modem or network access before enrolling. 21 students state that they had access to a PC and modem at home as well as a PC and modem and/or network access at their workplace.
E-mail address
50 students (78%) had an e-mail address before they started. Even so, 22 students (35%) state they have chosen to accept the offer of Internet access through NKI.
The use of computers at home or workplace for study
We also asked the students where they normally use the computer in connection with their studies. 26 (41%) answered that they used the computer for study only or mainly at home, 12 (19%) only or mainly at work, while 26 (41%) said that they used the computer for studies both at home and at work.
Who pays the fees?
Only 15 students (23%) say that they are paying their studies completely out of their own pocket. 36 students (56%) state that their studies are paid fully by their employer. Note that this applies to all the 20 teachers in the Oslo group. 9 (14%) students state that their studies are partly paid by themselves and their employer, while 4 students (6%) have their courses paid by the offices of employment or rehabilitation.
We have seen that interest in this method of studying did not seem to be decisive for the enrolment. In addition to the question concerning reasons for enrolling we included some questions on whether the Internet had any influence on the decision to enrol for the course or study programme in question and about probabilities of enrolling in other forms of study. The questions were designed as a 5-graded scale, where the extreme 'positive' counts 1, the middle alternative 3 and the extreme 'negative' 5.
Table 12. Did the fact that the course was offered over the Internet have any significance for your decision to enrol? (N=64) .
| Great significance | 1 | 2 | 3 | 4 | 5 | No significance |
| 45 | 23 | 11 | 3 | 17 |
M=2.2 Md=1.7
It is clear from the answers that for a large majority of students the fact that the course was offered on the Internet (either as e-mail and computer mediated communication and/or presented on the WWW) did influence their decision to enrol. It seems that on the 'negative' side few are indifferent to the question, but 11 (17%) say that this had no importance.
The following questions covered the probability of enrolling in an equivalent course if offered as: full time face-to-face, part time face-to-face or correspondence based study.
Table 13. Probability of enrolling in full time (N=60), part time (N=61) or correspondence study (N=61) (Percentages).
| Very probable | 2 | 3 | 4 | 5 | Not probable | M | Md | |
| Full time study | 3 | - | 7 | 7 | 83 | 4.7 | 4.9 | |
| Part time study | 10 | 13 | 16 | 15 | 46 | 3.7 | 4.2 | |
| Correspondence study | 25 | 16 | 20 | 7 | 33 | 3.0 | 3.1 |
It is clearly seen that very few of the students would consider enrolling for an equivalent full-time course. Some more students would look at a part time course as a possible choice. Concerning traditional correspondence study, a quite large proportion would consider this as a possible form of study. The answers seem to confirm that it is the flexibility of distance study that is the main argument for their choice of study form, however it seems clear (from table 8) that the Internet is a positive aspect when the decision to enrol is taken.
The questionnaire contained a number of 5-graded scales to assess the students' views on different aspects of teaching and learning. These are analysed below, and means and medians of views are calculated. The differences between these parameters give an impression of relative value judgements.
NKI holds the view that students generally prefer a large degree of flexibility (or that our main competitive strength lies in the ability to offer maximum flexibility). When we first started to offer courses based on computer mediated communication, we emphasised the aspect of group communication and social aspect of learning. As a result, computer conferencing courses were organised on a per semester basis with fixed starting dates and fixed schedules of progression aiming to facilitate academic and social communication between students. And - in fact - our early surveys showed that the 'computer conferencing students' were much more in favour of organising a fixed progression plan than were the 'correspondence students' studying the same courses (Rekkedal 1990). It should be noted, actually, that the Information Technology Programme and Information Technology for Teachers Programme are organised with fixed starting dates and common progression through the semester. According to the new strategy, from autumn 1997 all Internet courses are organised with the same flexibility as the correspondence based courses, a solution that may have advantages and disadvantages relative to the students' emphasis on group communication and possible preference for autonomy concerning time.
In this survey the questions were put as follows: NKI's distance teaching courses are largely based on flexibility in the form of free starting time and free progression. What is your opinion of this?
Table 14. Students' views on free starting time and free progression (Percentages, N=59).
| Advantage | 1 | 2 | 3 | 4 | 5 | Disadvantage | M | Md |
| Free starting time | 71 | 17 | 3 | 3 | 5 | 1.5 | 1.2 | |
| Free progression | 64 | 17 | 12 | 5 | 2 | 1.6 | 1.3 |
It is clearly shown that the respondents have a clear opinion that both free starting time and free progression do have more advantages than disadvantages. The preference for free starting time seems to be a little larger than preference for free progression plan. This is probably due to the fact that some students actually experience that they have a need for a push from the school or some strict requirements to be able to give priority to their studies. E.g. in an early survey some students answered "free pace of study" for both 'the largest advantage' and 'the largest disadvantage' of correspondence study (Rekkedal 1978). In the previous mentioned study one student wrote: "If electronic distance studies had been more flexible, it would have been a superb choice". And our conclusion on this point was: "...it is a major challenge to develop methods and organizations in distance education based on computer conferencing systems which take care of the distance student's need for autonomy and flexibility." (Rekkedal 1990, p. 92)
There are very small differences between students studying different types of courses and programmes concerning their views on freedom from time restrictions with the exception of the group of Oslo teachers showing a somewhat lower preference for both free starting time and free progression (a result which should be judged in relation to the fact that they started as a group and also took part in an introductory seminar). However, also in this group the majority prefers freedom of time measured by both questions.
Concerning the students' open comments the question on freedom of time is the most commented one. It seems clear that the majority prefer freedom independent of what they have experienced themselves. There are some few exceptions. Some of the teachers in the Oslo group obviously appreciate the fact that they have belonged to a group. Some comment that they need some pressure from specific demands on time limits for submitting assignments to keep a sufficient speed of study.
Table 10 gives an overview of different didactic elements and the students' attitudes towards their value for their learning as measured by the question: "The teaching contains different elements. Try to assess these concerning value for your learning"
Table 15. Didactic aspects of the teaching and their value for learning (Percentages, N=60).
| Very Important | 1 | 2 | 3 | 4 | 5 | Of little importance | M | Md |
| Content quality/relevance | 65 | 17 | 8 | 3 | 2 | 1.6 | 1.3 | |
| Quality of the study guide | 55 | 28 | 15 | - | 2 | 1.7 | 1.4 | |
| Quality of assignm. for subm. | 37 | 52 | 10 | 2 | - | 1.8 | 1.8 | |
| Reading/studying | 57 | 33 | 10 | - | - | 1.5 | 1.4 | |
| Work with assignm. for subm. | 50 | 42 | 8 | - | - | 1.6 | 1.5 | |
| Tutor's comments | 45 | 28 | 20 | 7 | - | 1.9 | 1.7 | |
| Academ. comm. with fellow st. | 17 | 17 | 30 | 27 | 10 | 3.0 | 3.1 | |
| Social comm. with fellow st. | 8 | 7 | 32 | 25 | 28 | 3.6 | 3.6 | |
| Individual comm. with tutor | 33 | 33 | 23 | 8 | 2 | 2.1 | 2.0 | |
| Academic conferences | 9 | 20 | 39 | 20 | 12 | 3.1 | 3.1 | |
| Social conferences | 3 | 12 | 25 | 27 | 32 | 3.7 | 3.8 | |
| Collaborative learning | 8 | 23 | 28 | 20 | 20 | 3.2 | 3.1 | |
| Individual flexibility | 55 | 23 | 15 | 5 | 2 | 1.8 | 1.4 |
Interpretation of the above information is not necessarily easy - as the students answer with reference to their actual experiences in the NKI Internet courses. This means that as NKI has chosen to design the Internet courses, whether based on WWW learning material or not, with great emphasis on flexibility, courses structured into study units with assignments for submission to be commented on by a 'personal' tutor, collaboration and communication with fellow students will receive less attention. Thus, in a way this can be seen as a 'self-fulfilling prophecy'. On the other hand, with some few exceptions there seem to be small differences between students with different experiences, e. g. between students studying programmes which have emphasised conferences and group based progression during the semester.
Generally, we may note that the students emphasise aspects known from individual distance education, such as quality of the study guide, individual reading/studying, work with assignments for submission, tutor's comments, individual communication with their tutor and individual flexibility as most important for their learning, while the specific aspects of computer mediated communication, such as social and academic communication with fellow students and social and academic conferences as well as collaborative learning seem to be regarded as less important. Concerning the first mentioned aspects, it is of course important to realise that these also can be more efficiently organised in distance learning systems applying the Internet and WWW for distribution of material and communication.
Concerning differences mentioned above between students studying different courses, students studying Information technology for teachers are more positive towards the learning effect of 'work with assignments for submission' and concerning the learning effect of 'individual flexibility'. (The last point is probably worth noting as just this group of students actually had less flexibility as they had to adjust to a fixed starting date and fixed progression schedule, however, with individual flexibility within these limits). These students together with students in the Information technology programme were more positive towards 'academic conferences' and 'collaborative learning'.
The electronic or virtual environment offered to the students may be used or taken advantage of in different ways. The use may be passive as when reading or active when writing to fellow students or tutors or contributing to conferences. Passive use may help the student in his or her individual learning process, but does not contribute to the social system of learning, to other students' learning or to the construction of a social climate. The students actual participation in the communication was also measured by a number of different variables.
Table 16. Actual use of resources/means for learning during the study programmes (Percentages, N=60/59).
| Very much | 1 | 2 | 3 | 4 | 5 | Very little | M | Md |
| Write to the tutor | 15 | 15 | 20 | 40 | 10 | 3.2 | 3.7 | |
| Write to fellow students | 2 | 2 | 12 | 37 | 48 | 4.3 | 4.4 | |
| Write to academ. conf. | 2 | 2 | 10 | 27 | 60 | 4.4 | 4.7 | |
| Write to social conferences | 2 | - | 12 | 15 | 72 | 4.6 | 4.8 | |
| Write to the administration | - | 8 | 17 | 32 | 43 | 4.1 | 4.3 | |
| Read in academic conf. | 5 | 12 | 13 | 27 | 43 | 3.9 | 4.3 | |
| Read in social conferences | - | 2 | 10 | 24 | 64 | 4.5 | 4.7 |
It seems very clear that the students make little use of the communication possibilities that are organised for them as part of the Internet/computer mediated communication courses. This fact might either indicate that these possibilities are of relatively little importance, that the actual use satisfies the needs they have for communication for efficiently completing their studies, or that the organisation or content of conferences is unsatisfactory. Writing to the tutor and to the administration and reading in academic conferences are the most frequent ways of using the communication possibilities.
We asked the students studying courses that did not include material presented on the WWW whether they would see it as an advantage to have the study material presented on the WWW. On a 5-graded scale from 'absolutely an advantage' to 'does not matter' half the students ticked the middle alternative with the rest distributed on both sides. In spite of the directions on the form, this question was also answered by many of the students taking WWW courses.
Table 17. Would you look upon presenting the learning material/study guide on the WWW as an advantage (Percentages).
| Absolutely an advantage | 1 | 2 | 3 | 4 | 5 | Does not matter | M | Md |
| Students in courses without WWW material (n=29) | 20 | 7 | 48 | 10 | 14 | 2.9 | 3.0 | |
| Students in courses with WWW material (n=17) | 29 | 12 | 47 | 6 | 6 | 2.5 | 2.7 | |
| All students (N=44*) | 22 | 11 | 46 | 9 | 11 | 2.8 | 2.9 |
*N less than the sum of the two groups because two respondents study both type of courses
Students who are studying courses involving WWW material are more positive towards presenting the material on the 'web' than are students of Internet courses not including this type of course presentation. However, the enthusiasm is not very large in any of the groups.
Table 18. Would you see it as an advantage if the course material was distributed in print (on paper) in addition to the WWW presentation (Percentages, N=33).
| Materials also on print | Absolutely no need | |||||
| an absolute advantage | 1 | 2 | 3 | 4 | 5 | for printed materials |
| 24 | 21 | 18 | 21 | 15 |
M=2.6 Md=2.6
We have internally discussed whether the students would prefer receiving the WWW material also in printed form. The question might have been misunderstood, as many of the NKI courses include WWW material as well as text books. Thus, some students may have understood the question as we actually were asking about their views on the courses as they are designed at present. Generally, we may conclude that views differ a lot between the students. Different views may result from different experiences and needs in different courses. This question should be further examined.
Table 19. What is your opinion of the following aspects of WWW courses based on your experiences? (Percentages, N=33).
| Very important | 1 | 2 | 3 | 4 | 5 | Of little importance | M | Md |
| Study material on the WWW | 39 | 18 | 21 | 18 | 3 | 2.3 | 2.1 | |
| Internal hyper-links | 33 | 30 | 18 | 12 | 6 | 2.3 | 2.1 | |
| Hyper-links to external resources | 47 | 9 | 31 | 9 | 3 | 2.1 | 1.8 |
Most students tick on the positive side for all three aspects of WWW materials. It seems that the use of hyper links to external resources is the facility that is most appreciated by the students.
Concerning open comments, some students specifically comment that study material should be on paper, while there are comments that the course is well suited for WWW applications and that external links are the most valuable aspect.
Table 20. The actual use of WWW facilities. (Percentages, N=33).
| Very important | 1 | 2 | 3 | 4 | 5 | Of little importance | M | Md |
| Use of internal hyper-links | 24 | 15 | 42 | 9 | 9 | 2.6 | 2.8 | |
| Use of external hyper-links | 24 | 21 | 24 | 15 | 15 | 2.8 | 2.7 | |
| Search in external resources | 27 | 18 | 36 | 6 | 12 | 2.6 | 2.6 |
Again, we may note that there are large differences in how much the students have actually used the WWW facilities. Differences in use may, of course, be related to how much of the course/programme the students have actually completed. The number of students is a little low to really examine such relationships. A surface look at these relations does not indicate that the explanation for the use or lack of use of WWW facilities lies in the amount of study work completed before answering the questionnaire.
Table 21. Assessment of the 'Listserv' conferencing system presently used. (Percentages, N=47).
| Very good | 1 | 2 | 3 | 4 | 5 | Very bad |
| 9 | 19 | 40 | 23 | 9 |
M=3.0 Md=3.0
It is very clear that the students do not express positive attitudes towards the conferencing system that they have experienced during their Internet studies. A majority has ticked the middle alternative. In our opinion there is little reason to be satisfied with this result. Normally, students seem to be quite positive in general towards their study experiences, and here there is an overweight on the negative side. Also with reference to the next question where the students were asked to indicate preferences between different choices, this is a result that deserves further examination. In most evaluation surveys of a similar nature, respondents tend to be generally negative to alternatives which they have not experienced. Here, the situation is quite opposite.
45 students answered the question on preferred conferencing system of the given alternatives:
The low number of answers is probably due to lack of experience by the other students. Still, of the three 'Listserv' is the least preferred conferencing system. Of the 45 students who have given priority to any of the three proposed conferencing systems, 18 (40%) prefer 'News', 16 (36%) say that they prefer a 'WWW based system', while only 9 (20%) state that they prefer 'Listserv', the system most used in their NKI Internet studies.
Table 22. Experience of technical/communication problems (Percentages, N=57).
| Very much | 1 | 2 | 3 | 4 | 5 | Very little | Not at all |
| 9 | 12 | 21 | 14 | 23 | 21 |
As far as we can judge from these answers, it seems that only around 20 percent of the students tick the negative side of the scale. 21 percent have not met problems, and 37 percent tick on the side of 'little' or 'very little'. If the respondents are representative, one should perhaps be reasonably satisfied with these results. On the other hand, at present we would expect that the students who have enrolled for Internet courses are more experienced computer users than the general public. For computer based distance education to succeed the technical problems have to be more or less eliminated.
Concerning those who say that they have had problems related to the technology, 16 students say that they have used the 'NKI support telephone', 24 say that they have 'solved the problems themselves', 11 have got 'support from others' and 12 ticked 'other ways' (answers not mutually exclusive).
When it comes to open answers on difficulties concerning technology and communication, it seems that they fall in into three categories:
We asked the students how long it had taken to receive answers back from their tutor. The question was answered by only 44 students (as a large group had not submitted any assignments). Turn-around time has been considered to be of great importance in determining the quality of correspondence education. Due to the fact that most distance teaching systems emphasise the two-way communication process between student and tutor, it is imperative that the time lag between submission and feed-back, questions and answers is kept to a minimum. We have in an early experiment shown that reduced turn-around time may increase completion rates in correspondence education (Rekkedal 1973, 1983). Although other researchers have questioned the generality of these results (e. g. Taylor et al. 1986, Field 1987, Diehl 189a, 1989b), it is generally seen as one of the advantages of computer mediated communication that the communication process can go much faster.
Table 23. Turn-around time (How long has it generally taken to receive answers back from the tutor? (days). (Percentages, N=44).
| Turn-around time in days | |||||||
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 days or more |
| 34 | 25 | 11 | 5 | 5 | 2 | 5 | 14 |
M=3.1 Md=2.1 (8days or more counted as 8)
Related to the above considerations, results showed in Table 23 is not satisfactory. Compared with our survey 8 years ago when 17 percent answered 1 day, 50 percent 2-3 days, 33 percent 4-7 days and no students reported more that 7 days, the respondents in this survey report indicate that the turn-around time has been longer. There is no reason for students to have to wait more than a week to get feed-back to their submitted papers.
Table 24. Satisfaction with turn-around time (Percentages, N=53).
| Very satisfied | 1 | 2 | 3 | 4 | 5 | Very dissatisfied |
| 28 | 34 | 23 | 11 | 4 |
M=2.3 Md=2.1
The students are generally satisfied with the turn-around time. There is naturally a high degree of correlation between experienced turn-around time and satisfaction (gamma = .68, p<.001). Those who have received their feedback within 1 to 3 days are generally positive, when the turn-around time exceeds 4 days, there is a tendency that the students tick the middle alternative, and the few students who have experienced a turn-around time of 6 days or more, tend to be negative. From the answers to this survey we would conclude that the students studying Internet courses at NKI seem to be satisfied with a turn-around time of up to three days.
The open comments clearly show that there at big differences concerning the different tutors' response times. Some students have received answers from their tutors very quickly and are very satisfied, while it is obvious that some students have received their answers back from the tutor after waiting much longer than they would expect.
Table 25. The student's attitudes to prices of courses and programmes (in relation to other private offers of education (Percentages, N=50).
| Expensive | 1 | 2 | 3 | 4 | 5 | Cheap |
| 4 | 16 | 58 | 16 | 6 |
M=3.0 Md=3.0
The students' views centre around the 'neutral' alternative. A tentative conclusion would be that the price structures chosen are acceptable. The answers might be difficult to interpret, however, as many of the respondents have their studies paid by the employer.
Table 26. Answers to the question 'Would you recommend studies on the Internet to other who were qualified and interested in the subject?' (Percentages, N=61).
| Yes, absolutely | 1 | 2 | 3 | 4 | 5 | No, absolutely not |
| 53 | 30 | 15 | 2 | 3 |
M=1.6 Md=1.4
The question is answered by all except 3 students. These results must be considered to be extremely satisfactory. Only 3 students (5%) tick the negative side of the scale. More than 80 percent would recommend Internet studies to others.
The questionnaire also included some open questions on the following aspects (answers recorded in the appendix):
The students mention primarily aspects related to the NKI distance study system in general, such as freedom and flexibility. Furthermore, it is specifically mentioned that it is positive that the study programmes are available on the Internet, the advantages of the WWW and links to resources, that the courses are interesting and updated, that the tutors are helpful, inspiring and give good comments, and that it is faster than e. g. correspondence study.
(See appendix) Comments concern specific aspects of the different courses, e. g. viewpoints of skipping Pascal and expanding on C++, That 'The Tutor in Distance Education' should be presented on the WWW (as it was from May 1997), that material should be on paper as well as on WWW, that tutors should give more feedback and help.
Presented on the Web, that the progression is too fast, and that some course content should be revised and updated.
It is suggested that the WWW should be used for what it is good at (meaning possibly links and resources, rather than presenting material) and that 'structure' should be emphasised more.
Some tutors should be quicker to return the assignments commented upon, when it takes some time, at least confirm the reception of the assignment. It is also mentioned that the quality of the tutors and their work vary considerably.
Few comments are given. One student mentions that NKI should take more responsibility for administering the exams (as students who wish to sit an examination locally have to make some arrangements themselves). It is also mentioned that the NKI study counsellor has been helpful, and that tutors should be easier to reach by phone.
This report has presented an evaluation study of a group of students enrolled in one of NKI's Internet based courses or study programmes in one year (April 1st 1996 to March 31st 1997). The students have studied courses and programmes with large differences in course content, use of WWW resources, emphasis on group communication, and on individual flexibility such as free starting time and individual pace of study. A questionnaire was mailed to all students registered as active students on July 1st 1997.
The study was carried out as a pilot evaluation related to an European Leonardo da Vinci project concerning developing distance education based on the World Wide Web. The questionnaire focused on the students' background (who choose Internet studies?) and their access to the Internet, the students' views on different aspects of studying via the net, such as communication, group work, the use of the WWW and the use of different learning resources, and aspects related to distance study in general, such as individual flexibility. Furthermore, the study analyses technical difficulties, views and recommendations for the further development of NKI Internet courses and specific advice concerning content and teaching of the individual courses.
The majority of the students has access to Internet both at home and at work. A large majority of the students has their studies paid fully or partly by their employers. It seems clear that the fact that the studies are offered on the net is an important reason for enrolling. Few of the students would have enrolled for part-time or full-time study, while quite a large group indicates that correspondence study could have been a possible choice.
There is an overwhelming majority preferring studies with free starting time and un-paced progression; the preference is (with some few exceptions) independent of their own experience as students. Concerning the learning value of different didactic aspects, the students stress the importance of aspects well-known from correspondence study, such as the quality of the study guide, reading and studying, working with assignments for submission, tutors' comments and individual flexibility, while specific aspects related to group communication seem to be less valued. With the exception of writing to the tutor, students have taken little advantage of the possibility of taking part in academic and social conferences by either reading or writing. Some courses in the study programmes have included material presented on the WWW. It did not seem that course material presented on the WWW was the aspect of study valued most highly. The availability of external resources including structure and links to these resources seems to be the aspect most appreciated. The students are neither specifically positive or negative to the conferencing system used (Listserv distribution list). This might be a result of the relatively small use the students have made of the conferences.
The students have experienced different technical and communications related problems. As far as we can judge, these have generally not been very significant. Most students have coped rather well with the technical challenges of Internet studies. There have been sporadic problems, because of specific faults in software and administrative routines, which have now been solved. The study clearly shows that the turn-around time is at least as important for Internet students as for correspondence students. Average turn-around time is approximately 3 days. Some students report that they have waited longer for their assignments than the time we would see as acceptable in an Internet based teaching/learning system.
It is specifically encouraging that a large majority of the students say that they would recommend Internet studies to other people.
The general impression from the evaluation study is that the students are reasonably satisfied with their experiences as Internet students with NKI Distance Education. It generally seems that the students value the aspects of study that have been emphasised in distance education systems, such as material for individual self instruction, two way communication with enthusiastic tutors supplying individual support and personal comments and guidance.
The students are not so concerned about social and academic group communication. The reason may be either that these aspects are not so necessary for learning, or that they have taken so little advantage of the possibilities that they really do not know, or that the possibilities and advantages are not exploited fully by NKI administration and tutors.
Concerning the use of the WWW we need to experiment more in different courses to find out how the Internet resources can be applied in better ways. Presenting material on the WWW is probably not the most important development. The big challenge is to use the WWW and its resources for stimulating meaningful activities including links to external resources based on a course presentation structure which makes the learning more interesting, efficient - and effective.
Future evaluation should focus more on how to use the Internet and WWW for didactic purposes with an emphasis on the aims, objectives and teaching methods of each specific course.
Diehl, G. E. (1989a): The effect of field scoring on time to completion in career development courses. Alabama: USAF Ext. Course Inst.
Diehl, G. E. (1989b): Comparison of two post-test feedback modalities on end of course performance. Alabama: USAF Ext. Course Inst.
Field, J. G. (1987): Report on proposed new TMA routing procedures. Milton Keynes: British Open University, EIT. (Internal paper)
Paulsen, M. F. and T. Rekkedal. 1996. Technology for Adult Learning in Norway Including a Case Study on the NKI Electronic College. In Adult Learning and Technology in OECD Countries.
Paulsen, M. F. 1992. From Bulletin Boards to Electronic Universities: Distance Education, Computer-mediated Communication, and Online Education. 1992. University Park, Pennsylvania: The American Centre for the Study of Distance Education.
Paulsen, M. F. & Rekkedal, T.(1990): The Electronic College. Selected Articles from the EKKO Project. Bekkestua: NKI/SEFU.
Rekkedal, T. (1973): Innsendingsoppgavene i brevundervisningen. Hvilken betydning har det å redusere omløpstiden? En eksperimentell undersøkelse. Oslo: NKI.
Rekkedal, T. (1983): The Written Assignments in Correspondence Education. Effects of reducing turn-around time. Dist. Ed.,4:2, pp. 231-252.
Taylor, J. C. et al. (1986): Student persistence in distance educatin: a cross-cultural, multi-institutional perspective. Dist. Ed., 7:1, pp. 68-91.
Rekkedal, T. (1978): Tekniske studier. Korrespondanseundervisning og klasseundervisning. Delrapport 2: Brevskoleelevene. Stabekk: NKI.
Rekkedal, T. (1990): Recruitment and Study Barriers in the Electronic College. In: Paulsen, M. F. & Rekkedal, T.: The Electronic College. Selected Articles from the EKKO Project. Bekkestua: NKI/SEFU.
Taylor, J. C. et al. (1986): Student persistence in distance educatin: a cross-cultural, multi-institutional perspective. Dist. Ed., 7:1, pp. 68-91.
Assessment of aspects of WWW:
Progression:
What can NKI do to increase quality of the conferencing system?
Technical/communication problems:
Specifically positive about NKI Internet/WWW courses:
Suggestions for improvements of the actual course:
Suggestions for improvements in the use of WWW:
Suggestions for improvements of the teaching:
Suggestions for improvements concerning administrative aspects: