Article in Harish Rathore & Rudolf Schuemer (eds): Evaluation Concepts and Practice in Selected Distance Education Institutions. ZIFF Papiere 108, Hagen: FernUniversität, 1998
This article aims at giving an overview of ideas, principles and practices concerning quality assessment and evaluation at NKI Distance Education. The main emphasis will be placed on system evaluation - in both micro and macro perspective. The article discusses concepts and practices related to quality assessment and evaluation. NKI practices concerning student evaluation are also accounted for as one aspect of internal evaluation. It has been the intention to describe and discuss NKI practices and experiences, rather than writing a scientific article. The emphasis has thus been put on concepts, methods and experiences, not on presenting specific results from research and evaluation projects. This article represents an update of the contribution to the report on evaluation practices and experiences published by the FernUniversität (Schuemer 1991, Rekkedal 1991). The Norwegian laws and regulations concerning governmental accreditation and control of distance teaching institutions was drastically changed in 1993. Some important aspects of this change are also discussed.
NKI holds the view that evaluation has to be an integral part and continuous activity in any educational undertaking. Without evaluation and assessment of students during the learning process, conditions for learning are far from ideal. In the same way evaluation of the organisation, the teaching/learning process, the learning material and media for learning is necessary for institutional learning and increasing quality.
25 years ago, as a graduate student at the University of Oslo, the author of this article was planning the research project for his degree in education. He was interested in adult education and in correspondence education in particular. Until then very little research had been carried out within this area, and the author's intention was to make a large scale pilot survey on recruitment, study achievements and completions/drop-out in correspondence study in Norway. More or less by incidence he came in contact with NKI and asked if he would be allowed to carry out the research based on NKI files of correspondence students. He was welcomed and given completely free hands in collecting the amounts and types of information he wished. A year later his thesis was completed.
The information presented did not at all give a glorious picture of the results of the institution. Although the main focus of the research was to find relations between background factors, personal factors and situational factors with different criteria of success in correspondence study, one could not oversee the fact that the completion rates presented were low. The results were, in fact, quite disappointing, and one could guess that many administrators would have taken the decision to try to keep such information confidential.
The rest of the story, in our view, gives a picture of NKI's views on institutional evaluation at that time. Instead of being asked to keep a low profile on publication of results, the administration decided that the full report should be distributed to all individuals and organisations involved in or specifically interested in correspondence education in Norway. An abbreviated report containing all factual information was published in Norwegian and English and distributed nationally and internationally. The author was present once when the director was asked how he dared to publish the results of the drop-out part of the study, and the answer was that he did not dare not to publish, as openness of institutional results is necessary for increasing quality. The immediate reaction at the institution was that all employed - academics, administrators and clerical staff - became involved in a project aiming at assessing the organisation and administrative, teaching and counselling routines and procedures to state long-term and short-term goals for changes to increase the quality of teaching material and student services. At the same time the author was engaged to plan and conduct a research and evaluation scheme with the same goal - quality control and quality development of the NKI distance education system. Since then, research and evaluation to assure and increase the quality of the teaching system have been focused on as an integral activity in the development and administration of all distance study programmes.
NKI is the second largest non-governmental educational institution in Norway. It is organised as a non-profit foundation with governmental control and financial support from the state. Today, the organisation has four different departments offering services to the public:
The Polytechnic College was established in 1995 by a merger of the two NKI Colleges, the College of Engineering and the College of Computer Science. The Polytechnic College offers two and three years programmes in computer science and engineering. In contrast to what is happening in many academic institutions today, where one moves from traditional academic studies into distance education and new modes of teaching, the NKI College of Engineering was originally developed from an existing distance education programme. The second of NKI units is Næringsakademiet which offers vocationally oriented studies at secondary level as part-time and full-time programmes in several towns and cities. All the studies emphasize practical applications of computers. NKI Publishing House develops materials for both NKI activities and for sale on the open market.
NKI Distance Education is the second largest distance teaching institute in Norway recruiting 7.000 to 10.000 students a year, or approximately 35.000 course enrolments. The programmes offered cover technical/vocational training, management, computer science and business training on secondary and tertiary level.
To support the activities NKI has established a Research and Development Unit responsible for research in education and evaluation. The unit has given priority to research and development activities within distance education and was integrated into NKI Distance Education in 1993.
Evaluation is part of most human activities involving decisions and simply means assessing the quality of something. We evaluate when we buy a suit or even a hamburger, when we decide to go to the theatre instead of the cinema, and when we decide which play to see. After the action we assess again to see whether we are satisfied or would make another choice at another time.
At the general level NKI follows the ideas and principles of educational technology in developing and delivering education. In general models of education and teaching, evaluation always plays an important part. Evaluation is the starting point for the feedback loop in the process. At every stage of development the results are in some ways formally or informally, subjectively or objectively assessed and necessary action taken. Ideally, but not as often as we would wish, a total education programme is offered in a pilot version for formal evaluation and revision before the programme is offered on a larger scale. The usual procedure in these cases is to market a new programme in a pilot version which is tried out on a restricted group of students before the final version is put on the market. Later, as long as the programme is offered, successive evaluations and revisions are carried out and the programme is incrementally improved and updated over time.
According to the theories of educational technology, evaluation ought to be a rather objective process, where learning results measured by some clearly defined performance standards are compared with the defined learning objectives (see e.g. Popham 1981). Our view is that this might be an ideal situation, at least in some cases. However, in practice, in NKI, and we believe in most other distance education institutions, the objectives defined during development of the course are seldom expressed in formal terms suitable for objective control in a strictly goal-related evaluation. Thus, the evaluation procedures tend to become more subjective than the ideal of the systems approach would prescribe. This seems to be a common observation. While criterion-referenced tests including performance standards originally were presented as a promising tool for securing accountability in education and thus constitute a simple and inexpensive tool for decision-makers, in practice over the years, the ideal theory has not lived up to expectations. For a discussion of performance standards and criteria, see Burton (1977) and Glass (1977).
As educational institutions in general have been influenced, often through governmental intervention, to adopt concepts and systems of quality developed in other sectors of society, and as these systems have been introduced in connection with the recent changes related to Norwegian distance education we shall look shortly at some of these concepts as well.
Qualityis most often defined 'fitness for purpose' related to the needs of the user/customer (Juran 1988), which indicates that quality depends upon a subject's view of what is the purpose of that phenomenon. In education the customer is not easily identified. The government pays, the immediate user is the student, secondary users are employers etc. Quality, thus, is a value judgement interpreted by different stakeholders, government, teachers, administrators, students, employers etc. On the other hand, to assure and assess quality we must have a clear notion of what it is. Another definition could be that the 'product comply with defined requirements'. Consequently, purpose and requirements, then, should be defined by the significant stakeholders. Birnbaum (1989) has stressed this diversity and pointed out three dimensions of quality in higher education: the meritocratic (the institution's conformity to professional and scholarly norms with the academic profession as reference group), the social (the degree to which the institutions satisfies the needs of important collective constituents) and the individualistic (the contribution the institution makes to the personal growth of students (from Van Vucht & Westerheijden 1993).
Some other important concepts are 'quality control', quality assurance', 'quality management' and 'quality assessment'. Quality control is defined in technical environments as: 'the operational techniques and activities that are used to fulfil the requirements for quality' (ISO 8402). Van Vucht & Westerheijden (ibid.) add that concerning higher education the term also includes the state control strategy concerning quality in higher education. Quality assurance is 'all those planned and systematic actions necessary to provide adequate confidence...' Quality management is defined as 'that aspect of the overall management function that determines and implements the quality policy'. The ISO document defines 'quality audit', while Van Vucht and Westerheijden (ibid.) recommend the term 'quality assessment' as a better term in the field of higher education and describe it as 'a systematic examination to determine whether quality activities comply with planned arrangements and whether the product (the educational process) is implemented effectively and is suitable for achieving objectives' (Ibid. p. 12).
There has been a lot of discussion in the educational community on who should carry out evaluation to guarantee maximum objectivity and usefulness of the results, see e.g. Scriven (1975). In our view, the question is more acute in situations where external bodies have the responsibility to accredit institutions and/or programmes according to certain criteria, and in situations where granting bodies wish to control the effects of money granted to certain programmes. According to Scriven (1975) there is no easy way around the question of objectivity. There is no way of guaranteeing that external evaluators are not biased, and there is a good chance that external evaluators are not able to go sufficiently into the institution and programme to come up with a useful evaluation design and get hold of the information of real importance for the evaluation.
There are many reasons why evaluation and other measures to control and improve quality have been so important and have been taken seriously by distance education institutions:
Thus, a tradition of research and evaluation has emerged in both public and private distance teaching institutions, and this is often linked with systematic efforts towards improving quality and methods. The links between research and practical implementation are usually close, which also makes it easier to promote a 'practitioner-based' type of evaluation. For instance, the British Open University, had from the start a specific department responsible for research and evaluation. This example has been followed by many of the distance education universities established later in Europe and other parts of the world (see e.g. Schuemer 1991).
During the last years there has been an increasing emphasis that those directly involved and responsible should play an important role in the evaluation process. The idea is mainly that if the evaluation really is to have a positive impact on the improvement of the quality of the educational system, the practitioners must take part in the evaluation process (Thorpe 1988)
As this article will illustrate, in NKI there are examples that evaluation may be undertaken by a member of the programme staff, by other member(s) of the responsible unit in the organisation, by another unit within the organisation or by external evaluators contracted by the teaching institution itself or by some external controlling or funding body. However, there is no doubt that the emphasis is on internal evaluation as the main measure for quality improvement.
Norway was the first country in the world to regulate distance education by a specific legislation, the Act of 12 November 1948 concerning Correspondence Schools. According to the Act all correspondence schools and every single course must be accredited by the Ministry of Education before it can be put on the market.
From 1949 until 1992 this evaluation was administered and carried out by a controlling body, The Correspondence Schools Council. The legislation was changed from 1993 when distance education became regulated as part of the Act on Adult Education. We will come back to this change later - as it has had considerable impact on external control and quality assurance in distance education in Norway.
By regular intervals the Council was supposed to conduct an intensive and inclusive evaluation of all distance teaching institutions. Thus, when subject to these evaluations, the total NKI distance teaching organisation and activities were examined. The reports were based on internal reports from NKI, meetings with key personnel and external analysis by the Correspondence School Council's representatives. The final reports from these evaluations dealt with marketing and economic systems, tutoring, support and counselling, qualifications for full and part-time employees, the quality of individual courses and study programmes that are offered, turn-around time of assignments, experimental activities and administrative routines, i.e. all the factors that were seen to affect the total quality of the educational programme offered.
The institution gained quite some new and valuable insights through these intensive evaluation procedures.
Traditionally, the Correspondence Course Council was not occupied with evaluation of the total study programmes or composition of courses. The evaluation concentrated on assessing the whole institution and the individual single courses.
Norway also has a nation-wide system of recognition of college and university programmes and exams. In this connection NKI programmes on tertiary level have been externally evaluated. During this process, curricula, textbooks, exams, teachers and examiners are evaluated according to certain criteria defined by the Norwegian Universities.
The Correspondence Course Council was also responsible for evaluating each separate course. To carry out the evaluation of the courses, the Council appointed external subject experts. In our view, these procedures may have been justified at the time when they were initiated. However, in recent years the rigid system of external control was both by the Ministry and the institutions seen to have more weaknesses than strengths. Some of these are also discussed by Holmberg (1989).
Below we will try to account for the changes in legislation and external control regulations introduced to Norwegian distance teaching institutions in 1993. Although NKI always has seen its own internal evaluation procedures to be the backbone of quality work, there is no doubt that legislation with regard to external accreditation and control schemes do influence internal procedures.
There is hardly any doubt that the Norwegian Act concerning Correspondence Schools, the work of the Correspondence School Council and the Ministry's accreditation of correspondence courses and correspondence schools during the years has had a positive impact on the development of correspondence education in Norway. It would also be correct to argue that there are few examples of institutions that have operated with insufficient quality or doubtful business practices during the period in which the Act concerning Correspondence Schools has been in effect in Norway.
The accreditation scheme ensured that each individual course was evaluated by independent consultants before it could be put on the market. As a result the schools established routines for quality assurance through the choice of authors, the editing work and in some cases the use of independent consultants before the courses were submitted to the Correspondence School Council for evaluation. Naturally the schools wanted to avoid the delays and negative rating associated with not having prepared their material sufficiently before the application for accreditation was submitted. The evaluation by independent consultants provided feedback that affected the actual development of the courses, so that changes and revisions could be carried out before the course was put on the market.
Another factor that had a positive impact on quality assurance was the limiting of the process of accreditation to a period of three to five years. In this way Norwegian correspondence schools had to establish systematic routines for revision, reorganisation and updating of the course material and the development of new courses.
In addition to the accreditation of individual courses, the Correspondence School Council was, as mentioned above, responsible for supervising the schools.
It is likely that the authorities' follow up, supervision and economic support have played an important role in ensuring that distance education in Norway, which so far has mainly been represented by the independent distance education institutions, is recognised nationally and internationally for its high level of quality and competence. The situation is also a result of the institutions' willingness and ability to carry out systematic quality improvements, partly through research and development work. Some of this work at NKI will be described later in this article.
The repeal of the Norwegian Act concerning Correspondence Schools, the placing of the approved, independent distance education institutions under the jurisdiction of the Norwegian Adult Education Act, and the delegation to the institutions of independent responsibility for their quality improvement work were each in many ways the result of a natural development.
This change is partly a result of certain weaknesses in the old accreditation and supervision scheme and partly a result of the relatively high level of competence that the independent distance education institutions have attained. Furthermore, the technological and pedagogical developments in distance education and recent ideas about how quality improvement work should be organised and implemented in general have also tended to support the change. Last but not least, we should mention the close co-operation and trust between the institutions and the Ministry that have been established in recent years. All of these factors are closely related and the result is that evaluation and quality assurance have primarily become the institutions' own responsibility.
Some weaknesses of the accreditation scheme under the Norwegian Act concerning Correspondence Schools can be related to factors involving the question of objectivity and how the institutions might utilise the findings of the Correspondence School Council's consultants. The question of objectivity cannot be resolved in any particularly easy way (Scriven 1975). It is not possible to guarantee that external appraisers will be able to delve sufficiently deeply into the course to give a reasonable appraisal or that they will be more objective than internal appraisers. Another problem is that those responsible for a course may loose their own initiative in the quality appraisal. For example, it is not very promising for an institution's quality improvement work if they regard an external approval as a guarantee that a product meets the quality requirements that they themselves ought to be ensuring.
Due to questions of competence and objectivity, the Correspondence School Council's consultants had to be recruited from institutions that did not offer distance education. This meant that in many ways they were not very well qualified to evaluate correspondence courses and correspondence school education. Often they were poorly qualified to appraise what distance education entails on the whole and how the material in an individual course should function as one element in the total study programme. They could easily end up evaluating the actual course material for itself and comparing it directly with text books that were often developed for other target groups.
There is little doubt that some consultants also started out with negative attitudes to distance education: "My field cannot be taught through a correspondence course." In some cases the consultant has had a more teacher-oriented attitude to education as opposed to the student and/or learning-oriented attitude, which has characterised distance education for a long time. These factors may have made it difficult to accept the teaching philosophy of the course.
Whereas the consultants appointed from other institutions could be regarded as objective, it is clear that many, particularly in recent years, were not. The lack of objectivity might have been caused as other institutions regarded the distance education programme as a competitor. The recruiting base for certain types of education has been scanty at times, and some institutions may have been slow to adapt to demands from students or the authorities for less traditional and more flexible study programmes.
It should certainly also be mentioned that external appraisals may cause those people responsible for the course to pay more attention to the consultant's appraisals than they do to the students' needs. In some cases, in fact, it turned out that the consultant demanded changes in the course that the schools' course development personnel, on the basis of their knowledge of the field and of distance education, regarded as detrimental to the quality of the course.
During the years that the Norwegian Act concerning Correspondence Schools was in effect, the schools underwent an extensive competence building. Since 1970 some of the schools, mainly lead by NKI, had been conducting significant research and development work, either individually or in collaboration, which resulted in internal competence building and external recognition. Under these circumstances, the external supervision of details seemed somewhat unnecessary.
Taken together, these factors have brought the authorities to the conclusion that distance education should be regulated as a kind of adult education, and that accredited distance education institutions themselves, in collaboration with the authorities, are best able to take responsibility for quality control and quality assurance.
After the public regulation of distance education in independent institutions had been integrated into the Norwegian Adult Education Act, effective 1 January 1993, the responsibility for ensuring the quality of the learning material, the teaching and the practical implementation of the study programmes was delegated to the individually approved distance education institutions. The Norwegian Association for Distance Education (NADE) was requested by the Ministry to prepare guidelines for quality standards in distance education (see NADE 1996, Ljoså & Rekkedal 1993).
In the documents related to the bill it was specified that the evaluation of quality ought to have a broad basis. This was been expressed as follows:
NADE's Standing Committee on Quality was appointed late 1992. It is supposed to be the association's expert body in quality matters. It is supposed to work on matters involving quality criteria, quality standards, and quality assurance and improvement in distance education. The first main task of the committee was to develop Quality Standards for Norwegian distance teaching institutions.
The individual distance education institutions differ greatly among themselves in purpose, type of activity, resources and size. It is therefore difficult to devise quality standards that are equally applicable in spite of these differences. NADE's standards are supposed to be recommendations and must give the individual institution sufficient freedom to define quality requirements on the basis of its own circumstances and possibilities. At the same time they must establish certain minimum requirements that are expected to be met if the institution is to be able to maintain a justifiable level of quality.
The quality standards have both an internal and an external function. Internally at the individual institution, they are supposed to serve as guidelines for the institution's own quality improvement work. They should not relieve the institution of the responsibility for defining its own quality improvement policy and quality goals, nor should they prescribe in detail how the quality improvement work should be carried out. However, they should help the institutions to become more aware of their responsibility for quality in various areas, serve as an aid in the institutions' self-evaluation and define the agreed standards of NADE, which an accredited institution is expected to meet.
After consulting several reports and systems on evaluation and quality assurance, the committee took its point of departure in a matrix of problem areas for evaluation of education reported from the University of Lund in Sweden (Nilsson 1992). The NADE Quality Standards divide the distance education institutions' activities into four main categories:
Each of these main categories is again divided into four "phases":
These are combined in a matrix of 16 elements called quality areas. For each of these quality areas certain factors have been specified, which should be examined when the institution evaluates its own quality. The quality standards that have been specified are grouped and numbered according to the areas and factors included in the matrix.
NADE's Quality Standards are recommended standards, and in most cases they are formulated rather generally so that they can be applied in very different institutions.
The main reason for accounting for the change in control and accreditation schemes relating to distance education on the national level is that these governmental regulations have profound effects on the institutional level concerning formal quality assurance system and evaluation procedures. Experience so far has shown that the institutions have become subject to quite rigid external requirements for documentation, primarily in connection with reporting to and supervision by the Ministry of Education, Research and Church Affairs. It is, of course, seen as an advantage for both the institution itself and the Ministry that quality improvement measures and quality assurance can be documented in a well-arranged and easily accessible form. However, it is a danger that the emphasis is turned from effective evaluation and quality work directly related towards improving the teaching system to formal systems suited mainly to satisfy external demands. NKI has a firm policy to devote the resources to the former.
As mentioned above, NKI has set up a separate unit for evaluation and research activities. While external evaluation has had some positive effects on the quality of the teaching system, external evaluation functions mainly as summative.
Like most other distance teaching institutes NKI is mainly concerned with formative evaluation and assessment as a means for revision and development (see Scriven 1981).
Internal evaluation is carried out by the unit for research and development both on its own initiative and on contract from the distance teaching unit. We agree completely with the principles put forward by Scriven (1975) for controlling evaluation bias that results should not be fed back only to the persons responsible for the programme or evaluation object, but also to their superiors. The normal procedures for internal evaluations carried out by the NKI research and development unit includes full internal publication.
Evaluation is also initiated and carried out by the staff directly responsible for the administration of a certain course or programme. A programme administrator may develop and send questionnaires or carry out a telephone interview with students and tutors on a course etc. These evaluations tend to be more informal in design, development of instruments, and in reporting the results. From time to time the research unit is asked to give advice on such projects.
At NKI we have experienced that formative evaluation of courses gives the best results when conducted by an internal specialist unit, not directly involved in the operation of the programme, but working in close co-operation with the programme staff. These evaluations should preferably be supplemented by evaluations done as a natural routine by the staff responsible for the programme, because everyone engaged in education (or any other activity) should see evaluation as an integral part of his/her work.
It seems clear that evaluation activities initiated and carried out by the programme administration staff should be highly stimulated. They certainly give useful information. At the same time there is no doubt that the results, interpretations and conclusions often are biased. The results should be, and often are, discussed with people less involved and preferably trained in educational research and evaluation techniques.
Sometimes there are good reasons for contracting external support in evaluation. It might be that it is of special importance to secure that the results are as unbiased as possible and accepted as unbiased by the external market. Sometimes we need extra manpower or additional competence.
What or who is to be evaluated? In the introduction to this article we stated that we would concentrate on the evaluation of teaching systems, not student evaluation. However, as relevant student learning is the ultimate criterion for measuring teaching quality, student evaluation is, in fact, crucial also in systems evaluation.
At NKI student evaluation takes place continuously during the studies. Distance students submit their assignments and receive from their tutor comments, corrections and evaluations as subjective overall comments and a formal grade. Courses below tertiary level normally have a quite high submission density, every submission is expected to cover 10 to 25 hours of study. Tertiary level courses have much lower submission density, one submission is normally expected to cover 30 to 50 hours of study (Rekkedal & Blakstad 1994). Most courses end with a formal examination, either internal with internal examiners, internal with external examiners from public colleges/universities (according to requirements for national accreditation of college and university exams) or public nation-wide exams.
The following paragraphs represent nearly direct quotations of the advice given to NKI tutors to help them in evaluating and grading student assignments (Rekkedal 1997):
When evaluating performance, the setting of grades can be a special problem. Written assignments in distance education must not be regarded as precisely parallel to a test in a conventional school. At school, a test is given mainly in order to examine what the students have learnt. In a distance learning course, the written assignments are an integral part of the tuition and the basis for the two-way communication which consists of the student's answers (with possible questions), and the tutor's corrections, comments and draft answers. We can therefore say that the teaching to which the student is entitled and which the distance course contains in total in each study unit is not completed until the student has received and gone through the tutor's comments and evaluation of the answers to the exercises. Consequently, it is a bit of a paradox when we assign grades for each answer in distance teaching - we give grades on the student's performance somewhere in the middle of the learning process of the task that is evaluated! This paradox is still more obvious in cases where the average of the grades given during the study constitutes the final grade for the course.
Students, if they are aware of such procedures, may refrain from sending in answers when they meet problems in the course because they know this will lead to a weaker performance and subsequently bad grades. It is in just these situations that the student has a special need for the guidance and help the tutor can give. In the worst cases, such circumstances lead to the student's complete loss of courage, and to his giving up of the studies altogether.
We have yet to find a satisfactory solution to this problem. However, we will indicate some ways in which it may be partly overcome.
To get the highest grade, the student must show ability in independent treatment and evaluation of the material. In addition, the answer must be well presented and lucid. If a student has solved all the exercises in an assignment, free of errors, and the tutor has put a 'correct' mark beside each exercise without a comment, the student may be inclined to think that he/she has earned the highest grade. However, in many cases the tutor will give a slightly lower grade because he/she feels the right solutions can be even better done - with regard to content, logic, language, or even aesthetics. In order to avoid misunderstandings, one should comment along the way in the answers. In addition, the tutor should always state his/her reasons for the judgement.
Concerning low grades and incomplete answers, as mentioned above, it should be noted that many students become discouraged by having to revise or repeat an assignment. It could also cause the student to drop out. Sometimes it is good advice to give a preliminary grade and let the student decide whether to revise or continue further in the course.
In courses where one specific study unit is not a prerequisite for studying the next, the student could just continue after being informed about consequences for final grades or for earning a course certificate.
Most assignments for submission in NKI distance learning courses are of the essay type or different types of problem-solving assignments in subjects like mathematics, natural sciences and technology. Multiple choice and other types of objective tests have been introduced in some subjects - without much success. The main reason is that experience shows that making real good assignments with adequate 'distractors' demands a lot of time and competence. If in the near future we should use on-line scoring and feedback in courses offered "electronically", some submissions might consist of multiple choice type questions.
Assignments for submission are mainly measures for learning. Grades are meant to give information to help students to learn and to motivate them for further work. All NKI study programmes prepare for formal exams. Students who for some reason do not sit the exams, receive a transcript of study results when completing a study programme including final grades for each course. The transcript is a documentation of completion and might be taken into consideration by employers, but has no formal value. NKI study programmes may prepare for the following types of exams:
According to Norwegian requirements for national accreditation of college and university exams, all exams at tertiary level must have two examiners, one internal and one external. The external examiner is normally teaching equivalent subjects at another institution.
After having completed the distance study programme and having passed all the NKI internal exams related to the programme, the student receives a certificate from NKI. The certificate lists all courses completed including marks given by the distance tutor and exam results.
To some extent examination results are used in course evaluation. In our experience, feedback from tutors and students in other forms than just grades given for individual assignments gives a better basis for assessing the teaching quality of a course or parts of a course. However, from time to time comparisons are made between examination results of distance students and full-time or part time students taking the same courses. Such comparisons may at least give some indication of the quality of the distance study material.
In general, we are not in favour of putting too much emphasis on comparative studies. One has to take into consideration that students are self-selected with regard to the different forms of study, and that in general, part-time students drop out more frequently than full-time students. The raw results from some of our statistics indicate, however, that distance students achieve better results than part-time-on-campus students (Paulsen & Rekkedal 1990).
As described above, the organisation is assessed by external bodies as part of the Ministry's general accreditation procedures of distance education. After the change in legislation in 1993, this external control has shifted to emphasise quality-assurance systems more than the assessing of quality as such - as a direct consequence of delegating responsibility to the institutions. Thus, it has become even more clear that it is the internal assessment activities which secure necessary development according to external and internal requirements.
Within the field of distance education NKI has carried out extensive survey research programmes to see whether the organisation functions according to its own goals and objectives.
Recruitment and drop-out surveys have been considered to be of great importance. By means of recruitment surveys we obtain information to decide whether NKI recruits students according to national and institutional policy so that NKI should function nationally as a supplement to other institutions in securing possibilities for education despite of a person's geographical, social or educational background.
We have carried out drop-out surveys to decide whether drop-out rates are acceptable in terms of internal and external criteria. The studies have been designed to find reasons for dropping out and relations between drop-out and personal, situational or course related factors. On the institutional level NKI has set specific goals to increase completion rates. E.g. on the institutional level we found in 1990 that the average level of course completions among students studying entirely at a distance was a little above 50%. A three-year plan was implemented to raise the average completion rates to a minimum of 60%. During 1993 to 1996 the average completion rate was at 69% (including students studying entirely at a distance and students in combined studies - distance study supplemented by local face-to-face classes).
When the author chaired the Research Committee of the International Council for Distance Education some criticism was raised against the Committee's support of research projects on drop-out and completion rates. The basis for the criticism being that distance educators' preoccupation with drop-out problems lead to the traditional educational community seeing high drop-out rates as a special problem of distance education. At NKI we consider drop-out statistics to be one important indication of institutional success (although we agree that not every dropout necessarily proves there is a problem in the system). We do not believe that distance education generally has a more serious drop-out problem than other forms of part-time adult education (and as far as possible comparisons go this belief is statistically confirmed). As distance educators we ought to be proud of accounting also for our failures, and thus, be able to do something about them.
In another dimension NKI carries out institutional evaluation every year in connection with recurrent long-time planning. During the planning activities an overall assessment of today's situation is undertaken and compared with a desired for situation three years ahead, and specific long- and short-time objectives for improvement are specified.
As mentioned, curricula and programmes are externally evaluated in connection with nation-wide accreditation schemes. However, if our aim really is to strive for higher quality, we have to take internal formative evaluation seriously.
When launching new programmes, at least some evaluation activities are always initiated on the programme level. Later in this article we shall give an example of an intensive and continuous evaluation programme which took place in connection with the two first presentations of a new programme for managers in the field of health and social services. An external evaluator was contracted to follow up students and teachers through a whole year. The evaluator was given free hands in deciding evaluation design, procedures and instruments. The evaluator decided to base the approach in Stake's (1976) theoretical views on 'responsive evaluation'.
Printed material and two-way communication with a distance tutor is still the basis of the majority of NKI distance study courses. Thus, high quality of the printed material is a sine qua non for the total system to function satisfactory. During the development of the courses, evaluation and assessment take place in different ways. Some examples:
An 'instructional designer' may be a member of the course development team and go through the material during the authoring phase acting as some kind of 'surrogate student' to assess the learning quality and suggest changes.
Preliminary material is scrutinised by external subject consultants to check the academic quality. Until 1993, the material was controlled by the consultant from the State authorities before it was printed. The delegation of responsibility to the individual institution has increased need for adequate internal control of the quality of learning material.
In some cases the course is printed in a preliminary version. With the help of postal questionnaires or more intensive methods, such as telephone or direct interviews, student attitudes and experiences are collected before final production. The distance tutors are asked to inform the administration about students' difficulties, printing mistakes etc. as a matter of routine.
During the last fifteen years other media, such as telephone, fax, computers (including computer mediated communication and lately the Internet and www), broadcast or taped audio and video have been introduced. When starting courses where new media and technology are involved, NKI often follows the programmes with specific research and evaluation projects. Some of these are shortly mentioned as case studies below. Similar procedures are normally followed to assess financial and educational aspects of new procedures for tutoring, student support and administration.
The distance tutors represent the students' main contacts during their studies. The quality of the tutors' work is extremely important for the total experienced quality of the system. For a long time the work of the tutors has been continuously surveyed. We have also established formal systems for evaluation of tutors. All new tutors have to go through a distance course (either by correspondence or via Internet and the www) covering important aspects of distance learning and tutoring. Prospective tutors are evaluated during this course to decide whether they possess the desirable qualities of a distance tutor (see e.g. Bååth & Wångdahl 1976).
Different criteria have been used for assessing quality. Some of these are mentioned below.
Recruitment may be one criterion for deciding whether the institution, programme or course lives up to the intentions. It is a national policy in Norway that one should have equal opportunities for education and that education on all levels shall function as an instrument for democratisation and equal status in the society. It is a fact that adult education does not live up to this ideal goal. In many instances adult education actually increases differences by recruiting more students with a high prior level of education. With a formal institutional policy of supplementing public school by expanding opportunities for education, recruitment surveys are important. NKI has carried out a number of such surveys.
On the other hand, studies of persistence are important in deciding which groups are best served. We also consider completion and drop-out rates to be important criteria for deciding the quality of study material and student services.
Grading, however, has not been found to function as the best evaluation criterion. The reason for this is that grading is normally subjective, during the continuous assessment often used individually to motivate and stimulate the students in their studies. Even as far as exams are concerned, we have to admit that evaluation is a result of the examiners' subjective decisions and not based on criteria-referenced measurements. And, as mentioned above, because of self-selection through recruitment and drop-out, comparisons between results in different student groups do not give valid data for assessing relative quality.
To a large degree subjective information from students, tutors and experts is used for assessing the quality of evaluation objects: learning material, teaching processes and student services.
At NKI we have to some extent differentiated informally between educational research, research-based evaluation and informal evaluation, projects carried out by personnel without specific competence in research methodology.
Thus, both research and research-based evaluation and informal projects and continuous assessment projects have their place in the total system of evaluation. Over the years, we have carried out both experimental research, survey research, developmental testing and case studies in this connection.
In line with the discussion above both quantitative and qualitative analyses are applied. According to our experiences it seems clear that in the case of evaluation of a certain product - whether a specific learning package or a complete programme, for revision purposes, intensive methods based on in-depth face-to-face interviews or interviews given by telephone are the methods which offer us more useful information than large scale studies based on postal questionnaires, which allow for more quantitative statistical treatment and analyses.
It seems clear that the results of most research projects may be used for evaluation purposes. During the years, we at NKI have carried out several experimental research projects on different aspects of distance education.
The distance education scene is in many ways suited as a laboratory for experimental research on different methods and forms of teaching (and learning) with clearly defined variables, random selection of groups and measurable results, specifically concerning criteria related to study success, drop-out and completion of studies.
A study based on an experimental design (including the testing of statistical significance) is a necessary condition for 'demonstrating' causal relationships between differences in treatment and results. If one wishes to examine whether one way of organising teaching is 'better' than another, one may select two groups of students at random, carry out the two different teaching arrangements and control for possible sources of errors. If the experimental group achieves significantly better than the control group (and if the difference between the groups is larger than that one to be expected as the result of incidental variations), we may conclude that the experimental variable has had a positive effect. On the other hand, an administrator, needing information for rational decisions, might very well act on the basis of data not statistically significant, as well as a variety of other more or less subjective information.
In general, at NKI we have had very positive experiences that the results gained by means of experimental research projects have been as valid, in the sense that administrators and academics seem to emphasise that measures proven by experimental research to have positive effect are followed up and tend to maintained in the system.
Experimental research may also have some disadvantages. There are many aspects of teaching and learning that do not easily lend themselves to experimental research. And when one has decided to carry out an experiment, it becomes important to complete the study under carefully controlled conditions, i.e. it becomes difficult to make changes in the teaching arrangements, even when it could be seen as a preferable solution. Sometimes, it could be more effective for the researcher to follow the process of teaching and learning and to respond with changes aiming at adapting the process to achieve optimal conditions for learning. Thus, the experiment may hinder rational and preferable changes for the benefit of the students taking part in the research study. Consequently, in many instances it is more convenient and more effective to carry out studies without the rigid controlling procedures of experimental research.
All experimentally designed studies at NKI have been supplemented with interviews or questionnaires to examine other effects and results than those measured statistically as part of the experiment, such as achievements (measured in grades), study progression and drop-out. We have emphasised to collect extensive information for possible decisions concerning further developments and changes in teaching, organisation or administration.
As mentioned, NKI has put much emphasis on carrying out experimental research testing different hypotheses with regard to teaching techniques, as well as on the support and guidance of distance students.
The experiments carried out at NKI have generally been carried out according to the following design (in some instances more than one experimental group have been included):
R : E X1 O1
R : C X2 O2
R means that the students have been selected to the groups at random
E is experimental group
C is control group
X1 stands for the variable that is examined, while
X2 stands for the treatment given to the control group
O1 and O2 stand for measurements of results, such as achievements/study success/completion.
In the following we shall give some examples of experimental studies carried out at NKI.
Turn-around time is defined as the time elapsed between the student mailing the assignment until it is received by the student corrected and commented upon by the tutor. In the experiment, the turn-around time was reduced in an experimental group relative to a control group, without the tutor knowing who belonged to which group. According to information from the students on a questionnaire, the experienced turn-around time was reduced from a median of 8.3 days to 5.6 days. The result was a measured increase in completion rates from 69% among the control students to 91% among the experimental students, the difference being statistical significant at .001-level. According to student responses, the critical limit of turn-around time to be accepted as satisfying was one week, a conclusion which seems to be supported by later research (Rekkedal 1983). This experiment was chosen as basis for an international co-operative project organised by the ICDE Research Committee aiming to replicate previous research in different settings (Taylor et al. 1986).
In another experiment we tried to examine the effect of following up inactive students by a sequence of postcards and motivational letters. The sequence was started automatically when a student had not submitted assignments for one calendar month, and stopped when the student started to submit assignments again - or established contact with a student advisor. Differences in study activity between 240 students in each of one randomly selected control and one experimental group were measured. We found clear and statistical significant differences between the two groups after three months. During the third month of the experiment, 46% of the experimental and 31% of the control group students were active studying and/or in contact with their advisor (Rekkedal 1972a).
The above project initiated another experiment involving 3 experimental groups and a control group to examine the effect of introducing an introductory course in study techniques and systematic follow-up of new students. This experiment concluded carefully that the course in study techniques combined with initial follow-up may reduce early drop-out (Rekkedal & Hallem 1975). The experimental studies also include a study on the use of pre-produced tutor comments and standard solutions in distance education, as well as a couple of media research studies, 'the telephone as a medium for tutoring and guidance' (Rekkedal 1989) and 'telefax for two-way communication' (Rekkedal 1992).
This experiment arose out of experiences from the above mentioned projects, literature search and a specific aim of serving new students in a better way.
During the planning stage we carried out some intensive group interviews with several new students. These interviews showed that the students seemed to be generally satisfied with their experiences in distance study.
The students reported, however, one common difficulty: They were reluctant to contact the administration, the counsellors or their tutors when they met problems, and they were uncertain about whom to contact in order to seek advice on different problems.
We felt that the rationalisation and 'industrialisation' of distance education which seem to be necessary to cater for large student groups at acceptable costs in a 'large scale' distance teaching system probably result in a division of labour and a depersonalisation of the teaching processes. This situation seemed to cause greater difficulties for many students than we were aware of.
In the early days of correspondence education, the schools were often established by enthusiastic persons stimulated by an idea or a specific situation. What these people lacked in formal pedagogical knowledge, they compensated by entrepreneurial vision and devotion to their work. As professional distance teaching organisations developed, some of what we earned in theoretical orientation and efficient systems might have been lost in reduced personal teaching/learning relationships.
Thus, we decided to design an experiment where we wished to personalise teaching by introducing what we called 'the personal tutor/counsellor'.
In the experiment one person took on the roles of different tutors in different subjects, as well as student advisors/counsellors. Within the responsibilities of this tutor we also tried to include other measures which we believed were important to help the distance learner to complete his/her studies successfully. In short, we tried to construct a system which would constitute a "new" way of organising the tutor's work during the initial phases of the study period, the main emphasis put on increasing the experienced quality of the didactic functions of the distance tutor and the two-way communication between the tutor and the student. In total, 10 different aspects of tutor-student interaction were included in the experimental variable.
The students selected for the experiment were assigned to a personal tutor who followed them closely during the first 3 to 11 courses of the study programme for a certain qualification. By this formal change in organisation, a number of aspects of the tutor's work and the division of labour between the administration, the counsellors and the tutor were changed.
We also selected a control group, which was not deprived of any of the services which had been introduced on the basis of theory and research during the preceding years, and which at that time constituted a normal part of the NKI distance teaching system. The main difference in the treatment of the experimental and the control group was that the experimental students were taken care of by one personal tutor combining administrative, teaching and counselling functions, which normally were divided between different persons, departments and specialists.
Data were collected from the normal NKI files and study records, and minute records kept by the tutor on the communication with the students, follow-up letters, special measures, telephone calls etc. A small questionnaire was developed to assess the students' attitudes. It contained questions on correspondence study in general, the tutors' work and counselling to examine whether different aspects of being a correspondence student were experienced differently by the students in the two groups.
The students were selected for the experiment for a period of 5 months. Data on study progress were collected at the time of completion of the first defined stage of the study programme, at the time of cancellation or exactly 8 months after enrolment (for students who neither had completed nor dropped out).
It is easy to see that the experiment has some drawbacks concerning:
We were clearly aware of these problems when deciding to carry out the project. We simply wanted to examine a total system, instead of looking at isolated variables - the aim being to see if the new organisation "could produce a better total quality system". Consequently, we find that possible generalisations to other systems must origin in the basic ideas, rather than in specific findings.
The experimental role of the tutor is described below. The experiment covered 10 different aspects of the tutor's work.
| Aspect | Experimental group | Control group |
| 1. Tutor | Same tutor during the first 3-11 courses | Different tutors in different courses |
| 2. Employment | Permanently employed full office time | Part-time employment at home, paid per assignment |
| 3. Tutoring/ counselling | Same person responsible for all student communication | Responsible for written assignments only, other persons for general counselling |
| 4. Turn-around time | Assignments returned the same day from the school | Assignments sent via the tutor's home address |
| 5. Study technique | Same tutor teaches study techniques | Specific part-time tutor in study techniques |
| 6. Follow-up of new students | Tutor takes contact with all new students via mail or phone | Automatic routines with form letters |
| 7. General follow-up | Tutor takes contact with all inactive students via mail or phone | Automatic sequence of form letters |
| 8. Telephone tutoring | Students may phone the tutor. Tutor calls when needed | No systematic use of telephone tutoring |
| 9. Tutor presentation | Personal presentation with photo and phone numbers enclosed with the study material | Presentation of each tutor enclosed with first assignment returned from the tutor in each separate course |
| 10. Pre-produced tutor comments | Developed for all courses. Applied when needed | May have been used by some tutors |
Figure 1. Aspects included in the 'personal tutor/counsellor experiment'
As a result of the experiment we hoped to find some answers to the following questions: Will this organisation have any effects on the number of students starting their studies (defined by submitting one or more assignments)? Will this organisation have any effects on drop-out rates during the early stages of study? Will this organisation have any effects on the students' pace of study and general study activity? Will this organisation affect the students' attitudes toward correspondence study in general and/or on specific aspects of correspondence study? What will the consequences of this new tutor role be regarding the organisation of other aspects of the distance education system as well as the operational costs of the system?
We found marked and significant differences between the groups. 8 months after enrolment the experimental group had a significant higher rate of completion. The number of active students was comparable in the two groups, while the control group had a significantly higher number of inactive students. Similar results were found after one year of study. The experimental students were more active in their studies and had completed a larger number of study units and single courses during the experimental period.
Concerning 'start/non-start' we found no significant differences between the groups. The non-start rate was 'low' in both groups, 9 and 13 percent respectively. Neither did we find any significant difference concerning lapse of time between enrolment and registration of the first assignment.
In general, the experimental students reported more positive attitudes towards 'correspondence study in general', 'the study material' and 'the work on assignments', aspects not included in the project. None of these differences were, however, statistically significant. This was hardly unexpected, as these aspects were not part of the experimental variables. Concerning 'feelings of isolation' we found less problems in the experimental group, but again the differences were not statistically significant. This might be surprising as one could expect that the increase in personal communication we hoped to create by the integrated personal tutor/counsellor role would reduce possible feelings of isolation in the studies.
On the other hand, we found clear and significant differences on the following areas: The experimental group students expressed more positive attitudes towards 'the general quality of the tutor's work', 'assistance and support from the tutor', 'guidance and counselling from the institute, tutor or counsellor' and 'follow-up by the institute and/or the tutor', and whether 'telephone tutoring is of any help in distance study'.
The work on the 'personal tutor/counsellor' concept and this experiment resulted in NKI organising the initial phases of study according to these principles. We found that introducing the personal tutor/counsellor on a normal basis was both financially and pedagogically sound. However, during the last couple of years organisational changes, new media and some difficulties concerning cost and efficiency control of the tutor/counsellor role in NKI have gradually led to different organisational, administrative and teaching structures. Basic ideas from the project have survived and are implemented by emphasizing the counselling functions of the distance tutor, whether part-time or full-time, and in connection with the introduction of new media and methods, such as systematic telephone tutoring, fax communication, computer mediated communication and two-way video conferencing.
Concerning survey research, our first project carried out an intensive recruitment and drop-out analysis of one year intake to the distance study programmes (Rekkedal 1972b). This survey was followed up by two other surveys. The first looked in less depth into three years intake to see whether recruitment and completion trends had changed, the latter in relation to different procedures and student services introduced during the period. An other study compared recruitment to distance study and full-time engineering courses and followed the students through a four-year period and was ended with a completion/drop-out survey. This longitudinal study of distance students gave insights into differences between the students' plans, attitudes and expectations at enrolment and their experiences and attitudes given at the time of completion or cancellation (Rekkedal 1973b, 1976, 1978).
The results from these studies may very briefly be summarised in the following statements:
Qvist-Eriksen (1979a, 1979b) carried out a large-scale evaluation survey of NKI combined courses (distance courses supported by local face-to-face teaching). Questionnaires were distributed to 142 teachers and 739 students. The teachers were asked to evaluate the quality of the materials, general aspects of the programmes, NKI services and procedures, roles of the distance tutor and local tutor etc. Similar aspects were covered by the students' questionnaires. The results were used for revision of the system as such, specific programmes and procedures and course material.
From the early eighties NKI has carried out research and tried out different kinds of technology. These projects have often been organised as 'field studies' of specific media and technologies, sometimes in co-operation with other Norwegian institutions, such as the Norwegian Telecom Research. This research has covered technologies such as telephone and fax applications (Rekkedal 1989, 1992), video, television and two way video conferencing (Holden 1992, 1993), audiographics (Rekkedal & Vigander 1990) and computer-mediated communication (Paulsen 1989a, 1989b, Paulsen & Rekkedal 1990a, 1990b).
The EKKO Project on computer mediated communication in distance education may stand as one example of testing of new media and communication technology. In spite of quite modest expectations of revolutionary changes when introducing new media in the distance learning system, we looked upon computer communication and conferencing as specifically promising. We found that many of the ideas of personal communication examined and supported in previous experiments, such as 'the personal tutor/counsellor', could be built into a system of computer-mediated communication. Computer-mediated communication changes a lot of well-known variables in distance education (see for instance the discussion of computer conferencing as a 'new (3rd. or 4th.) generation' or 'shift of paradigm' in distance education; cf. Lauzon & Moore 1989, Garrison 1986, Holmberg 1990). Still, we believed that the medium could be introduced within the frames of a large scale system without too many difficulties.
The NKI EKKO Project (the acronym EKKO stands for 'electronic combined education' in Norwegian) started in 1986. The aim of the project was: to develop a computer-based conferencing system specifically designed for distance teaching and learning and to test it in different contexts to gain pedagogical and administrative/organisational experience within distance education based on computer conferencing - in order to install conferencing as a standard option for NKI distance students.
The project has been organised as a long-term development task following these stages:
The EKKO Project has covered system development and theory studies as well as small surveys. The project has produced some research reports and many articles and conference papers. One specific student thesis analysed problems concerning adaptation and integration of distance education technology in the student's family (Bjørgen 1992). Morten Flate Paulsen, initiator of the EKKO Project, is working on a thesis on 'pedagogical techniques for computer-mediated communication' (Paulsen 1993).
After nearly 10 years of research into computer conferencing, it is now offered as a standard option in several NKI courses and programmes, specifically at tertiary level. During these years we have continuously developed the computer and administrative systems and developed and tried out computer conferencing based distance learning in different subjects. Information has been collected systematically, both by the project manager and by the research unit. The methods applied have included continuous logging of communication activities, questionnaires and interviews with 'electronic college' students, prospective students, correspondence students (for comparisons) and teachers, and comparison of examination results. All data have been continuously published and reported.
Through our research we have learned that computer-mediated communication as a technology is quite different from other media in distance education. However, we have not drawn the conclusion that computer conferencing in itself represents a new generation of distance education (Nipper 1989). Since 1994 we have changed from applying specific conferencing software to using the Internet and www.
We have experienced that there are large differences in introducing computer conferencing in large-scale and small-scale distance education systems. Distance teaching by computer conferencing puts large demands on the tutor's time and efforts. Increased costs in the teaching phase should ideally be compensated by reduced costs involved in course development. We have found that it is difficult to reduce development costs without reducing the quality. Thus, computer conferencing in a large scale system may easily become very expensive.
To be able to exploit the possibilities of group activities we have decided to organise the computer conferencing courses with fixed starting dates and fixed progression schedules, which has led to computer conferencing being less flexible than correspondence education. It seems that many students prefer the flexibility and individual freedom of correspondence study, while others stress the advantage of group communication in computer conferencing as a necessary condition for their participation in distance education. We see great challenges in trying to develop computer conferencing to take care of both the social needs of some students and the needs of other students who prefer the flexibility of individual study with possibilities of more efficient communication.
As mentioned previously in this article, NKI emphasises formative evaluation in connection with the development of new courses and programmes. The following is one example of a systematic evaluation project as part of specific programme development. Some years ago NKI started the planning for developing a two-year distance learning programme in "Management and Administration for the Health and Social Services Sector". The programme in a preliminary version was ready for testing in 1990. During the development phase external experts had been contracted to evaluate the curriculum plans as they were finalized. Each single course was assessed and accredited by the government before it could be tested in the trial run (as part of the State-control routines at that time). The whole programme was submitted to the National Co-ordinating Body for University Studies, assessed and became formally recognised equal to one-year-full-time study in the national system.
It was decided not to put the programme on the regular market during the first year. A contract was signed between one major hospital and NKI to carry out one full trial run. Later another trial group was accepted. During the test period major aspects of the programme, individual courses, study material, distance and face-to-face teaching/learning processes were formally evaluated for updating and revision before the programme was offered on the regular market (Koch 1990).
An external evaluator was engaged to design and carry out the evaluation programme. The evaluator bases the design on Stake's (1976, 1981) responsive evaluation or transactional model for evaluation (Peersen 1992).
The evaluation plan focused on three main areas:
The results are caused by an interaction of frame factor variables and process variables.
In line with Stake's theory, the evaluator wanted to find out what is of value to the students by gathering expressions of worth from the various participants. She stresses that the aim of the evaluation is to understand rather than to explain or to convey propositional knowledge.
During the evaluation process she therefore applied different methods. Some examples:
The evaluation programme was meant to function as formative both for the trial period students and for revision of the total preliminary programme.
This article has tried to account for some NKI concepts of system evaluation, some practices and experiences. Some results from evaluation and research projects have also been presented. We have found it difficult to go into details of results because the different surveys, experimental research projects and evaluation projects cover many areas and dimensions. Describing specific results in more detail would go beyond the scope of this article.
There is one important point we would like to stress. Systematic collection of data and reporting, in our experience, is the absolute fundamental basis for lasting changes and quality developments. We have seen that good ideas may be tried out and lead to a positive change in practice. However, if one does not take the time and resources to really examine and report the effect of specific changes and developments by methods which people trust, changes seem to have a tendency to be only short-lived. When people in the system are actively involved in the developments, and the developments are proven to be effective by accepted methods of research and evaluation, the developments tend to be followed up - and live.
Bajtelsmit, J. W. (1988): Predicting Distance Learning Dropouts: Testing a Conceptual Model of Attrition in Distance Education. Report to the ICDE Research Committee.
Birnbaum, R. (1989): The Quality Cube: how college presidents assess quality. In: Quality in the Academic: Proceeding from a national symposium, National Center for Postsecondary Governance and Finance. University of Maryland.
Bjørgen, A. M. (1992): Teknologi og hverdagsliv. Fjernstudenters bruk og opplevelse av datakonferansesystemet EKKO med familien som læringsmiljø. Bekkestua: NKI/SEFU
Burton, N. W. (1977): Performance Standards. Occasional papers series, 11. Evaluation Center, College of Education, Western Michigan University, Kalamazoo.
Bååth, J. A. & Wångdahl, A. (1976): The tutor as an agent of motivation in correspondence education. Pedagogical reports 8. University of Lund, Lund.
Franke-Wikberg, S. & Lundgren, U. P. (ed.) (1981): Att utvärdera utbilding. Del 1. Stockholm.
Garrison, D. R. (1986): Multifunction microcomputer enhanced audio teleconferencing: Moving into the third generation of distance education. International Journal of Innovative Higher Education, 3:1, pp. 26-29.
Glass, G. V. (1977): Standards and Criteria. Occasional Papers series, 10. Evaluation Center, College of Education, Western Michigan University, Kalamazoo.
Holden, G. (1992): Videokonferanser som del av undervisningsopplegg i fjernundervisning. Bekkestua: NKI/SEFU.
Holden, G. (1993): Multipunkt videokonferanse. Evaluering av fjernundervisning via video. Bekkestua: NKI/SEFU.
Holmberg, B. (1989): Theory and Practice of distance Education. London, Routledge.
Holmberg, B. (1990): A Paradigm Shift in Distance Education? Mythology in the Making. ICDE Bulletin 22, pp. 51-55.
ISO 8402 (1986): International Standard - Quality Vocabulary.
Juran, J. M. (1988): Quality Control Handbook. New York, McGrawHill, 1988.
Koch, K. (1990): Evaluering av høgskolestudiet i administrasjon og ledelse. (Internal paper). NKI, Bekkestua.
Lauzon, A. C. & Moore, G. A. B. (1989): A fourth generation distance education system: Integrating computer-assisted learning and computer conferencing. The American Journal of Distance Education, 3:1,
pp. 38-49.
Ljoså, E. & Rekkedal, T. (1993): From external Control to Internal Quality Assurance. Background for the development of NADE's Quality Standards for Distance Education. Oslo, NADE.
NADE (1996): Kvalitetsnormer for fjernundervisning. Revidert utgave. Oslo, NADE (English version of the NADE Quality Standards 1993 version in Ljoså, E. & Rekkedal, T. (1993).
Nilsson, K.-A. (1992): Utvärdering för kvalitet. En handledning för grundutbildingen. Pedagogisk utvecklingsarbete vid Lunds universitet nr. 92-179. Lund, University of Lund.
Nipper, S. (1989): Third generation distance learning and computer conferencing. I Mason, R. & Kaye, A. (1989): Mindweave. Communications, Computers and Distance Education. London: Pergamon.
Paulsen, M. F. (1989a): En virtuell skole. Del I. Fundamentet i EKKO-prosjektet. Bekkestua: NKI.
Paulsen, M. F. (1989b): En virtuell skole. Del II. Erfaringer fra EKKO-prosjektet. Bekkestua: NKI.
Paulsen, M. F. (1993): Pedagogical Techniques for Computer-mediated Communication. Pedagogical Online Seminar, Oslo, February 12, 1993.
Paulsen, M. F. & Rekkedal, T. (1990a): The Electronic College. Selected articles from the EKKO project. Bekkestua: NKI.
Paulsen, M. F. & Rekkedal, T. (1990b): Den elektroniske høgskolen. EKKO-prosjektet, Del III. Bekkestua: NKI.
Peersen, V. S. (1990): Evalueringsplan. Internal document. Bekkestua, NKI.
Peersen, V. S. (1992): Evaluering av høgskolestudiet "Ledelse og administrasjon for helse- og sosialsektoren. Bekkestua: NKI/SEFU.
Popham, W. J. (1981): Målbaserade administrative strategier för omfattande pedagogiska system. In Franke-Wikberg, S. & Lundgren, U. P. (ed.): Att utvärdera utbilding. Del 2. Stockholm.
Qvist-Eriksen, S. (1979a): Kombinert undervisning. Delrapport 1. Hvorledes klasselærerne vurderer NKI-skolens kombinerte undervisningstilbud. Stabekk, NKI.
Qvist-Eriksen, S. (1979b): Kombinert undervisning. Delrapport 2. Hvorledes elevene vurderer NKI-skolens kombinerte undervisningstilbud. Stabekk, NKI.
Raynor, T. E. (1985): Identification of potential distance learning drop-outs. Report to the ICDE Research Committee.
Rekkedal, T. (1972a/1973a): Systematisk elevoppfølging. NKI, Oslo. (See also Epistolodidaktika 1973:2, pp. 57-63.)
Rekkedal, T. (1972b): Correspondence Studies. Recruitment, achievement and discontinuation. NKI, Oslo. (See also Epistolodidaktika 1972:2, pp. 3-38.)
Rekkedal, T. (1973b): Tre årskull brevskoleelever (English summary: Three years' enrolments to NKI-skolen's correspondence courses), NKI, Oslo.
Rekkedal, T. (1976): Tekniske studier. Korrespondanseundervisning og klasseundervisning. NKI, Oslo.
Rekkedal, T. (1978): Tekniske studier. Korrespondanseundervisning og klasseundervisning. Delrapport 2. NKI, Stabekk.
Rekkedal, T. (1983): The written assignments in correspondence education. Effects of reducing turn-around time. An experimental study. Distance education, 1983:2, pp. 231-251.
Rekkedal, T. (1985): Introducing the personal tutor/counsellor in distance education. NKI, Stabekk.
Rekkedal, T. (1989): The telephone as a medium for instruction and guidance in distance education. NKI, Bekkestua.
Rekkedal, T. (1991): Evaluation Concepts and Practices at NKI, Norway. In Schuemer R. (ed.): Evaluation Concepts and Practice in Selected Distance Education Institutions. FernUniversität, ZIFF.
Rekkedal, T. (1992): Telefax in individual distance study. English summary in Telefax som medium for toveis kommunikasjon i individuell fjernundervisning. Bekkestua: NKI/SEFU.
Rekkedal, T. (1997): The tutor in distance education. (3rd. edition of AECS Course for tutors.) in print. AECS.
Rekkedal, T. & Blakstad, I.-L. (1994): Submission Density and Assignment Workload Revisited. NKI, Bekkestua.
Rekkedal, T. & Hallem, S. A. (1975): Follow-up of correspondence students. The effect of an introductory course in study technique and encouraging letters to new enrolees. English summary in Begynneroppfølging av brevskoleelever. Oslo: NKI.
Rekkedal, T. & Vigander, K. (1990): Forsøk med bruk av Telewriter i matematikkundervisning. Bekkestua: NKI/SEFU.
Report to the Odelsting No. 55 (1991-1992) (Ot. prp. nr. 55 1991-92).
Schuemer, R. (ed.) (1991): Evaluation Concepts and Practice in Selected Distance Education Institutions. FernUniversität, ZIFF.
Scriven, M. (1975): Evaluation Bias and Its Control. Occasional papers series, 4. Evaluation Center, College of Education, Western Michigan University, Kalamazoo.
Scriven, M. (1981): Fördelar och nackdelar med målfri utvärdering. In Franke-Wikberg, S. & Lundgren, U. P. (ed.) Att utvärdera utbilding. Del 2. Stockholm.
Stake, R. E. (1976): Program evaluation, particularly responsive evaluation. Occasional papers series, 5. Evaluation Center, College of Education, Western Michigan University, Kalamazoo.
Stake, R. E. (1981): Programutvärdering - besvarande utvärdering. In Franke-Wikberg, S. & Lundgren, U. P. (ed.) Att utvärdera utbilding. Del 1. Stockholm.
Taylor, J. C. et al. (1986) Student persistence in distance education: A cross-cultural, multi-institutional perspective. Distance Education, 7, 1, pp. 68-91.
Thorpe, M. (1988): Evaluating Open & Distance Learning. London, Longman.
Van Vucht, F. A. & Westerheijden, D. F. (1993): Quality management and quality assurance in European higher education. Methods and mechanisms. Brussels, Commission of the European Communities, 1993.