The literature on evaluation in education is extensive. There is no lack on theoretical publications, textbook like writings, handbooks and specific reports on evaluation in education in general.

The researchers in the CISAER project have their main theoretical and philosophical platform within distance education. Also within distance education there is an extensive literature published on evaluation research, methods and procedures, as well as on institutional policies and specific evaluation studies (see e.g. Holmberg 1995, Schuemer 1991, Rathore & Schuemer 1998, Thorpe 1988).

There are many reasons why the evaluation of education or programmes has been taken seriously in distance education circles. Some of them are:

- The way distance education has been organised, there are fewer possibilities for direct feedback from students to those responsible for the education. This means that those who plan, develop and implement the education have to formalise their work with the aim of providing information about how the education works. This is not as necessary in systems where the teachers, who are often responsible for both planning and execution are in continuous contact with their students.

- Distance education is normally organised in a system characterised by a division of labour among a number of "expert groups." Formalised knowledge about this process thus becomes more necessary in order for the information to affect system changes.

- Distance education is often initiated on the basis of defined political resolutions and objectives concerning looking after needs in the society, e.g. increasing the availability of education for defined groups. Thus, the evaluation of results will be desirable in order to determine the extent to which the political intentions are realised.

- Distance education is an area characterised by new developments, both as regards pedagogical solutions, media and technology utilisation, and organisation. Various models and systems are tried out, and information is needed about their effects and effectiveness.

- In many situations distance education has been developed in a tradition of systems thinking and educational technology. In this tradition systematic feedback, evaluation and revision and/or further development all play an important role.

- We can probably also add that both public and private distance education have had a need to develop and ‘demonstrate’ quality because in some situations they are battling with a perception of being only second-rate. It should be stressed that this is not just limited to private institutions.

Thus, it is typical that most large institutions devoted to distance education have established separate departments or institutes for research and evaluation, or have formalised their quality improvement and quality assurance work in other ways as soon as they were established. Examples of this are Britain's Open University with its Institute of Educational Technology and FernUniversität in Germany with its Zentrales Institut für Fernstudienforschung (Central Institute for Distance Education Research) and Zenter für Fernstudienentwicklung (Centre for Distance Education Development).

Some key concepts and terms are discussed below. Some of these are focal in understanding what is meant by "evaluation" in the CISAER study. The terms have been discussed in different writings on evaluation, such as Schuemer (1991), Stake (1976), Thorpe (1988), Norris (1990) and others, e. g. Papers of the Occasional Papers Series, College of Education, Western Michigan University (Campbell 1976, Stake 1975, Scriven 1975, Stufflebeam 1975)

According to Schuemer (1991) "there do not seem to be any generally accepted definitions for "evaluation" or "evaluation research", and it may seem "to be as many definitions as there are evaluators" (Franklin & Trascher 1976, p. 20 (from Schuemer).

Evaluation can be anything from expressing subjective viewpoints from quite meagre information to conscientious and rigorous research. Suchman (1967) has suggested a differentiation between "evaluation" and "evaluation research", where "evaluation" concerns the process of assessing the worth of a product, process or programme, but does not necessarily involve any systematic activity as the basis for judgement, while "evaluation research" presupposes application of systematic scientific research methods.

One generally accepted condition and requirement for good research is that it should be objective and lead to new, truthful and generalisable knowledge. Methods that give some guarantee of validity and reliability as well as the researcher’s objectivity and critical reflection are seen as necessary conditions for research. Evaluations in general do not meet these requirements. Whether we describe our activities as evaluation or evaluation research, educational evaluation should strive at meeting the criteria of the research paradigm. It is claimed that evaluators ought to take a critical view on their own methods (see e.g. Stufflebeam (1974) on Meta-evaluation). This is one reason for this quite extensive introduction and overview of evaluation terms and concepts.

However, most evaluation projects do not meet the research criterion of objectivity. Few, if any, evaluation projects can be said to be absolutely objective. If we wish to examine reality as it is and experienced by the informant, some subjectivity seems always to be part of any evaluation process.

This is also why there is a continuous debate on who is to carry out evaluation, internal evaluators or external evaluators – or both, which advantages and disadvantages different solutions involve etc. E.g. according to Scriven (1975) there is no easy way around the question of objectivity. There is no way of guaranteeing that external evaluators are not biased. And there is a good chance that external evaluators are not able to go sufficiently into the institution and programme to come up with a useful evaluation design and get hold of the information of real importance for the evaluation. In fact, dependent on the situation, external evaluators might consciously or unconsciously take a negative view (see e.g. Rekkedal 1998, Ljoså & Rekkedal 1993). The opposite might also be the case, perhaps dependent on who the evaluator works for: " It is recognized, particularly by Mike Scriven and Ernie House, that co-option is a problem, that the rewards to an evaluator for producing a favorable report often greatly outweight the rewards for producing an unfavorable report" (Stake 1975, p. 5). During the last years there has been an increasing emphasis that those directly involved and responsible should play an important role in the evaluation process. The idea is mainly that if the evaluation really is to have a positive impact on the improvement of the quality of the educational system, the practitioners must take part in the evaluation process (Thorpe 1988, Rekkedal 1998).

In addition to concerns about professionalism related to insight in methodologies/techniques and objectivity decisions about of who should evaluate depends on insight in the programme in question and how evaluation results are supposed to be used. Evaluations may often be carried out as commissioned research and as such dependent on political, value based and financial constraints. Often external commissioned projects tend to concentrate on summative evaluation, rather than formative evaluation and have little value in increasing quality (see below). Stake (1975) stresses the need to understand an activity to be able to possibly better the quality. This indicates the need for those involved in the educational programme to be involved in the evaluation process. Stake advocates "responsive" evaluation defined as evaluation that is oriented directly on the programme activities rather on programme intentions, that responds to the information needs of the informants and that secures the value perspectives of the different informants are openly addressed and presented. The user perspective is pivotal in that evaluation should come up with important and valid information that is useful to the users – i.e. teachers, planners, organisers, administrators, authors, and the society as a whole. As pointed to above, the user perspective and the need for arriving at useful results that really will be used for educational development, make it necessary to involve the users. Arguments for a strategy based on the teacher as researcher, is perhaps most clearly formulated by Stenhouse (1975), that the most effective way of improving teaching practice is through the teachers’ criticism of practice (see also Tyler 1949).

According to Thorpe (1988) the professional evaluation researcher may help design the evaluation programme and help in formulating researchable problems, but if we wish to make optimal use of results in improving practice, the practitioners who may better understand cause-effect relations and possibilities for increasing quality should be involved.

The distinction between formative (during the development of a programme or programme unit) and summative (after the programme has been completed) evaluation has often been focussed, (see Scriven (1967) or Stake (1967)). But as Stake (1976) points out, the distinction is not clear-cut. E.g. It is not easy to distinguish between summative evaluation of a component and formative evaluation of the whole programme. "The key is not so much when as why. What is the information for,…" (Stake 1976, p. 19) is it for further change and development or for actual use?

There is a clear distinction between formal and informal evaluation studies. E.g. at NKI both the research department and personnel in other departments, such as academics, editorial staff, marketing managers or student advisors may carry out informal evaluation studies. Informal evaluation is a natural part of any educational activity, and can be carried out with few requirements on methodology (Rekkedal 1991), while "…the formal evaluation study is under an obligation to pass tests of accuracy, validity, credibility, and utility." (Stake 1976, å 19).

An evaluation study may be concerned with one particular case, with no intention of generalising findings to other courses or programmes. Or the study might be done specifically for deciding on the value of a general approach. "The more the study is expected to be a basis for generalisation the more the need for controls, controlled variation or careful description of uncontrolled variation" (Stake 1976, p. 19).

The evaluation study might concentrate on the process of teaching and learning or the product, the emphasis being dependent on a belief whether the main value should be put on the intrinsic values of the learning experience, or measurable results in student achievement after completing.

Studies can be more or less descriptive or include value judgements. Evaluation studies could be in either extreme side of a continuum from strict descriptions to emphasise mainly subjective judgements. A balance is probably more common. Often there will probably be some relation between quantitative objective measurement and descriptive emphasis on one side and a "responsive" (Stake 1976) approach based on subjective data on the other side.

According to Stake (1975, 1976, 1983) evaluation studies vary "…as to how much the issues of evaluation are determined by observation of activities and by realisation of the concerns of the participants of the programme. Pre-ordinate studies are more oriented to objectives, hypotheses, and prior expectations…. Responsive studies are organised around phenomena encountered – often unexpectedly – as the programme goes along" (Stake 1976, p. 20).

Stake (1976) also includes a dimension of holistic versus analytic approach and points to the "case study" method as an example of the holistic approach, which tries to preserve the wholeness and complexity of a programme in distinction to multivariate analyses indicating relationships between individual descriptive variables.

There has in the educational research community been a dispute among researchers, proponents and opponents of qualitative and quantitative methods in research and in evaluation. In the seventies this dispute often concerned whether to use quantitative methods within the paradigm of natural science research, including experimental designs and measurements tested for statistical significance, or methods from humanities, field studies, open ended interviews and observations (see e.g. Campbell 1976). Partlett & Hamilton (1972) were proponents for "illuminative" evaluation as a reaction towards the quantitative paradigm including the psychometric tradition. Stake’s (1983) term "responsive" evaluation also accepts to sacrifice precision in measurement in increasing usefulness of findings. It seems clear that during the last 20 years most researchers open-mindedly apply a wide range of methods depending on the aims and objectives of the specific evaluation project. This shift in views and perspectives is clearly expressed by Cronbach (1982) and Patton 1990 (from Schuemer 1991):

"Some of the debates regarding evaluation have encouraged the mistaken impression that objective, quantitative, focussed methods are incompatible with humanistic, qualitative wide-band inquiry. In fact, the two should be working hand in hand." (Cronbach 1982, p. 301)

"Because qualitative and quantitative methods involve different strengths and weaknesses, the constitute alternative, but not mutually exclusive, strategies for research. Both qualitative and quantitative data can be collected in the same study." (Patton 1990, p. 14)

As there is no lack of literature on educational evaluation, there is neither lack of definitions. Stufflebeam (1974, p. 13) gives three different definitions:

• Evaluation as "determining whether educational objectives have been achieved"

• Evaluation as "the process of providing information for decidion making"
• Evaluation as "the ascertainment of merit"

Likewise Schuemer (1998, p. 4) has collected a number of statements on what evaluation is:

• "Evaluation is an observed value compared to some standard. Stake (1983, p. 291)
• "Evaluation is marshalling of information for the purpose of improving decisions" (Thompson 1975, p. 8)
• "Evaluation is the process of delineating, collecting and providing information useful for judging decision alternatives. …Evaluation is the determination of the worth of a thing. It includes information for use in the judging the worth of a program, product, procedure or objective or the potential utility of alternative approaches to attain specific objectives" (Wentling & Lawson 1975, from Wittman 1985, p. 20)
• "…program evaluation refers to the use of research methods to measure the effectiveness of operating programs." (Rutman, 1980, p. 17)
• "The purpose of evaluation research is to measure the effects of a program against the goals it set out to accomplish as a means of contributing to subsequent decision making about the program and improving future programming." (Weiss 1972, p. 4).
• "Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs." (Rossi & Freeman 1989, p. 18)
• "By the term evaluation, we mean systematic examination of events occurring in and consequent on a contemporary – an examination conducted to assist in improving this program and other programs having the same general purpose." (Cronbach et al. 1980, p. 14)
• "…we may define evaluation broadly as the collection and use of information to make decisions about an educational program. The program may be a set of instructional materials distributed nationally, the instructional activities of a single school, or the educational experiences of a single pupil. Many types of decisions are to be made, and many varieties of information are useful. It becomes immediately apparent that evaluation is a diversified activity and that no one set of principles will suffice for all situations." (Cronbach 1983, pp. 101-102)
• "Evaluation is the process of delineating, collecting and providing information useful for judging decision alternatives." (Stufflebeam et al. 1971, p. 43, from McCormick & James 1989, p. 172)
• "Evaluation is the process of conceiving, obtaining and communicating information for the guidance of educational decision-making with regard to a specified programme." MacDonald, 1973, pp 1-2, from McCormick & James 1989, p. 172)
• "Curriculum evaluation is the process of delineating, obtaining and providing information useful for making decisions and judgments about curricula." (Davis 1981, p. 49, from McCormick & James 1989, p. 172).

From the definitions above, we may conclude that some concepts are focused on and repeated (it seems also clear that the different writers have been far from independent on each other). Some of these terms and concepts are: systematic methods, conceiving, collection and communicating information, value and worth considerations, judgements, and above all - decisions.

We have no problems in accepting the definition put forward by Thorpe (1988, p. 5) directly related to distance education: "Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education and training, as a part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have." Although in our specific project generalisations and decisions to be taken by other parties than those involved in the courses examined would have prime importance.

Researchers have suggested different models of evaluation or general approaches dependent on different views what evaluation is, and which aims it should have. E. g. Stake (1976) suggests that theoretically, after presenting his 8 dichotomous dimensions for describing evaluation designs (such as ‘formative-summative’), one could end up with 264 different designs. Why this is not so, is because the dimensions are related, both conceptually and in practice. He ends up presenting 9 different models or general approaches to educational evaluation (Stake 1976, pp 21-27):

Relying on criterion referenced tests to match pre-specified objectives (e.g. Tyler 1949).

Gathering of data, interpreting problems and recommendations done by faculty, teachers and internal administrators, initiative may come from internal or external pressure, decisions not necessarily part of staff’s responsibility.

Appointed panel, not necessarily (or usually) with expertise in education but with high social esteem and social responsibility, studies a problem and makes a formal report.

In contrast to the student-testing approach, the emphasis is on observation of educational processes – often applying methods from anthropology, ethnography, history and journalism.

According to Stake (1976), there is a great respectability of this approach emphasing experimental design and it is often recommended by researchers when asked to advice on evaluation approaches. It has been criticised for poor records concerning its ability to assist practitioners and policy makers. (Experimental research has been applied with success in NKI in connection with decisions on developing teaching strategies in distance education, e.g. Rekkedal (1983, 1985).

Stake (1976) differs between the two whether the evaluation study is done to assist programme managers or policy making with a larger time-span or for a wider constituency. Both draw on social sciences not usually involved in educational research, and the study often include cost-benefit and productivity analyses.

Scriven introduced the concept of goal free evaluation as a method to avoid evaluator bias and increase possibilities of "…discovering any effects, without the tunnel vision induced by a briefing about goals" (Scriven 1975, pp. 31-32). The Adversary approach divides the resources for evaluation in two; one part to show the shortcomings of the programme, the rest to show its merit.

House (1980, from Norris (1990), Schuemer (1991)) builds on Scriven’s and other’s classification of models for evaluation and presents an eight-category taxonomy of major evaluation approaches:

1. Systems analysis
2. Behavioural objectives
3. Decision making
4. Goal-free
5. Art criticism
6. Professional review
7. Quasi-legal
8. Case-study

Since the very first start of systematic evaluation studies in education, the aim has been to support planners and policy-makers in social planning and control. Ideas of scientific management spread from business and industry to the administration of education. In the same way, the concept of quality management and quality assurance has spread from industry to the service sector and to public administration – and education.

In industry Quality is most often defined 'fitness for purpose' related to the needs of the user/customer (Juran 1988), which indicates that quality depends upon a subject's view of what is the purpose of that phenomenon. In education the customer is not easily identified. The government pays, the immediate user is the student, secondary users are employers etc. Quality, thus, is a value judgement interpreted by different stakeholders, government, teachers, administrators, students, employers etc. On the other hand, to assure and assess quality we must have a clear notion of what it is. Another definition could be could be that the 'product comply with defined requirements'. Consequently, purpose and requirements, then, should be defined by the significant stakeholders. Birnbaum (1989) has stressed this diversity and pointed out three dimensions of quality in higher education: the meritocratic (the institution's conformity to professional and scholarly norms with the academic profession as reference group), the social (the degree to which the institutions satisfies the needs of important collective constituents) and the individualistic (the contribution the institution makes to the personal growth of students (from Van Vucht & Westerheijden 1993).

Especially since the early 1980s these concepts have also been a central focus of attention in the debate of higher education and in higher education policy making. The question has for instance been addressed by a committee reporting to the Commission of the European Communities (Van Vucht & Westerheijden 1993).

The report points out some factors that explain the attention to quality in higher education:

• Societal concern about the increase of public expenditure in general and consequently the necessity of defining priority of education relative to other socially desirable activities.
• The expansion of the higher education system and rapid growth in the student body (e.g. in Norway the number of students in higher education has grown from below 100.000 to more than 160.000 in five years).
• Increased openness in most sectors of the modern societies. Higher education institutions must show the society at large what they are doing and how well they are doing it.
• Increased international mobility of teachers, researchers and students and internationalisation of the European labour marked. There is a growing need to be able to assess the equivalence of qualifications, standards and credits.
• Extrinsic values of higher education, the service higher education provide to society, have come into focus, relative to the intrinsic values, such as search for truth and pursuit of knowledge (Ibid.).

Similar viewpoint is put forward by Franke-Wikberg (1992), when describing some general trends in quality and evaluation in Western Europe. She stresses that there are only two general models of evaluating the quality of education, 'product-oriented aiming at control of quality' and 'process-oriented evaluation aiming at quality improvements' and shows how emphasis has changed between these main models over time. According to Franke-Wikberg (ibid.) product evaluation dominated in the era of instructional technology in the 60s, gave way for more process oriented models in the 70', but seems to have revived in the late 80s and 90s.

Van Vucht and Westerheijden (1993) summarise the general trends of the new methods of quality management in higher education stressing that the focus is the process of teaching, and that the aims are both institutional quality improvement and accountability towards the society:

• Nationally there is an agent responsible for managing the system at a meta-level. This agent should be a co-ordinator, acting independently from the government politics and policies and not having to impose an specific approach on the institution.
• Self evaluation is a common element in the 'new' quality management systems (see Thorpe's arguments specifically in distance education).
• A third common element is the mechanism of peer review and visits on site by external experts. In our connection it is emphasised that in quality management systems with a European dimension international experts should be included in the review committee.
• Reporting the results and experiences is another important element. The report should not have the function of judging or ranking institutions. The aim should be to help the institution to improve its programmes.
• There should not be established any direct relationships between quality review reports and funding.

I the same way as attention on quality matters has increased in education in general, distance educators have to a large extent been aware that quality must be in focus, examples are different quality systems presented during the last few years (SATURN 1992, NADE 1993, AECS 1994). We have just touched upon recent quality work here to indicate that there is a clear relation between central issues in educational evaluation and quality assurance, although the concepts used may differ somewhat.

With reference to the overview above on evaluation, we might say that the CISAER Project generally is concerned with evaluation. The evaluation concept is clearly very broad and includes collection of information (survey), systematisation (analysis), value judgments (evaluation (?)) as bases for decisions (recommendations). Thus, all activities involved in the project are parts of what is normally understood by educational evaluation.

Sub-activities in the CISAER project can be seen as evaluation studies on micro-level. E.g. the partners have during the project period carried out internal evaluations of their own activities concerning development of WWW based training and teaching activities on the WWW. Some of these have been published as part of the project documentation of other projects, such as interviews with students, surveys of student attitudes, e-mail interviews with online teachers etc. (See e.g. http://www.nki.no/eeileo/research/project.htm)

The CISAER Project has also carried out a large number of interviews with academics and/or administrators of WWW based courses. These interviews also separately constitute evaluations on a micro level, while the total analysis of the catalogue and the interviews can be considered as macro level survey study on WWW based education and training. Some more extensive case studies will also be included in the project documentation.

The complete project constitutes a macro-level evaluation study, and as such it could be said that the project to a large extent applies a triangulation methodology in its variety of approaches to the gathering of information and the number of researchers with different, however, not conflicting perspectives.

As pointed out, the researchers in the CISEAR Project have decided to base their evaluation work in the philosophy and theory of distance education. Only courses and programmes that are not offered only to full-time campus based students are supposed to be included in the catalogue. Through a theoretical analysis of distance education, Keegan (1999) concluded: "… that web based education is best regarded as a subset of distance education and that the skills, literature and practical management decisions that have been developed in the form of educational provision known as 'distance education' will be applicable mutatis mutandis to web based education. It also follows that the literature of the field of educational research known as distance education, is of value for those embarking on training on the web."

Based on the definition of distance education proposed by Desmond Keegan (1990, pp 44-45), Paulsen (1998b) suggests definitions and relationships between distance education, online education, computer mediated communication, Internet and the WWW:

"There are many terms for online education. Some of them are: Virtual education, Internet-based education, web-based education, and education via computer-mediated communication (CMC). I suggest using a definition of online education, which is based on Desmond Keegan's definition of distance education. Hence, online education is characterized by:

• the separation of teachers and learners which distinguishes it from face-to-face education,
• the influence of an educational organization which distinguishes it from self-study and private tutoring,
• the use of a computer network to present or distribute some educational content,
• the provision of two-way communication via a computer network so that students may benefit from communication with each other, teachers, and staff.

In online education, communication is conducted as computer-mediated communication (CMC) which I define like this:

Computer-mediated communication (CMC): Transmission and reception of messages using computers as input, storage, output, and routing devices. CMC includes information retrieval, electronic mail, bulletin boards, and computer conferencing. CMC also comprises synchronous and asynchronous communication.

• Information retrieval systems: CMC systems that facilitate searching and retrieval of information.
• Electronic mail systems (E-mail systems): CMC systems that facilitate exchange of messages between individual users.
• Bulletin board systems: CMC systems that facilitate archival and retrieval of messages posted by originators with write access and retrieved by users with read access.
• Computer conferencing systems: Computer-mediated group communication systems that archive messages so that they can be read and commented on by all members of a conference.
• Synchronous communication: Real-time communication.
• Asynchronous communication: Communication where the message is stored until the receivers find it convenient to retrieve it.

Internet is one example of a network that could be used to conduct online education and CMC.

WWW is one of many services on the Internet. However, e-mail and WWW are the two most important services for online education on the Internet." (Paulsen 1998b)

According to Keegan (1999), and of specific interest in the CISAER Project with emphasis on courses on the Internet and WWW in training and education, the relationship between distance education and Internet/WWW can be represented diagrammatically as:

The illustration above must not be understood as if asynchronous communication applications of the WWW are incompatible with group work and some use of co-operative and/or collaborative learning techniques.

The project plan states that:

1. "there is little doubt that the World Wide Web is the most phenomenally successful educational tool to have appeared in a long time. … Though there is much empty rhetoric about the Web, there is also much genuine educational activity involving it. This project aims to investigate the quantity, type and quality of courses and course materials now offered on the Web."
2. "The aim of the project is to provide a comprehensive, state of the art survey of course provision on the WWW with professional analysis, balanced evaluation and far-reaching recommendations which will provide the field of VET (Vocational Education and Training) in the EU with a tool for dealing with this new training dimension."
3. "The CISAER Project will contribute to improving quality and innovative capacity of Member States’ vocational training systems and arrangements. It will highlight the achievements in the field of WWW course provisions by European Colleges…, but will point to a very considerable presence of North American corporate and university providers who are assuming leadership in this area. There will be a precise focus on quality of provision, quality of learning outcomes and quality of certification and transfer of learning from WWW-based courses to the workplace…"
4. "The aim of this project is founded in article 127 of the Maastricht Treaty, which emphasises the exchange of information and experiences on issues common to all the training systems of Member States, and the White Paper, Teaching and learning: Towards the Learning Society’s view that the development of an education and training system of the quality required can only be achieved through the general application throughout the European Union of the best practice in the different states."

As will be seen from the paragraphs above, the CISAER study is largely a survey research project. Thus, overviews and analyses constitute a substantial part of the project. However, these overviews, in the form of a comprehensive catalogue, represents information useful for evaluation – for the researchers involved in the project, for institutions represented, and for other interested parties, such as European trainers, training institutions and policy-makers in the EU and at the national level.

Evaluation on the macro-level also includes interviews with key persons in the institutions represented in the catalogue, and analyses and systematisation of information from the interviews. At the project meeting in Portugal in June 1998, the project team decided to carry out approximately 20 interviews each, which would make up approximately 80 interviews with representatives from institutions in the catalogue. Due to the fact that the interviewers did not reach some contacts and that some few contact persons did not respond, the number of final usable interviews added up to 72. As an example this is how the Norwegian team carried out the interviews:

The project agreed interview form was presented as a Word document with an introduction, and explanation an sent by e-mail to the registered contact persons at 24 selected institutions:

The accompanying letter said that the interviewer would phone the interviewee afterwards to carry out a telephone interview. The interviewees could also answer the interview by e-mail if so preferred. About half the interviewees preferred to answer the interview by e-mail, while the other half was interviewed on the phone. The phone interviews were taped and transcribed by the interviewer. To secure correct understanding both by the interviewees (of the questions) and the interviewer (of the message), the following procedure was applied:

• The interview form included a description of the CISAER Project and the information we were searching for
• The form also included some constructed example answers
• After the interview on the phone had been drafted, the transcript was sent to the interviewee for confirmation, correction and possible additions and clarifications
• Responses received by e-mail were, if found necessary, supplied with suggestions for changes and additions, which were confirmed by the interviewee either on the telephone or on e-mail
• Most of the interviewees, who responded by e-mail were also contacted by phone, in a couple of cases the final transcript was a result of a combination of e-mail and phone interviews
• When the interviewee had accepted the transcript of "final version" the interviews were presented on a passworded website available for the interviewees only. Her they could read their own answers, compare with others, and if necessary suggest changes to be made

Referring to the evaluation models and dimensions of evaluation discussed above we could probably say that this part of the CISAER project is more like a survey study. It hardly could be said to suit any specific model of evaluation, although it has some of the characteristics of decision making, professional review and case studies (House 1980). However, concerning the evaluation dimensions, the interview studies included the analyses to follow is characterised by being:

• research based and perhaps being survey research rather than actual evaluation
• external more than internal (although the information is clearly supplied by internal key persons, who also may make use of the information from others to assess their own WWW based training)
• summative more than formative (although the information might be used as input in formative evaluation by the interviewees)
• quite formal rather than informal
• being particular for each institution, however with clear intentions of being useful for generalisations
• clearly more product oriented than process oriented
• descriptive rather than judgmental
• pre-ordinate (as it is based on a form and the researcher’s basis in distance education theory and practice), however quite responsive as there is a large degree of openness in questions and in the possibility of interviewees to select and judge their own information
• more holistic than analytic
• much more qualitative than quantitative

The CISAER Interview Guide contained the following headings:

The CISAER project has carried out a systematic analysis of information included in the catalogue and information gathered through the interviews. This analysis resulted in a systematic overview of web-based education with based on the main themes and categories from the catalogue and interview guide (Paulsen 2000).

The CISAER has carried out some comparative analyses between initiatives in North America and Europe to see whether Europe can be considered to be a serious player in the field. These comparisons analyse differences and similarities between government sponsored initiatives in various countries to promote telelearning, corporate initiatives in teaching via the Internet, initiatives by individual universities, and regional initiatives (Mason 2000).

A comparative analysis of specific courses has also been carried out.

One central part of the CISAER Project will be the evaluation of a selection of courses. The courses will be selected from different types of institutions represented in the CISAER catalogue. As Keegan (1999) has pointed out, the types of institutions offering distance education can be categorised as either:

• Open universities
• Distance departments of existing universities
• Government distance training institutions
• Proprietary distance training institutions

The evaluation of the courses will mainly be based on the material available on the WWW. In addition, the evaluation of the material on the WWW will be supplemented with information collected from academics and/or administrators responsible for the courses analysed. We hope to get access beyond passwords to a sufficient number of courses to be able to describe and present the variety of courses and use of WWW for presentation and distribution of course content and for communication, and also present examples of good, excellent and recommendable experiences of training on the WWW.

Before accounting for the CISAER strategy, variables and criteria for our WWW course evaluations, we shall shortly refer a couple of recent WWW course evaluation with specific relevance for our study.

Boshier et al. (1997) have carried out an extensive study of courses on the WWW. They examined 127 courses (75% from North America) and analysed and evaluated the courses according to a 43-variable coding schedule. The variables were divided into 5 categories:

1. Site Background: Id number, Type of Institutional Sponsor (e.g. University, Government Agency), Country of Origin, Level of the Course, Subject Matter (e.g. Mathematics, Social Sciences), Whether a Stand Alone Course.
2. Accessibility: Cost, Ease of Connection, Crashed Connections Per Session, First Impression, Transmittal/Constructed Mode of Learning.
3. Web Architecture: Twelve dichotomous variables (no/yes) concerning presence of Course Notes, Offline Materials, Still Graphics, Animation, Quicktime Movie, Real Audio, Threaded Discussions, Chatroom (or E-mail), Student Work, Relevant Links, Personal Information Form Required, any part of the site Under Construction.
4. Face Validity of the Site: Owners ‘Up-Front’ About Affiliation, Instructor Identity Revealed, Photograph of the Instructor, Instructor Credibility, Soundness/Integrity of the Course.
5. Attractiveness: Links to Outside Sites, Alluringness of Links, Level of Enjoyment, Evidence of Verve/Imagination, User Friendliness, Recommend to Others, Impact of ‘Glitter’.

Boshier et al. (Ibid.) further analysed the courses according to a proposed 3.dimensional model involving "possible interactions between three variables that distinguish the ‘best’ from the ‘worst’ web courses" (Ibid. P.338). These variables were Attractiveness, Interaction and Accessibility. The authors also supply guiding questions, which web course designers should ask themselves concerning these three variables.

Although, the analysis and evaluation carried out by Boshier et al. were not directed specifically towards distance education courses for part-time students, their work is directly related to the questions to be answered also in the CISAER evaluation study. Thus, the variables and categories treated by Boshier et al. is of great relevance to our study. One other possible source of error in the above study, as far as we can judge, is that the all the courses examined on the WWW were not password protected. We would assume than many, if not most, high quality courses are closed to non-paying or unknown visitors. The CISAER Project team has made efforts to get permission to examine a selection of courses behind the password protected areas.

I a later article published by three of the researchers of the above study (Wilson et al. 1998) questions concerning the overwhelming influence of North America, and specifically the USA, are discussed in depth. This is a question that also should be taken seriously by European educators and distance education providers. The CISAER evaluation will also examine how European courses on the WWW handles questions of cultural influence from the English language and specifically Americanisation of courses through the use of external links etc.

Concerning the didactic dimensions of teaching methods, teaching techniques, teaching devices and teaching functions (see also Mason 1991) and facilitation techniques in WWW courses we have applied the model suggested by Paulsen (1998):

In an earlier article, Paulsen (1997) has summarised the relation between methods, techniques and devices as below:

Further, included in the evaluation of the courses (and also the analyses) we have looked into other dimensions of specific interest in a distance education perspective, such as:

Large Scale – Small Scale

Flexible Enrolment – Fixed Enrolment

Free Progression – Fixed Progression

Individual Study – Co-operative Learning

Transmittal Teaching Mode – Constructive Teaching Mode

One sub project of CISAER has been carried out by the Diploma students at NKI who have evaluated a number of courses on the WWW selected from institutions in the catalogue. Because of language difficulties some countries had to be excluded from these evaluations (Fagerberg et al. 2000).

This article has discussed educational evaluation with reference to different definitions, methods and models of evaluation. We have focussed on evaluation methods with direct relevance to the CISAER Project and how evaluation is to be understood in our context of distance education and WWW based training and education.

It is argued that the CISAER Project offers many points of resemblance with educational evaluation. However, the CISAER Project is also more like a survey research project which includes value judgements. According to definitions of educational evaluation as the broad task of conceiving, delineating, obtaining, providing and communicating useful information for judging decision alternatives and for making decisions, the CISAER Project is both on macro and micro level partly a survey research project and an evaluation project.

The CISAER Project intends to provide decision-makers, on the European level, the national level and on the institutional and individual level with reliable and valid information concerning training and education on the WWW.

The CISAER Project has no authority in certification of courses, and the project has no intention of comparing or ranking courses or judge identified courses as good or bad. The project wishes to describe in full the different aspects of WWW courses and how the WWW is applied for teaching and learning in different contexts. The main aim of the evaluation part of the project is to give examples of recommendable practice, which can support decision-makers, providers and users, the make to best use of the WWW for teaching and learning.

AECS. (1994): Quality Guide. AECS, 1994.

Birnbaum, R. (1989): The Quality Cube: how college presidents assess quality. Iin: Quality in the Academic: Proceeding from a national symposium. National Center for Postsecondary Governance and Finance. University of Maryland.

Boshier, R., Mohapi, M., Moulton, G., Qayyum, A., Sadownik, L. & Wilson, M. (1997): Best and worst dressed web courses: Strutting into the 21^st century in comfort and style. Dist. Ed. 1997, 2, pp. 327-349. Toowoomba, University of Southern Queensland.

Campbell, D. T. (1976): Assessing the Impact of Planned Social Change. Occasional Paper Series 8. Evaluation Center, College of Education, Western Michigan University.

Cronbach, L. J. (with the assistance of Shapiro, K)(1982): Designing evaluations of educational and social programs. San Francisco, Jossey-Bass.

Cronbach, L. J. (1983): Course improvement through evaluation. In: Madhaus, G. F., Scriven, M. & Stufflebeam, D. L. (eds.): Evaluation models. Viewpoints on educatinal and human social services evaluation. Boston/The Hague, Kluwer.Nijhoff, pp. 101-115.

Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F. & Stufflebeam, D. L. (eds.)(1980): Toward reorm of program evaluation. San Francisco, Jossey-Bass.

Davis, E. (1981): Teachers as curriculum evaluators. London, Allen & Unwin.

Fagerberg, T., Hellem, R. & Scotto, M. (2000): An Analysis of Courses on the Web. Diploma Project. Bekkestua, NITH.

Franke-Wikberg, S. (1992) Quality in Higher Education: An International Issue under Debate. Paper to the EADTU Conference Quality, Standards and Research in European Distance Education. Umeå, 1992.

Franklin, J. L. & Trascher, J. H. (1976): An introduction to programme evaluation. New York, Wiley.

Holmberg, B. (1995): Theory and Practice of Distance Education. 2^nd. ed. London, Routledge.

House, E. R. (1980): Evaluating with validity. Beverly Hills: Sage.

Juran, J. M. 'Quality Control Handbook.' N. Y., McGrawHill, 1988.

Keegan, D: (1990): Foundations of Distance Education. 2nd. ed. London, Routledge.

Keegan, D: (1999): Seamless interfaces: distance education and web-based training. Istruzione a distanza, 1999, 1. Università di Roma III (in print).

Ljoså, E. & Rekkedal, T. (1993): From external control to internal quality assurance. Background for the development of NADE’s Quality Standards for Distance Education. Oslo, NADE.

MacDonald, B. (1973): Educational evaluation of the national development programme in computer assisted learning. Proposal prepared for consideration of the Programme Committee of the National Programme.

Madhaus, G. F., Scriven, M. & Stufflebeam, D. L. (eds.)(1983): Evaluation models. Viewpoints on educatinal and human social services evaluation. Boston/The Hague, Kluwer.Nijhoff, pp. 101-115.

Mason, R. (1991): Moderating educational computer conferencing. DEOSNEWS 1 (19).

Mason, R. (2000): Use of Web for Higher Education: Is Europe a Serious Player? CISAER Project Paper.

McCormick, R. & James, M. (1989): Curriculum evaluation in schools. 2^nd. Ed. London. Routledge.

NADE (1993): Quality Standards. Oslo, NADE.

Norris, N (1990): Understanding Educational Evaluation. Londan, Kogan Page.

Partlett, M. & Hamilton, D (1972): Evaluation as Illumination: A New Approach to the Study of Innovatory Programmes. Occasional paper 9. Centre for Research in Educational Sciences. University of Edinburgh. Also published in: Hamilton, D., Jenkins, D., King, C., MacDonald,

B. & Parlett, M. (eds.)(1977): Beyond the numbers game: a reader in

educational evaluation. Basingstoke, Macmillan) ch.1.1 pp.6-22.

Patton, M. Q. (1990): Qualitative evaluation and research methods. 2^nd ed. Newbury Park, Sage.

Paulsen, M. F. (1997): Teaching Methods and Techniques for Computer Mediated Communication. http://www.nettskolen.com/fag_art/22/icdepenn.htm.

Paulsen, M. F. (1998a): Teaching Techniques for Computer-Mediated Communication. Ph.d. Thesis. Penn. State. University.

Paulsen, M. F. (1998b): Definitions. Internal CISAER Working Document.

Paulsen, M. F. (2000): An Analysis of Institutions that Offer Courses on the Web. Project Paper CISAER.

Rathore, H. & Schuemer, R. (eds.)(1998): Evaluation Concepts and Practice in Selected Distance Education Institutions. ZIFF-Papiere 108. Hagen, FernUniversität.

Rekkedal, T. (1983): The written assignments in correspondence education. Effects of reducing turn-around time. An experimental study. Distance Education, 1983:2, 231-251.

Rekkedal, T. (1985): Introducing the personal tutor/counsellor in distance education. NKI, Stabekk.

Rekkedal, T. (1989): The telephone as a medium for instruction and guidance in distance education. NKI, Bekkestua.

Rekkedal, T (1998): Quality Assessment and Evaluation. Basic Philosophies, Concepts and Practices at NKI, Norway. In: Rathore, H. & Schuemer, R. (eds.), pp. 39-65. http://www.nettskolen.com/fag_art/34/quality.html

Rossi, P. H. & Freeman, H. E. (1989): Evaluation. A systematic approach. 4^th. Ec. Newbury Park, Sage.

Ruthman, L. (1980): Planning useful evaluations. Evaluability assessment. Beverly Hills, CA, Sage.

SATURN (1992): SATURN Quality Guide for Open and Distance Learning. Pilot guide.

Schuemer, R. (ed.) (1991): Evaluation concepts and practice in selected distance education institutions. FernUniversität, ZIFF.

Scriven, M. (1967): The Methology of Evaluation. AERA Monograph Series on Curriculum Evaluation, No 1. Chicago, Rand Mcnally.

Scriven, M. (1975): Evaluation bias and its control. Occasional papers series, 4. Evaluation Center, College of Education, Western Michigan University, Kalamazoo.

Stake, R. E. (1967): The countenance of educational evaluation. Teachers College Record, 68(7), pp. 523-540.

Stake, R. E. (1975): Program evaluation, particularly responsive evaluation. Occasional Papers Series, 5. Western Michigan University, College of Education.

Stake, R. E. (1976): Evaluating Educational Programmes. The Need and the Response. Paris, Centre for Educational Research and Innovation (CERI).

Stake, R. E. (1983): Program evaluation, particularly responsive evaluation. In: Madhaus, G. F., Scriven, M. & Stufflebeam, D. L. (eds.): Evaluation models. Viewpoints on educatinal and human social services evaluation. Boston/The Hague, Kluwer.Nijhoff, pp. 101-115.

Stenhouse, L. (1975): An Introduction to Curriculum Research and Development. London, Heineman.

Stufflebeam, D. L. (1974): Meta-Evaluation. Occasional Paper Series 3. Evaluation Center, College of Education, Western Michigan University.

Stufflebeam, D. L., Foley, W. J., Gephart, W. J., Guba, E. G., Hammond, R. I., Merriman, H. O. & Provus, M. M. (1971): Educational evaluation and decision making. Itasca, IL: Peacock.

Suchman, E. A. (1967): Evaluation research: Principle and practice in public service and social action programs. New York, Russel Sage Foundation.

Thompson, M. S. (1975): Evaluation for decision in social program. Westmead, UK, Saxon House.

Thorpe, M. (1988): Evaluating open & distance learning. London, Longman.

Tyler, R. (1949): Basic Principles of Curriculum and Instruction. Chicago, University of Chicago Press.

Van Vucht, F. A. & Westerheijden, D. F. (1993): Quality management and quality assurance in European higher education. Methods and mechanisms. Brussels, Commission of the European Communities.

Wentling, T. L. & Lawson, T. E. (1975): Evaluating occupational education and training programs. Boston, MA, Allyn & Bacon.

Wilson, M., Qayyum, A. & Boshier, R. (1998): World Wide America? Think globally, Click locally. Dist. Ed. 1998, 1, pp. 109-123. Toowoomba, University of Southern Queensland.

Wittman, W. W. (1985) Evaluationsforschung – Aufgaben, Probleme und Anwendungen. Berlin, Springer.