Module 4 – Evaluation

Module 3 introduced you to the design and development phases. Evaluation plays an important cyclical and central role in the design process , as indicated in the diagram below. The implementation phase is not specifically addressed in this course but you are encouraged to later implement your design and perform an evaluation that you plan in the assessment for this course.

ADDIE model

Learning outcomes

On completion of this module, you should be able to:

  • describe evaluation in flexible learning settings;
  • discuss the process of educational evaluation as it relates to flexible learning situations, the literature, and the learners’ own contexts and experiences;
  • demonstrate an understanding of the principles and procedures of conducting an evaluation of an educational program; and
  • compile a brief evaluation plan for a selected program in a flexible learning setting.

What is educational evaluation?

Evaluation may serve different purposes including measuring the worth of sound educational activity and facilitating decisions about educational activity. Worthen and Sanders (1987, p. 22) argue that:

Evaluation is the determination of a thing’s value. In education, it is the formal determination of the quality, effectiveness, or value of a program, product, project, process, objective, or curriculum.

An alternative view of educational evaluation places the emphasis on the techniques used in collecting data that may be used in determining worth. Such definitions usually indicate that evaluation is concerned with the administration of formal tests for determining the effectiveness of a particular educational activity, or, alternatively the observation of behaviour to provide useful information on the effectiveness of an educational activity.

Program evaluation is a complex process. Guba and Stufflebeam (1970) have contributed to much of the current thought on program evaluation by observing that evaluation should be ongoing and not aimed only at outcomes, or the degree to which objectives have been met. They see evaluation as a process which demands a systematic approach; it involves obtaining useful information; it describes and then requires the judging of worth or value; and the overall purpose is to serve some form of decision making. They believe that educational evaluation should aim at improvement.

Reading

Trochim (2006) briefly outlines his definition of evaluation, the goals, strategies and types of evaluation as well as some questions and methods.

Trochim, W. M. K. (2006). Introduction to evaluation. Retrieved from http://www.socialresearchmethods.net/kb/intreval.php

Links on the site offer access to The Planning-Evaluation Cycle and An evaluation culture. Explore the site for ideas that may be useful in your project.

Evaluation of flexible learning contexts

Flexible learning contexts provide particular challenges for evaluating program effectiveness. Morgan and O’Reilly (1999) indicate that while there are similarities between evaluation in traditional settings and those of flexible learning, there are several issues that distinguish the latter.

Morgan and O’Reilly (1999, pp. 98-9) suggest that:

The similarities exist in both the processes (goal setting, design, data collection, analysis) and outcomes (the quality of learning, effectiveness of materials and services, suitability of courses for learners’ needs).

They maintain the main differences include:

  • Flexible learning tends to be less open to inspection compared to face-to-face settings, which can be observed and evaluated directly.
  • There is more to evaluate, due to the team effort entailed in the development and provision of flexible learning. There is also a wider variety of options and pathways for learners to go about their learning. All need to be included in an evaluation.
  • There is a likely to be a greater range of learners in terms of age, reasons for studying, employment circumstances, competing commitments, and so forth. Students often have differing criteria for success, and this leads to more complex data regarding issues such as attrition, course completion, and approaches to learning and assessment.
  • Flexible learning is still an innovative approach to educational provision, particularly with respect to the use of new technologies.

However, there are some promising attempts to tackle the problems associated with the evaluation of technologies used in education and training programs, as is indicated in the following optional reading.

Models of evaluation

The validity of evaluation is potentially enhanced if it is based on some theoretical perspective. In the following discussion, three approaches to evaluation activity are briefly summarised. Each one of these approaches is derived from a particular perspective and as such has a unique focus on the evaluation process depending on the aim of the evaluation exercise and often the context in which the evaluation is to be conducted. Basic models and principles of evaluation can be used to develop a plan for evaluation in any learning setting.

Some approaches to evaluation activity are as follows:

  1. Decision-oriented evaluation. Proponents of this approach include Stufflebeam (1971). The focus of this perspective is comparing “what is” with “what should be”.
  2. Value-oriented evaluation. This approach has been also described as the “humanistic approach”. It stresses the value judgments made in evaluating programs (i.e., the merit or worth of programs). It is therefore, inherently humanistic in nature, focuses on the total effect of the program, product or process and is sensitive to initially unknown or unintended program effects. A notable exponent of this approach is Scriven (1993) who proposes the idea of “goal-free evaluation”.
  3. Naturalistic evaluation. This approach aims at discovery and verification through direct observation as in ethnography. It relies on the impressions of the evaluators located in situ who do not manipulate the conditions before or during their enquiry and pose minimal constraints on the behaviour of their participants.

Stufflebeam's CIPP Model

Analysis phase

Stufflebeam's (1971) CIPP model (Context, Input, Process, Product) provides a basic framework for program evaluation in any learning setting, be it face-to-face or flexible learning environments. The model was developed in an attempt to show how evaluation serves decision-making. The evaluator, working closely with administrators, identifies the decisions the administrator must make and then collects, organises and analyses information and reports the findings to allow judgments to be made. In many cases, the decision maker (through force of circumstances) will have to be their own evaluator.

Stufflebeam proposed four types of evaluation decisions that frequently confront the education decision maker: planning, structuring, implementing and recycling decisions. From these types of decisions, he derived various evaluation strategies. Planning decisions are served by context evaluation; input evaluation serves structuring or design decisions; implementing decisions are served by process evaluation, and recycling decisions by product evaluation. Stufflebeam’s model serves all levels of decision-making: the development of goals, the shaping of proposals, the implementation of projects, and the examination of their achievements (or failures).

Context evaluation

Context evaluation activities are mainly descriptive and comparative. A context evaluation would be devoted to collecting data on the requirements of the profession, the education system, the institution, and the learners, as they relate to an educational program. It would be conducted in order to decide upon program priorities and the choosing and assessing of goals – equating to the analysis phase of the learning design process. The evaluation would help to determine the worth of the program goals by matching them to the requirements of the relevant stakeholders. The context evaluation serves planning decisions and often results from pressure to meet changed circumstances in education and training.

Input evaluation

An input evaluation looks for creative ideas to meet needs and goals, looks at how problems can be solved (it may search relevant literature, question personnel in other institutions, call on the expertise of the available staff, seek outside advice, or compare the proposal with other learning programs already in use), and then evaluates these alternative proposals. It serves decisions necessary to specify which means are likely to be most effective in achieving a given set of goals or a set of assessed learning needs (structuring decisions) as established through a context evaluation. Questions such as what content do we need, how will this content be presented, structured and delivered to meet the needs of the learners, what are the production team requirements, what resources will be required, what learning strategies will be used, and what is the budget, could be asked in an input evaluation.

Process evaluation

A process evaluation calls for the evaluation of the process of implementing a program or project (thus serving implementation decisions) to provide periodic feedback to project facilitators and managers. This evaluation verifies whether all the decisions made as a result of the input evaluation have actually been implemented and enables the evaluator to establish whether those design and development features considered essential for a particular program have actually been incorporated or are as effective as the planners hoped they would be. In other words, it examines the extent to which people understand the adopted approach, the degree to which they are actually carrying it out and how effective that approach has been in practice. Is the program on schedule? Is it being carried out as planned? Has it been designed taking into account the specific characteristics of the learners? Are the available resources being used efficiently? A process evaluation should provide an extensive record of the program that was implemented. The overall strategies in a process evaluation are that of identifying and monitoring.

Product evaluation

A product evaluation records project attainments (Was the project successful?) and serves decisions concerning the continuation, modification, or termination of the project (recycling decisions). This is a particularly important stage in the evaluation process as funding issues are of critical consideration within most organisations. For example, if a flexible learning program is perceived to be not ‘paying its way’, this evaluation may be used to facilitate decision making about its future. Results, both positive and negative, are observed. This evaluation focuses on the extent to which the requirements of the learners have been met and how the various stakeholders judge the worth and merit of the outcomes. Learners might be asked to indicate which parts of the course they have studied, which components they have used and how much time they have spent on their studies. They may also be asked to report on any problems they have encountered and rate individual components of a program in terms of their relative usefulness. Interpreting such responses, of course, must take into account the subjective nature of students’ answers.

Data collected for a product evaluation of a program generally focuses on an examination of student assessment to determine student performance – this is certainly an important factor in determining if a project has been successful. Other data that can be collected include ratings of the program by teachers, learners, employers, etc. and the general impressions of the evaluators. Studies aimed at measuring the subjective benefits experienced by individuals and the extent to which the qualification is recognised by other educational institutions and professional bodies can form part of a product evaluation. The recognition of the qualification can also be approached from the other direction through a survey of employers to establish the standing of the qualification and the acceptability of the graduates.

An excellent website, The Evaluation Center, has been created by Western Michigan University Evaluation Center in the USA under the leadership of Daniel Stufflebeam and Michael Scriven. This site includes a variety of evaluation checklists, guidelines for checklist development, and an in-depth examination of the checklist approach to evaluation for those interested in the theoretical and methodological underpinnings of checklists. In the checklists devoted to the CIPP model of evaluation Product Evaluation is broken into the four components of:

  • impact (assessing a program's reach to the target audience);
  • effectiveness (assessing the quality and significance of outcomes);
  • transportability (assessing the extent to which a program has (or could be) successfully adapted and applied elsewhere); and
  • sustainability (assessing the extent to which a program's contributions are successfully institutionalised and continued over time).

Kirkpatrick's Four Level Evaluation Model

One of the best-known evaluation models in industry, workplace and vocational education and training settings is the Four Level Evaluation model (Kirkpatrick, 1994), which looks at evaluation from the perspectives of:

  • Level 1: Student reaction to the course or program (the student's satisfaction with their learning experience)
  • Level 2: Student achievement of stated learning outcomes (normally judged via a review of students' achievement on summative assessment tasks)
  • Level 3: Behaviour change as a result of learning (has the learning achieved by students actually been put into practice? Has performance been improved or enhanced?)
  • Level 4: Have the behaviour changes achieved by learners through completion of the learning program resulted in benefits to the organisation? What results has the investment in training brought the organisation?

Reading

Read this overview of the Four Level Evaluation Model

Clark, D. (2012). Kirkpatrick's Four Level Evaluation Model. Retrieved from http://www.nwlink.com/~donclark/hrd/isd/kirkpatrick.html

Constructivist, Participatory and Formative Evaluation Models

In contrast to more traditional models of evaluation which focus on an external, objective evaluator making judgments after the event about the extent to which particular goals and objectives have been achieved as a result of an educational intervention, constructivist, fourth generation and empowerment evaluation models (see Fetterman, 1998; 2002; Guba & Lincoln, 1989; 2001) adopt a participatory, process evaluation approach where the evaluator acts as a facilitator and coach to support others to conduct evaluation of their own project, so that the evaluation becomes an educative process in itself, with an emphasis on formative (as distinct from purely summative) evaluation. These kinds of models tend to be used in the evaluation of community education programs and community development projects, and are often criticised as not being 'objective'.

References

Clark, D. (2012). Kirkpatrick's Four Level Evaluation Model. Retrieved from http://www.nwlink.com/~donclark/hrd/isd/kirkpatrick.html

Fetterman, D. (1998). Empowerment evaluation and the internet: A synergistic relationship. Current Issues in Education 1(4).

Fetterman, D. (2002). Empowerment evaluation: Building communities of practice and a culture of learning. American Journal of Community Psychology 30(1), (89-102)

Guba, E., & Lincoln, Y. (1989). Fourth Generation Evaluation. Thousand Oaks, CA: Sage Publications.

Guba, E., & Lincoln, Y. (2001). Guidelines and Checklist for Constructivist (aka Fourth Generation) Evaluation. Retrieved from http://www.wmich.edu/evalctr/archive_checklists/constructivisteval.pdf

Guba, E.G., & Stufflebeam, D.L. (1970). Evaluation: The process of stimulating, aiding and abetting insightful action. Bloomington, IN: Indiana University.

Jonassen, D.H. (1993). The trouble with learning environments. Educational Technology, Jan., 35–37.

Kirkpatrick, D. L. (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.

Morgan, C., & O’Reilly, M. (1999). Assessing open & distance learners. London: Kogan.

Scriven, M. (1993). Hard won lessons in program evaluation. San Francisco: Jossey-Bass Publishers.

Stufflebeam, D.L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education, 5(1), 19–25.

Trochim, W. M. K. (2006). Introduction to evaluation. Retrieved from http://www.socialresearchmethods.net/kb/intreval.php

Worthen, B.R., & Sanders, J.R. (1987). Educational evaluation: Alternative approaches and practical guidelines. New York: Longman.