Do evaluations always provide the ‘right’ information? I mean, are they valid, reliable, correct in a sense? Can we believe their findings? And after all, what is the merit of our work to society, or at least for the people who read our reports?
Not only evaluators contemplate about such questions from time to time. More and more commissioners and political decision-makers want to know if they can trust evaluations and what they can actually learn from them. Moreover, the added value of such exercises for providing the empirical basis for decision making remain often obscure. Hence it is no wonder that the demand for meta-evaluations rises.
We all know what evaluations are good for and how we should implement them, don’t we? There are numerous textbooks, training programs, podcasts, which provide methodological and practical guidance. Complying with such standards and following scientific codes of conduct should warrant valid, reliable and useful findings. – But can we prove it? How do we actually find out about the methodological quality of an evaluation, the validity and reliability of its findings, and eventually the usefulness of its conclusions and recommendations? Basically, how can we distinguish a good one from a bad one?
These are exactly the questions we want to answer in this workshop. We discuss criteria that help us decide if we can trust an evaluation, how we can assess its methodological quality, check if it was appropriately prepared and implemented, and eventually of use. We further jointly elaborate a meta-evaluation approach from the analysis grid to suitable methods for content analysis. Going beyond the evaluation report, we dive into the potentials of meta-evaluation as well as its limitations.
So, if you want to know how to evaluate an evaluation, come in and find out. Bring along – actually please send it a few days in advance – an evaluation you always wanted to know how good it is. I promise you, afterwards you will know (better).
- are able to assess the quality of an evaluation according to common standards,
- know which criteria should be used therefore and how they can be applied.
- They evaluate evaluations in practice and thereby learn about the added value of a systematic approach to it.
- Finally, the participants will gather an overview about practical tools and applications for meta-evaluations.
This workshop is particularly relevant for commissioners of evaluations who want to learn about the quality of the evaluations they receive or who want to develop their own quality assurance framework for evaluations. It is, however, likewise interesting for evaluation professionals who want to learn according to which quality standards their work may be assessed if it becomes subject to a meta-evaluation. Eventually, the workshop may be suitable to everyone who reads evaluation reports and contemplates about their trustworthiness.
This is an intermediate level workshop.
Participant need to have a basic understanding about evaluation as a management tool. While it is not necessary to have conducted an evaluation, ideally, the participants should have read a number of evaluation reports before and know about their strengths and weaknesses.
- Introduction into Meta-Evaluation
- What makes a good evaluation? – Quality criteria for evaluations
- Participants’ presentations of example evaluations
Outside class: Reflecting own’s evaluation quality under consideration of quality criteria
- Developing an analysis grid for a meta-evaluation
- Practical group work: development of an analysis grid
Outside class: Drafting a recipe for a good meta-evaluation
- Presentation of group work results
- Finalization of an exemplary meta-evaluation analysis grid
- Small-team work: Application of the analysis grid
Outside class: Meta-evaluating two evaluations
- Presentation and discussion of meta-evaluations
- Tools and methods for analyzing meta-evaluation results
Outside class: Analyzing meta-evaluation results
- Aggregation of individual meta-evaluation and results synthesis
- Findings from meta-evaluations conducted at CEval
- Wrap-up and feedback