Global Outreach: BETTER POLICYMAKING THROUGH EVALUATIONS – Interim Report
Parliamentarians have an important role to play in the field of evaluation: To use evaluations for their decision making, commission evaluations themselves, and to assign an important role to evaluation within the framework of their legislative competence.
In order to be able to use evaluations appropriately, they must know the value of evaluations and potential uses of evaluations for evidence-based policy making.
The International Program for Development Evaluation Training (IPDET) together with the Asia Pacific Evaluation Association (APEA), and the Global Parliamentarians Forum for Evaluation (GPFE) have organized a virtual training on evaluation for Parliamentarians and Parliament (Research) Staff from the Asia Pacific Region. This was the very first training offered as part of the IPDET Global Outreach-strategy.
The objective of this training was to impart more in-depth evaluation knowledge ensuring evaluative evidence in the policy making process.
“This workshop series will be an opportunity for you to get a better understanding of why evaluations are important for good governance and why and how evaluations can help you make policies stronger and public institutions more accountable in your countries.” – Alison Evans, IEG
- Parliamentarians from the APEA-member countries
- (Research) staff of parliamentarians and parliaments from the APEA-member countries
- Further interested people in the topic (e. g. VOPE leaders)
- Six online events in October and November 2021
- One on-site event in 2022 in an APEA member country
The Asia Pacific Evaluation Association (APEA) is the regional evaluation association in Asia Pacific, that aims to improve the theory, practice and use of evaluation across the Asia-Pacific region. It advocates building of capacity among individual evaluators, institutions and organizations representing Asia Pacific countries – as it advances the principles of transparency, accountability, and learning toward a just and progressive society.
“This is just a start. (…) the idea is to have a face-to-face training with you all next year, discussing about more technical aspects of evaluation and SDGs (…) At the same time, use of evaluations and evaluative evidence for decision making will be a great example for many parliaments in the Asia Pacific region as well as other regions.” – Asela Kalugampitiya, APEA
Global Parliamentarians Forum for Evaluation is a collaborative movement of international parliamentarians, committed to improving policy outcomes and social accountability, that advocates for evidence-based decision making. They know that evaluation can enable governments and parliaments to efficiently and effectively address national interests, and encourage policy makers and evaluation professionals to work together to build evaluation capacities around the world.
“This is the beginning of a new product, this is what we have always dreamt about getting parliaments properly engaged because they finally would be the people who would be able to make policies, that would use evidence-based decision-making.” – Kabir Hashim, GPFE
Session 1: Use of evaluations for political steering and participation of society
18th October 2021
The focus of the first session was to what extent evaluations can support the work of Parliamentarians and Parliament (Research) Staff and what makes high quality evaluations. In his short input Reinhard Stockmann raised the attention of the audience to the main roles of parliamentarians: Making Laws & Policies, Approving Budgets, and Oversight & Accountability. To fulfil these tasks, Parliamentarians and Parliament (Research) Staff are highly reliant on objective and evidence-based information to strengthen their arguments in political debates and work. This can be obtained through evaluations – but not all evaluations are the same! Therefore, criteria were presented to distinguish between bad and good evaluations.To illustrate how and to what extent evaluation is institutionalized in different countries experts from Uganda, Sri Lanka and Kyrgyz Republic were invited to a panel discussion, moderated by Reinhard Stockmann: Whereas Uganda already counts with a national evaluation policy, which also clearly defines the role of the parliament, Sri Lanka has a regulatory system for evaluation for more than 20 years, however, this is – still – not accompanied with a national evaluation policy and therefore only some parliamentarians are aware of evaluation – especially as trainings on evaluation is missing. Eventually, Kyrgyz Republic adopted a law on M&E in 2014, and also built and implemented a strategy for evaluation for state programs and laws, to further raise awareness for and knowledge about evaluation among parliamentarians.
“I think, the role that parliamentarians play in contemporary society is a really important one. They are not only key actors in the policy making process, but they are enormously influential people in relation to how they help shape culture and the ways in which evidence gets used.” – Dugan Fraser, GEI
“It is our mission of parliament keep the government accountable and efficient.” – Natalia Nikitenko, GPFE
Session 2: Introduction to fundamentals of M&E
26th – 28th October 2021
Based on the introductory session on 18 October the event series continued, to give a further as well as deeper insight into the fundamentals of M&E. The sessions were guided by Reinhard Stockmann and consisted of several theoretical inputs, Q&A-sessions and intensive group works, in which participants were able to deepen their new knowledge on project case studies. To guarantee a denser learning atmosphere, these sessions were open to a selected group of participants only.
In the first meeting of this session the learning group dedicated to M&E basics, which encompassed questions on the definition of the evaluation object, analytical perspectives of evaluation, but also for which purpose, by whom and how an evaluation is to be conducted. Further to this, the second meeting focused on the planning of evaluations. Participants were able to learn about implementing actors, design, methods but also financial resources needed, to actually implement evaluations. Finally, in the third meeting, the participants learnt about standards, results and utility. Standards for instance ensure and develop the quality of evaluations and with it also their final results and practical utility – the latter is from special importance, as utility is one of the core values of evaluations!
“Policy decisions should be based on objective evidence and facts, and no longer on common sense, ideology, tradition, religious beliefs etc. Political goals are best served, when valid and reliable evidence is used comprehensively to inform decisions.“ (…) Parliamentarians who know how to use such evaluation results can convince others with arguments, that are perceived as objective and credible.” – Reinhard Stockmann, CEval
In a very final group work, participants were asked to formulate concrete evaluation questions, taking into account their own and individual case study, the benefits of evaluation in the different program phases but also all other aspects of the content taught previously. All this with the objective, to train and internalize one of the most important tasks for Parliamentarians and Parliament (Research) Staff with view on evaluation: how to formulate adequate evaluation questions, and therefore, how to ‘think evaluative’.
Session 3: Challenges in evaluating SDGs
2nd November 2021
What are the conceptual challenges, in case progress in implementing the SDGs is not only to be monitored but also to be evaluated? In his theoretical input, Wolfgang Meyer draw the attention to three main challenges in the framework of sustainable development – System Integration, System Change and Inclusiveness – and how National Monitoring and Evaluation Systems might accompany the resolution of these challenges.
“We have to recognize that all data is local, so you have to collect all kind of data on a local level, because it is the way it happens. Therefore, we have always the problem to aggregate this information towards a national level. (…) Investing in the abilities for M&E on the local level means also investing in improving the quality of the national statistics.” – Wolfgang Meyer, CEval
Besides that, the participants were able to get deeper insights into country experiences. In Sri Lanka for example, after having three different bodies to take SDGs into account for development programmes – an own ministry, parliamentary committee, as well as a president task force – an independent parliament evaluation commission was established to coordinate all SDG connected development activities.
In the Philippines, M&E activities had been very fragmented, conducted in different government agencies without being connected to each other, and there were no evaluations standards or criteria on the use of evaluations. However, in 2015, when the SDGs were adopted, a memorandum on adopting a national evaluation policy framework was implemented to oversee and coordinate the implementation of the SDGs as well as to mandate all national government agencies to formulate an evaluation agenda.
“The sectoral view allowed us to understand the contribution of each sector to the national priorities as well as the SDG commitments. (…) The multi stakeholder partnerships allow all the actors to leverage their strengths and supplement capacities while working towards the desired goals.” – Shweta Sharma, NITI Aayog
India counts with two main monitoring mechanisms to assess the implementation status of SDGs: One is the SDG India index focusing on periodic monitoring, the other is the Voluntary National Review (VNR), conducted in 2017 and 2020. But they also face some central challenges: limited capacity in the field of M&E, availability of high-quality data, and inclusion of marginalized communities.
Costa Rica was the first country in the world which admitted itself to the SDGs. To assure that SDGs are embedded in planning instruments, a multisectoral and multistakeholder governance system, composed by a high-level council, which is led by the president of the republic and coordinated by the Technical Secretarial of the National Commission of the SDGs, was implemented. Additionally, an advisory committee exists to accompany this process. A further milestone in this process was the implementation of the SDGs into the national development and public investment plan. Important is also, that SDGs are a base criterion for public interventions in the national evaluation agenda.
“One of the most complex and most ambitious things we want to do is to preserve the spirit of an integrated development approach and how that reflects upon the evaluations we do and also our monitoring instruments.” – María del Pilar Garrido Gonzalo, Goverment of the Republic of Costa Rica
Session 4: Closing ceremony
11th November 2021
The event series ended with a closing ceremony, in which the following key take-home-messages were formulated by Reinhard Stockmann:
The information, facts and evidence obtained from evaluations provide Parliamentarians with evidence-based arguments, that they can use for:
Law & Policy Making, to
|
Oversight, to
|
Approving Budget, to
|
Overall, to
- place their own political decisions on a rational more objective, less vulnerable, basis;
- present themselves as well-informed politicians who competently represent the interests of their voters and should therefore be re-elected!
We as IPDET, APEA and GPFE are convinced of the fact that evaluation enables a better world through better policy.
Therefore, personal capacity building in the use of evaluation is necessary to generate evidence for policy making within parliaments and its staff.
In addition to these take-home-messages of the event, 63 participants from 15 countries were officially awarded for taking part in this event series. Furthermore, the participants had the opportunity to establish contacts with each other in a virtual speed networking session.
“As countries recover from the aftermath of the Covid-19 crisis, parliaments and parliamentarians play a key role in shaping and delivering evidence-based policies that can turn around the economic and social effects of the pandemic and can propel the country towards full implementation of the SDGs.” – Marco Segone, UNFPA
INSIGHTS FROM OUR PARTICIPANTS
Why did you decide to participate in the training?
“Nothing is possible without proper planning and evaluation. In Pakistan, there is no proper course on Parliamentary Systems and evaluation. Evaluation is only a small topic in certain academic courses in Pakistan. When I heard about the IPDET training, I went to the Speaker of the Parliament and convinced him that our staff and members should take part in the training. I also wanted to understand the technicalities of evaluation.” - Attaullah Khan, Participant from Pakistan, Parliament Staff
What is the main challenge in your country regarding evaluation?
“Policymakers, administrators, and government officials have no clear understanding on monitoring and evaluation. They think evaluation is like an audit and they do not understand that evaluation has a process.” - Mylvaganam Thilakarajah, Participant from Sri Lanka, Parliamentarian “In my country we face many difficulties with evaluation. (…) One is, it is a new discipline. Because of that, we do not have enough evaluators. Another thing is we give less priority to evaluation. Many of the projects and programmes in our country start and process but do not pay attention to the output. That is the main challenge. Evaluation is not merely a concept. It has a process. We should train our experts on how to evaluate projects and programmes.” - PLT Erandathi, Participant from Sri Lanka, Parliament Staff
What is your main take-away from the training?
“It was very enlightening and encouraging to see the immense evaluation work carried out by other Parliaments such as the Parliament of Sri Lanka and getting insights through presentations, group work and other country examples on how evaluation is undertaken and using the best practices from these to see how it can be applied to my work as a Parliament Researcher providing research support to the Standing Committee on Public Accounts.” - Participant from Fiji
What will you apply from the training in your work?
“Luckily while we were going through the training at the same duration in our committee we were planning for monitoring and evaluation of few program and projects running under government of Gandaki Province. So, I got the chance to edit our checklist and according to checklist we went through a lot information collection. We are still working on it. My group has completed 15 programs out of 61 programs monitoring and evaluation in 45 days. We are still working on it. Thanks to IPDET Training, my report seems a lot more precise now.” - Chandrakala Basnet, Participant from Nepal, Parliament Staff
Workshop insights and Feedback by Ayesha Godgama, Participant from Sri Lanka, Parliament Staff
LESSONS LEARNT
- The willingness to deal with the topic of evaluative evidence in policy-making and to apply it in the policy making process among Parliamentarians and Parliament (Research) Staff is enormous – at least in the Asian/Pacific region. There was strong active participation in working groups, and continuous participation in all workshops of the series.
- The willingness to deal with the topic of evaluation can be seen as an indicator that parliamentarians want to change their political style and put their decisions on a rational basis. Evaluation can be used as an important tool for this.
- The feedback of participants shows, that the content of the event series gives them tools and guidance working more efficient and effective in fields like analyzing (progress) reports of projects/programs/bills, budget research, drafting of and/or final decision-making on new laws.
- This indicates, that the goal to convince and equip participants with knowledge and skills to use evaluative evidence in the policy-making process was highly achieved.
- The overwhelming interest spurs IPDET, APEA and GPFE to continue this event series in Asia/Pacific in 2022. Furthermore, the topic shall also be offered in other world regions, as we are convinced that a better world is possible through better (evidence-based) policy-making.
LOOKING AHEAD
IPDET Global outreach activities will further focus on trainings for Parliamentarians and Parliament (Research) Staff:
- For Asia-Pacific the training will be continued and further developed in cooperation with APEA and GPFE:
- The goal is a broadening and expansion of the topics, from mostly awareness raising to competent application and use. Contents shall focus especially on practice-orientated aspects of Parliamentarians and Parliament (Research) Staff work.
- To gain an even greater impact and to guarantee better networking and interaction opportunities, will be done in an on-site training in an Asia-Pacific country.
- To further raise awareness about evaluation among Parliamentarians and Parliament (Research) Staff worldwide, this training series should be also delivered in other regions. This goes hand-in-hand with an expansion to other implementing partners.
- It is also important to focus on the further mainstreaming of M&E in overlying parliamentary structures, which goes beyond solely individual trainings. It is of utmost importance to show ways, how participants can not only apply their new knowledge within their daily work, but also how they can create evaluative structures within their countries.
From special importance was the cooperation with APEA and GPFE as regional and thematical partners. This kind of cooperation proved its worth in organizing and delivering the event series and should serve as a best-practice example applied for further IPDET Global Outreach trainings.
IMPLEMENTATION TEAM
Prof. Dr. Reinhard Stockmann:
Founder and Director of Center of Evaluation (CEval) at Saarland University and Member of the Steering Group of the International Program for Development Evaluation Training (IPDET) | Conceptual head of the development of the event series | Instructor Session 1 & 2, Speaker Session 4
Laszlo Szentmarjay:
IPDET Global Outreach coordinator | Tutor Session 2 | Overall coordination and implementation
Dr. Asela Kalugampitiya:
President of Sri Lanka Evaluation Association and Asia Pacific Evaluation Association (APEA)| Speaker Session 1 & 4 | Tutor Session 2 | Support to coordination and implementation
Hasithi Samarasinghe:
Co-Leader EvalYouth Sri Lanka | Tutor Session 2 | Support to coordination and implementation
Randika de Mel:
Co-Leader EvalYouth Asia and EvalYouth Sri Lanka | Tutor Session 2 | Support to coordination and implementation
CONTRIBUTORS
Dr. Alison Evans:
Director General Evaluation, Vice President, Independent Evaluation Group (IEG), World Bank Group | Speaker Session 1
Dugan Fraser:
Program Manager of Global Evaluation Initiative (GEI) | Speaker Session 4
Josephine Watera:
Head of M&E Unit, Parliament of Uganda | Panelist Session 1
Kabir Hashim:
Member of Parliament of Sri Lanka and Co- Founder and Chair of the Global Parliamentarians Forum for Evaluation (GPFE) | Speaker Session 1 & 4, Panelist Session 1
Marco Segone:
Director Evaluation Office, United Nations Population Fund (UNFPA) | Speaker Session 4
María del Pilar Garrido Gonzalo:
Minister of Planning and Economic Policy, Republic of Costa Rica, and Coordinator of the Technical Secretarial of the National Commission of the Sustainable Development Goals (SDGs) | Country expert Session 3
Mylvaganam Thilakarajah:
Treasurer of the Sri Lanka Parliamentarians Forum for Evaluation (SLFPE) and former member of Parliament of Sri Lanka | Panelist Session 3
Natalia Nikitenko:
Member of Parliament of Kyrgyzstan, Chair of the Eurasian Parliamentarians Forum for Evaluation and member of the Global Parliamentarians Forum (GPFE) Steering Committee | Panelist Session 1
Dr. Romulo E.M. Miral, Jr.:
Director General of the Congressional Policy and Budget Research Department, House of Representatives of the Philippines | Panelist Session 3
Dr. Shweta Sharma:
Development, Monitoring and Evaluation Office (DMEO) at NITI Aayog, Govt. of India | Country expert Session 3
Dr. Stefanie Krapp:
Head of Program – International Program for Development Evaluation Training (IPDET) | Speaker Session 1 & 4
Prof. Wolfgang Meyer:
Professor for Sociology at Saarland University and Member of the Steering Group of the International Program for Development Evaluation Training (IPDET) | Instructor Session 3