Activity: Better policy making through Evaluations
Overall relevance of the training series
The shift in policy towards evidence-based rational governance, that has accompanied the development of modern societies, has also led to an increased importance of evaluation. Evaluation has become a central instrument of policy processes.
This change affects not only the executive, but also the legislature. With their workforce of democratically elected representatives of citizenry, parliaments represent one of the central pillars of democratic process and are rightfully identified as playing a crucial role in driving and sustaining equitable development within the state. The support of growing demand for, and use of evaluation-evidence within parliamentary oversight systems is therefore an intervention that has the potential to benefit both poor and previously disadvantaged groups through strengthening political representation and building government’s accountability to citizens.
Parliamentarians can use results of evaluations in a number of ways: Parliamentarians who know how to use evaluation evidence can influence the budget-approving process, can play an active role in parliament debate, better justify their decisions to the public and strengthen their oversight role. Furthermore, using evaluations can help them to fulfil their constitutional duties. At large, parliamentarians around the world have a key role to play to ensure that national evaluation cultures and policies are developed.
In order to be able to use evaluations appropriately, parliamentarians must know the value of evaluations and potential uses of evaluations for evidence-based policy making. To impart this knowledge was the aim of this training series.
Watch the interview with Prof. Dr. Reinhard Stockmann:
- Parliamentarians from the APEA-member countries
- Research staff of parliaments from the APEA-member countries
- Further interested people in the topic (e. g. VOPE leaders)
- Six online events in 2021
- On-site event between 27-30 March 2023 in Colombo/Sri Lanka
Background of the 2023’ training
Following the completion of the virtual workshop series in October & November 2021, IPDET, within its IPDET Global Outreach Strategy, in partnership with the Asia Pacific Evaluation Association (APEA), Global Parliamentarians Forum for Evaluation (GPFE), and Asia Pacific Parliamentarians Forum for Evaluation (APPFE), hosted the first international on-site training on evaluation for parliamentarians and parliament staff from Asia Pacific from 27-30 March 2023 in Colombo, Sri Lanka.
The four-day training was conceived in recognition of the increasing use of evaluations not only as a key administrative tool for the executive branch, but also as a fundamental instrument of policy processes. As such, it aimed to provide participants with essential knowledge and tools to use evaluative evidence to play an active role in parliamentary debates, better justify decisions to the public, strengthen oversight roles, and fulfill constitutional duties.
Overall summary of the 2023’ training
The 46 participants from 18 primarily Asia Pacific countries discussed with great enthusiasm the opportunities that evaluation offers in policy-making. The participants showed overwhelming commitment in solving work tasks, analyzing practical case studies and conducting a simulated parliamentary debate.
The results of the final questionnaire support this positive feeling with evidence: Without exception, all dimensions examined achieved extraordinarily good results, some of them even the best ever achieved by an IPDET training, and certainly unprecedented by comparable M&E trainings of other organizers.
It became clear that not only did the training develop an understanding of the benefits of evaluation for parliamentary work, but also the firm will to use evaluations in the real political practice.
In particular, looking ahead, we are confident that the trainings conducted in 2021 and 2023 will have a catalytic effect:
- At APEA, follow-through activities are lined up to further engage parliamentarians in the region. For instance, the APPFE is planning to raise awareness and promote knowledge about the importance of using evidence, generated by evaluation for oversight, policy and decision-making through webinars and conferences. A regional consultation on NEPs is in the pipeline at the 4th APEA Conference in December 2023. A session at the Parliament (House of Representatives) of the Philippines is also being planned as part of the conference.
- The will to implement the knowledge and experience gained in the workshop in their own parliaments was reflected in the results of the final questionnaire: 90% of respondents stated that they want to convince their (parliamentary) colleagues of the usefulness of the evaluation. Around three quarters want to use evaluation results for decision-making in the future, demand that evaluations be carried out or anchor evaluation clauses in new laws and decrees. Even if only a part of these declarations of intent are put into practice, the training has actually helped to turn the workshop motto ‘Better Policy Making Through Evaluation’ into reality.
Especially the second point is a great success, because the development of national evaluation structures and processes, one of the central goals of the GEI, cannot succeed without the involvement of the legislature. The political change towards evidence-based, rational governance does not only affect the executive, but also requires a legislative foundation. Parliamentarians can not only use evaluation results for their debates and decision-making, but they can also make a significant contribution to setting the framework for evidence-based politics by enacting evaluation laws or building evaluation clauses into laws.
Lessons learned and outlook
- The organizational implementation went very smoothly thanks to highly motivated and extremely competent partner organizations. Of particular importance was the cooperation with APEA and GPFE as regional and thematical partners. This collaboration serves as an exemplary best-practice for future trainings.
- Trainings with parliamentarians and parliament staff as intended target group require a high degree of flexibility and perseverance. Up until the end, the organization team put a lot of administrative efforts into securing the participants for the training. Also, last-minute adjustments need to be considered.
- The excellent feedbacks do not just fall down from the sky. They are the clear result of rigorous and precise planning and organizing. This also includes tailoring the content to the actual needs of the primary target group and clear communication on their own preparation for the training. Regular follow up with parliamentarians and parliament staff are required to ensure their participation in the training.
- Trainings in the Global South require a diversified fundraising strategy to attract enough participants. The various cost factors of the training must be considered, whereby the first implementation, due to its novelty, always causes higher costs than subsequent training based on it. Potential sponsors must be made very clear about the purpose of support.
- It has become apparent, that there is a significant interest of parliamentarians and parliamentarian staff in attending trainings to further improve their skills and knowledge in evidence-based policy-making. The participants expressed their gratitude for the training opportunity, and were very engaged during the training.
- The willingness to learn about evaluation can be seen as an indicator that parliamentarians are seeking to shift their political style towards a more rational approach to decision-making. Evaluation can serve as an important tool in this regard.
- The feedback of participants shows, that the content of the training provides them with tools and guidance for working more efficiently and effectively in areas such as analyzing (progress) reports of projects/programs/bills, budget research, drafting of and/or final decision-making on new laws. This indicates, that the goal to convince and equip participants with knowledge and skills to use evaluative evidence in the policy-making process has been achieved with great success.
- There is a significant (and so far, unmet) demand for further trainings in this area. Therefore, to further raise awareness about evaluation among parliamentarians and parliament staff worldwide, this training series should also be offered in other regions. This includes partnering with other organizations to deliver the training.
- To build robust and effective national M&E systems, parliamentarians and parliament staff cannot be ignored as target groups. Therefore, it is important, to focus on the further mainstreaming of M&E in parliamentary structures, which goes beyond solely individual trainings. It is of utmost importance to demonstrate not only how participants can apply their new knowledge in their daily work, but also how they can establish evaluative structures within their countries.
Country spotlights
Lhatu and Tashi Samdrup, both Members of the National Council of Bhutan, on their motivation to attend the training and challenges regarding M&E in Bhutan:
How is your work connected to evaluation and what was your motivation for attending the training?
Lhatu: I am in the upper house of the parliament and we review lots of plans and policies of the government. But sometimes it is very difficult to raise the right questions and to find the facts. With evaluation we are better equipped to do so. The virtual training in 2021 was very useful regarding this and after it I joined the Asia Pacific Parliamentarians Forum. Therefore, I thought this training now is the best opportunity. On one side, to give me an enriching tool. On the other side, to better advocate and talk with colleagues, the government and stakeholders on the issue.
Another thing that has motivated me was networking. Because through networking you can share your experiences. Otherwise, if you do not have networking it becomes difficult to know, whom to contact.
Tashi Samdrup: I did hear about the training from a colleague. I went to the training description and realized: This is a very important training for us as parliamentarians! When we amend acts or frame policies and when we make recommendations to the government, it would be very helpful to have evidence-based evaluations. For the last three days I have learned a lot and I think the content of the training was very impressive. Not only the way of teaching, also the interactions between the instructors/facilitators and the participants, the case studies and group work. So, everybody can come forward and fully participate.
What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?
Lhatu: Bhutan is going through a drastic change, what we call transformation. In this process, a lot of changes are coming up by initiative of the government, e. g., in education, civil service recruitment processes, or tourism. These are areas where we can actually take this knowledge from the training in order to analyze, whether policies have been useful to the country or not.
Tashi Samdrup: In Bhutan we can find a lot of information and data. However, sometimes we lack coordination among the agencies and we do not get the right figures and details. Different agencies have different data, that is why we must have a central agency who is taking care of this.
Additionally, in the framework of this transformation process, most of the sectors want to adopt new systems. What we really do not know is, whether they carried out evaluations on the existing systems and what exactly needs to be changed. Through this training I think we will be able to advise some of our law- and decision-makers about evaluation of their policies. And then you can come up, based on the evidences, with very good policy decisions.
What are you planning to you apply from the training in your work?
Tashi Samdrup: In the House of Reviews, in special occasions, we can ask questions to the ministers about their decisions. Through this training, we know now about the importance of clear evidence. This gives us the opportunity to challenge ministers that they are not doing what the evidence suggests.
Lhatu: In Bhutan we can really ask those kinds of questions like ‘What kind of studies have you done to come to this kind of argument? You said you have this number of this and that, from where did you get this figure?’ This is a now big tool we have got from the training, because if you do not ask the right questions, you will also not get the right results, figures, etc.”
Each ministry also has their annual report. We can get a lot of information from there, but sometimes we do not or cannot really cross-check. That is the problem and we really need to work on our personal evidence- creation as well to support our own arguments. This means, we have to draw out our own studies. Only if we are accurate, then they can be accurate too.
What we also have is the Evaluation Association of Bhutan. They are still in an emerging phase but promoting evaluation a lot in the country. So far there is no institutionalized contact with them, but first contacts are made. Of course, they could give presentations in the parliament. This would be important, because not all parliamentarians are able to get this kind of training like us. But also, we two are now multiplicators and have a special role in promoting evaluation to other colleagues
Natalia Nikitenko, Former Member of Parliament of the Kyrgyz Republic, on networking and advocacy for evaluation:
How is your work connected to evaluation and what was your motivation for attending the training?
During my time as member of the parliament for eleven years I was chairperson for different committees and worked a lot with evaluators and VOPEs. Therefore, it is my strong conviction that evaluation is a more than proper tool for decision-makers, which can make changes on the political level.
In this time, we initiated the law on evaluation in Kyrgyz Republic, which was accepted by the parliament in 2014. The law sets the overall framework for the government, parliament, local councils, civil society, professional evaluators etc., and approaches the agenda on evaluation. However, building a national evaluation policy takes time. Having the legislation is not enough and only one of the important fragments. That is why we worked together with the VOPE to engage in capacity-building. The information from the virtual training in 2021 and the training this year is an excellent source for building my capacity as an expert in this sector. For example, I learnt a lot about structure of evaluation reports, evaluation methodologies, or how to inform stakeholders. All this definitely broadened my perspective on the topic of evaluation.
What is the importance of networks for the advocacy for evaluation?
I think it is extremely important to build regional connections in one bigger platform, which is the actual objective of the Global Parliamentarians Forum for Evaluation. That is why I really appreciate the in-person format. Right from the beginning within the Global Parliamentarians Forum for Evaluation, we had the ambitious plan to build the networks on regional level and one global platform in order to establish sustainable communication, in terms of promoting evaluation agenda and very practical exchange of information. For achieving these goals, it is important to continue building-capacity, and especially minimizing the gap between VOPEs and decision-makers. This training now helps us to continue cooperation on a regular basis in order to help the evaluation culture in the countries and to build national evaluation policies.
In Kyrgyzstan. for example, when we adapted the law in 2014, we did not know about the global evaluation movements or networks. We just worked on ourselves on the national level. Through our VOPE I was then step by step more involved in the international activities and met decision-makers from other countries, which brought a new perspective to me. Through this we were able to develop very practical methodologies, like evaluation of laws or evaluation of national programs. We also built the capacity of people working in the parliament and for the government, linked them to the VOPE, and linked them to the Global Parliamentarians Forum for Evaluation. And then we developed our own national evaluation policy on the parliamentarian level. We are far from having a national evaluation policy as a whole, but with all this we have good fragments to make a change in the country, improve the quality of decision-making and rise the agenda of evaluation. This is a result of our international involvement as a country.
What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?
Right now, we have political changes in the country for around two years. There is a new constitution, a new president, a new political system, which brings new challenges like building transparent system of good governance. However, this also brings new opportunities.
I am not a member of the parliament anymore, but now working as an expert with the parliament, ministries and UN organizations to help the government building system of good governance, accountability and to implement evaluations. This also includes trainings for parliament staff on evaluation and I am using all tools and channels to continue the work we have started many years before.
Mahira Rafique, Staff of Parliament, National Assembly of Pakistan, on institutional change:
How is your work connected to evaluation and what was your motivation for attending the training?
I work in the National Assembly of Pakistan. My department deals with stakeholder management, which means that all external institutions / development partners, which would like to work with the different departments of the National Assembly, need to work with and through our office. Furthermore, we oversee the 5-year Strategic Plan of the National Assembly.
The reason why I joined this training is because we felt a lack of M&E-knowledge, of how things are being driven by partners and if their objectives are aligned with those of us. As we do not have a dedicated evaluation office, my office encouraged me to attend this training.
What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?
There is a lack of knowledge about evaluation. For example, before attending this training, I did not know about the exact difference between a performance audit and an evaluation. I do not see this as a lack of will but a lack of knowledge/skill-sets about this field and dedicated attention to it. That is why we do not have in-house experts in this field. So, this particular training has exposed me to an entirely different realm of evaluation.
You participated in the virtual workshop series 2021. What was your main take-away from the trainings and how did you apply it?
I have been actually able to draft Terms of References (ToRs) for our performance auditors/evaluators. I have also commissioned at least two reports, one on Womens’ Parliamentary Caucus and the other on the SDGs’ Secretariat; both cross-party active platforms of the Parliament pursuing women empowerment and global development agenda, respectively. In this context, I was the key person who sat down with the donor organizations and for the first time, the ToRs and the assessment questions came from the Parliament itself. This was all thanks to the 2021 training.
With this, was it possible for you to generate/contribute to a change in your organization/parliament?
The key findings of these two reports were, that in both caucuses the agenda being pursued was not aligned to the Strategic Plan of the National Assembly. Instead, they were completely donor driven. So, we realized, that all our resources actually went into fulfilling partner driven goals and not our own. So, basically the priorities were misplaced.
This led to structural changes within our organization; we implemented structural change and made these caucuses formal part of the National Assembly Secretariat. Additionally, we hired around 12 in-house consultants who will directly work for us under pre-defined targets and JDs. Finally, we are not including partner- given consultants anymore. I think this is a huge historic reform, which had not taken place during the past two decades, ever since these two caucuses were established.
As a result of the training, have you engaged in further M&E-related activities?
Thanks to this training, I was very happy to be elected as the Vice President for the Global Parliamentarians Forum in the Asia Pacific. We have been able to develop a constitution for the forum and also created brochures on OECD-DAC evaluation criteria. Right now, I am distributing these brochures to the parliamentarians and especially to the SDGs’ Secretariat at the National Assembly, to use them as evaluation criteria for their oversight tasks of the various projects being undertaken by the Ministry of Planning and Development.
What are you planning to apply from the training in your work?
I think one thing that I have learnt is that evaluation is lacking in the Asia Pacific-region, overall. We have seen that Europe is way ahead when it comes to evaluation, especially at the parliamentary level. Relevant is that we are now not only questioning where the budget was spent in terms of moneys, but also about the quality and impact of such expenditures.
All in all, I feel there is a need for a culture of evaluation. We need to be more aware about evidence-based information and we need more skilled and qualified people in this professional niche.
A very interesting idea that I learnt here was from the Sri Lankan model, which was to have a dedicated evaluation office in the Parliament. The reason being that commissioning external evaluators is a lengthy process that needs a lot of approvals. So, the two ideas from Sri Lanka; establishing a National Evaluation Policy and setting up a dedicated office for evaluation within the Secretariat of the Parliament, is something which I would like to discuss further with the parliamentarians back home.
Romulo Emmanuel Miral, Jr, Staff, House of Representatives/Congressional Policy & Budget Research Department on the mainstreaming of M&E systems:
What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?
Unless we have a common understanding about evaluation, it is very difficult to make use of evaluation as an effective tool for development. We need to formulate a policy that will systematize the use of evaluation government-wide. After all, many of the problems and issues that we attempt to address through policies and programs are interconnected; if each sector or agency undertakes evaluation in its own way without coordinating and linking with other sectors or agencies, we will fail to sufficiently address the complexity and depth of these problems and issues. Many government agencies have their own M&E-systems, but there is no integrated M&E system for the whole of government.
To make this government-wide M&E evaluation system work, we need to improve the way we utilize administrative data. A by-product of a good monitoring system is quality administrative data that are useful to evidence-based evaluation, decision making, and policy making. Our legislators would be very happy to have in their hands the evidence that they need to effectively oversee the programs, policies, and projects of government, but unfortunately, the evidence or the data that they need, are not always there.
What was your main take-away from the trainings in 2021 and how did you apply it?
Overall, although the trainings have been brief, they were well-organized and they sufficiently covered the theories and concepts on evaluation. The lectures provided opportunities for the participants to gain a common understanding of evaluation.
The content of the online workshop series introduced me to systems, standards, criteria, and practices that give flesh to the theories and concepts of evaluation. This enabled me to delve into the chief functions of evaluation in government operations, namely, a) ensuring that our scarce resources are allocated to the correct programs and projects based on efficiency and effectiveness criteria, and b) guaranteeing that these programs and projects achieve their intended results and outcomes.
With this, was it possible for you to generate/contribute to a change in your organization/parliament?
At my level and at this point of the evolution of evaluation in our government, my best contribution is to advocate for and provide informational foundations for the enactment of a national evaluation policy. My target audiences, of course, are my principals, the legislators.
The more I learnt, the more I became convinced that we need to have a policy that would guide, structure, and integrate the M&E-systems of the different agencies of the government that are currently very fragmented. Our office, CPBRD, has been very active in pushing for the enactment of a national evaluation policy, which has already been in the legislative agenda of the Philippine Development Plan. This would institutionalize evaluation as a tool for policy- and decision-making.
I also tried to look for opportunities for our office to engage in capacity-building for evaluation. This is essential because institutionalizing an evaluation policy is one thing, and actually possessing and using the skills, competencies, tools, and mindset to operationalize the policy is another. With the help of UNICEF, we were able to access some capacity-building activities. Also, through networking with the Asia Pacific Evaluation Association we were also able to access online trainings and now this on-site IPDET training.
What are you planning to apply from the training in your work?
We continue our initiative on advocating for and providing information and learnings on institutionalizing a national evaluation policy. We are also trying to engage other government agencies towards the establishment of an integrated M&E system.
Mylvaganam Thilakarajah, Former Parliamentarian, Sri Lanka, on using evaluations for improved policy-making:
What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?
As a former colonialized country, we are still acting under colonialist practices we had like 100 years before. We have the 75th anniversary of independence this year. But still our laws and policies are very old and should be updated. However, our policy-makers are still lacking the capability of making decisions based on evidences and they are not much aware about evaluation and how we can benefit from it.
That is why we are conducting awareness program among the parliamentarians and the policy-makers and other sector stakeholders, like administration officers, the planning officers, and the civil society who can influence in decision-making. So, this project creates the environment for evidence-based decision through evaluation.
What was your main take-away from the trainings in 2021 and how did you apply it?
I did participate in the training in 2021 and also in the summer program in 2018 in Bern/Switzerland. I want to say that in 2018, when I was in Bern, it was great to share the knowledge with the evaluators, because I am not an evaluator, but I am for evaluations. Of course, that influenced me, that evaluation is very important for decision- making. And 2021 was specially designed for parliamentarians, I was able to focus on that, how parliamentarians would benefit from that. The take-away message from these trainings and the one now in Sri Lanka is: “We must meet on-site!”. Of course, all the online trainings are great regarding content, but you cannot talk to and get to know each other the same way, especially when it comes to share best practices.
What I did realize after the trainings is the following: First, we as participants can influence our/other parliamentarians with our new evaluation knowledge. Second, we can influence the institutions about evaluation. For example, for Sri Lanka we developed three major recommendations: 1. Enact the bill on evaluation. 2. Set up a standing committee on evaluation within the parliament. 3. Enhance the parliament research division as an evaluation unit. Third, most discussions on evaluation are done in English. To get more parliamentarians involved we need to transmit to local languages. There is no language barrier per se, but it would support spreading the word about the topic.
What are you planning to apply from the training in your work?
I am not a member of the Parliament anymore, but working for example for the Sri Lankan Evaluation Association (SLEvA) as a resource person and project manager now. What I do in this context is managing evaluations with the government institutions. We have selected three pilot evaluations, one from education, one from health, and one from agriculture. I am enjoying that, being a former parliamentarian becoming an evaluation consultant through the benefits of the all the trainings I attended.
What I am going to do after this training now, is to further share all the materials and knowledge with others. With all these stakeholders and the network, we can move forward and create policies or programs on evaluation for evidence-based decision-making.
My final message is the following: We need to serve our people and be accountable to them regarding what happened to the tax money we collected from them. Through evaluation we can find a way to take right decision on how to serve people. This is the scientific method to make decisions. We can understand through the experts and professionals and we can implement evaluation for the betterment of the country and the people.
Central information on the training
Impressions
Organization team 2023
Prof. Dr. Reinhard Stockmann:
Founder and Director of Center of Evaluation (CEval) at Saarland University and Member of the Steering Group of IPDET
Laszlo Szentmarjay:
IPDET Global Outreach coordinator and Member of the Steering Group of IPDET
Dr. Asela Kalugampitiya:
President of Sri Lanka Evaluation Association and Asia Pacific Evaluation Association (APEA)
Randika de Mel:
APEA Manager
Madhuka Liyanagamage:
APEA Coordinator
Dorothy Mae Albiento:
EvalYouth Asia Co-Leader
Special thanks to our sponsors and supporters:
and the Asian Development Bank (ADB)