Activity: Better policy making through Evaluations

Lhatu and Tashi Samdrup, both Members of the National Council of Bhutan, on their motivation to attend the training and challenges regarding M&E in Bhutan:

How is your work connected to evaluation and what was your motivation for attending the training?

Lhatu: I am in the upper house of the parliament and we review lots of plans and policies of the government. But sometimes it is very difficult to raise the right questions and to find the facts. With evaluation we are better equipped to do so. The virtual training in 2021 was very useful regarding this and after it I joined the Asia Pacific Parliamentarians Forum. Therefore, I thought this training now is the best opportunity. On one side, to give me an enriching tool. On the other side, to better advocate and talk with colleagues, the government and stakeholders on the issue.

Another thing that has motivated me was networking. Because through networking you can share your experiences. Otherwise, if you do not have networking it becomes difficult to know, whom to contact.

Tashi Samdrup: I did hear about the training from a colleague. I went to the training description and realized: This is a very important training for us as parliamentarians! When we amend acts or frame policies and when we make recommendations to the government, it would be very helpful to have evidence-based evaluations. For the last three days I have learned a lot and I think the content of the training was very impressive. Not only the way of teaching, also the interactions between the instructors/facilitators and the participants, the case studies and group work. So, everybody can come forward and fully participate.

What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?

Lhatu: Bhutan is going through a drastic change, what we call transformation. In this process, a lot of changes are coming up by initiative of the government, e. g., in education, civil service recruitment processes, or tourism. These are areas where we can actually take this knowledge from the training in order to analyze, whether policies have been useful to the country or not.

Tashi Samdrup: In Bhutan we can find a lot of information and data. However, sometimes we lack coordination among the agencies and we do not get the right figures and details. Different agencies have different data, that is why we must have a central agency who is taking care of this.

Additionally, in the framework of this transformation process, most of the sectors want to adopt new systems. What we really do not know is, whether they carried out evaluations on the existing systems and what exactly needs to be changed. Through this training I think we will be able to advise some of our law- and decision-makers about evaluation of their policies. And then you can come up, based on the evidences, with very good policy decisions.

What are you planning to you apply from the training in your work?

Tashi Samdrup: In the House of Reviews, in special occasions, we can ask questions to the ministers about their decisions. Through this training, we know now about the importance of clear evidence. This gives us the opportunity to challenge ministers that they are not doing what the evidence suggests.

Lhatu: In Bhutan we can really ask those kinds of questions like ‘What kind of studies have you done to come to this kind of argument? You said you have this number of this and that, from where did you get this figure?’ This is a now big tool we have got from the training, because if you do not ask the right questions, you will also not get the right results, figures, etc.”

Each ministry also has their annual report. We can get a lot of information from there, but sometimes we do not or cannot really cross-check. That is the problem and we really need to work on our personal evidence- creation as well to support our own arguments. This means, we have to draw out our own studies. Only if we are accurate, then they can be accurate too.

What we also have is the Evaluation Association of Bhutan. They are still in an emerging phase but promoting evaluation a lot in the country. So far there is no institutionalized contact with them, but first contacts are made. Of course, they could give presentations in the parliament. This would be important, because not all parliamentarians are able to get this kind of training like us. But also, we two are now multiplicators and have a special role in promoting evaluation to other colleagues

Natalia Nikitenko, Former Member of Parliament of the Kyrgyz Republic, on networking and advocacy for evaluation:

How is your work connected to evaluation and what was your motivation for attending the training?

During my time as member of the parliament for eleven years I was chairperson for different committees and worked a lot with evaluators and VOPEs. Therefore, it is my strong conviction that evaluation is a more than proper tool for decision-makers, which can make changes on the political level.

In this time, we initiated the law on evaluation in Kyrgyz Republic, which was accepted by the parliament in 2014. The law sets the overall framework for the government, parliament, local councils, civil society, professional evaluators etc., and approaches the agenda on evaluation. However, building a national evaluation policy takes time. Having the legislation is not enough and only one of the important fragments. That is why we worked together with the VOPE to engage in capacity-building. The information from the virtual training in 2021 and the training this year is an excellent source for building my capacity as an expert in this sector. For example, I learnt a lot about structure of evaluation reports, evaluation methodologies, or how to inform stakeholders. All this definitely broadened my perspective on the topic of evaluation.

What is the importance of networks for the advocacy for evaluation?

I think it is extremely important to build regional connections in one bigger platform, which is the actual objective of the Global Parliamentarians Forum for Evaluation. That is why I really appreciate the in-person format. Right from the beginning within the Global Parliamentarians Forum for Evaluation, we had the ambitious plan to build the networks on regional level and one global platform in order to establish sustainable communication, in terms of promoting evaluation agenda and very practical exchange of information. For achieving these goals, it is important to continue building-capacity, and especially minimizing the gap between VOPEs and decision-makers. This training now helps us to continue cooperation on a regular basis in order to help the evaluation culture in the countries and to build national evaluation policies.

In Kyrgyzstan. for example, when we adapted the law in 2014, we did not know about the global evaluation movements or networks. We just worked on ourselves on the national level. Through our VOPE I was then step by step more involved in the international activities and met decision-makers from other countries, which brought a new perspective to me. Through this we were able to develop very practical methodologies, like evaluation of laws or evaluation of national programs. We also built the capacity of people working in the parliament and for the government, linked them to the VOPE, and linked them to the Global Parliamentarians Forum for Evaluation. And then we developed our own national evaluation policy on the parliamentarian level. We are far from having a national evaluation policy as a whole, but with all this we have good fragments to make a change in the country, improve the quality of decision-making and rise the agenda of evaluation. This is a result of our international involvement as a country.

What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?

Right now, we have political changes in the country for around two years. There is a new constitution, a new president, a new political system, which brings new challenges like building transparent system of good governance. However, this also brings new opportunities.

I am not a member of the parliament anymore, but now working as an expert with the parliament, ministries and UN organizations to help the government building system of good governance, accountability and to implement evaluations. This also includes trainings for parliament staff on evaluation and I am using all tools and channels to continue the work we have started many years before.

Mahira Rafique, Staff of Parliament, National Assembly of Pakistan, on institutional change:

How is your work connected to evaluation and what was your motivation for attending the training?

I work in the National Assembly of Pakistan. My department deals with stakeholder management, which means that all external institutions / development partners, which would like to work with the different departments of the National Assembly, need to work with and through our office. Furthermore, we oversee the 5-year Strategic Plan of the National Assembly.

The reason why I joined this training is because we felt a lack of M&E-knowledge, of how things are being driven by partners and if their objectives are aligned with those of us. As we do not have a dedicated evaluation office, my office encouraged me to attend this training.

What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?

There is a lack of knowledge about evaluation. For example, before attending this training, I did not know about the exact difference between a performance audit and an evaluation. I do not see this as a lack of will but a lack of knowledge/skill-sets about this field and dedicated attention to it. That is why we do not have in-house experts in this field. So, this particular training has exposed me to an entirely different realm of evaluation.

You participated in the virtual workshop series 2021. What was your main take-away from the trainings and how did you apply it?

I have been actually able to draft Terms of References (ToRs) for our performance auditors/evaluators. I have also commissioned at least two reports, one on Womens’ Parliamentary Caucus and the other on the SDGs’ Secretariat; both cross-party active platforms of the Parliament pursuing women empowerment and global development agenda, respectively. In this context, I was the key person who sat down with the donor organizations and for the first time, the ToRs and the assessment questions came from the Parliament itself. This was all thanks to the 2021 training.

With this, was it possible for you to generate/contribute to a change in your organization/parliament?

The key findings of these two reports were, that in both caucuses the agenda being pursued was not aligned to the Strategic Plan of the National Assembly. Instead, they were completely donor driven. So, we realized, that all our resources actually went into fulfilling partner driven goals and not our own. So, basically the priorities were misplaced.

This led to structural changes within our organization; we implemented structural change and made these caucuses formal part of the National Assembly Secretariat. Additionally, we hired around 12 in-house consultants who will directly work for us under pre-defined targets and JDs. Finally, we are not including partner- given consultants anymore. I think this is a huge historic reform, which had not taken place during the past two decades, ever since these two caucuses were established.

As a result of the training, have you engaged in further M&E-related activities?

Thanks to this training, I was very happy to be elected as the Vice President for the Global Parliamentarians Forum in the Asia Pacific. We have been able to develop a constitution for the forum and also created brochures on OECD-DAC evaluation criteria. Right now, I am distributing these brochures to the parliamentarians and especially to the SDGs’ Secretariat at the National Assembly, to use them as evaluation criteria for their oversight tasks of the various projects being undertaken by the Ministry of Planning and Development.

What are you planning to apply from the training in your work?

I think one thing that I have learnt is that evaluation is lacking in the Asia Pacific-region, overall. We have seen that Europe is way ahead when it comes to evaluation, especially at the parliamentary level. Relevant is that we are now not only questioning where the budget was spent in terms of moneys, but also about the quality and impact of such expenditures.

All in all, I feel there is a need for a culture of evaluation. We need to be more aware about evidence-based information and we need more skilled and qualified people in this professional niche.

A very interesting idea that I learnt here was from the Sri Lankan model, which was to have a dedicated evaluation office in the Parliament. The reason being that commissioning external evaluators is a lengthy process that needs a lot of approvals. So, the two ideas from Sri Lanka; establishing a National Evaluation Policy and setting up a dedicated office for evaluation within the Secretariat of the Parliament, is something which I would like to discuss further with the parliamentarians back home.

Romulo Emmanuel Miral, Jr, Staff, House of Representatives/Congressional Policy & Budget Research Department on the mainstreaming of M&E systems:

What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?

Unless we have a common understanding about evaluation, it is very difficult to make use of evaluation as an effective tool for development. We need to formulate a policy that will systematize the use of evaluation government-wide. After all, many of the problems and issues that we attempt to address through policies and programs are interconnected; if each sector or agency undertakes evaluation in its own way without coordinating and linking with other sectors or agencies, we will fail to sufficiently address the complexity and depth of these problems and issues. Many government agencies have their own M&E-systems, but there is no integrated M&E system for the whole of government.

To make this government-wide M&E evaluation system work, we need to improve the way we utilize administrative data. A by-product of a good monitoring system is quality administrative data that are useful to evidence-based evaluation, decision making, and policy making. Our legislators would be very happy to have in their hands the evidence that they need to effectively oversee the programs, policies, and projects of government, but unfortunately, the evidence or the data that they need, are not always there.

What was your main take-away from the trainings in 2021 and how did you apply it?

Overall, although the trainings have been brief, they were well-organized and they sufficiently covered the theories and concepts on evaluation. The lectures provided opportunities for the participants to gain a common understanding of evaluation.

The content of the online workshop series introduced me to systems, standards, criteria, and practices that give flesh to the theories and concepts of evaluation. This enabled me to delve into the chief functions of evaluation in government operations, namely, a) ensuring that our scarce resources are allocated to the correct programs and projects based on efficiency and effectiveness criteria, and b) guaranteeing that these programs and projects achieve their intended results and outcomes.

With this, was it possible for you to generate/contribute to a change in your organization/parliament?

At my level and at this point of the evolution of evaluation in our government, my best contribution is to advocate for and provide informational foundations for the enactment of a national evaluation policy. My target audiences, of course, are my principals, the legislators.

The more I learnt, the more I became convinced that we need to have a policy that would guide, structure, and integrate the M&E-systems of the different agencies of the government that are currently very fragmented. Our office, CPBRD, has been very active in pushing for the enactment of a national evaluation policy, which has already been in the legislative agenda of the Philippine Development Plan. This would institutionalize evaluation as a tool for policy- and decision-making.

I also tried to look for opportunities for our office to engage in capacity-building for evaluation. This is essential because institutionalizing an evaluation policy is one thing, and actually possessing and using the skills, competencies, tools, and mindset to operationalize the policy is another. With the help of UNICEF, we were able to access some capacity-building activities. Also, through networking with the Asia Pacific Evaluation Association we were also able to access online trainings and now this on-site IPDET training.

What are you planning to apply from the training in your work?

We continue our initiative on advocating for and providing information and learnings on institutionalizing a national evaluation policy. We are also trying to engage other government agencies towards the establishment of an integrated M&E system.

Mylvaganam Thilakarajah, Former Parliamentarian, Sri Lanka, on using evaluations for improved policy-making:

What do you see as the main challenge in your country regarding evidence-based policy making/evaluation?

As a former colonialized country, we are still acting under colonialist practices we had like 100 years before. We have the 75th anniversary of independence this year. But still our laws and policies are very old and should be updated. However, our policy-makers are still lacking the capability of making decisions based on evidences and they are not much aware about evaluation and how we can benefit from it.

That is why we are conducting awareness program among the parliamentarians and the policy-makers and other sector stakeholders, like administration officers, the planning officers, and the civil society who can influence in decision-making. So, this project creates the environment for evidence-based decision through evaluation.

What was your main take-away from the trainings in 2021 and how did you apply it?

I did participate in the training in 2021 and also in the summer program in 2018 in Bern/Switzerland. I want to say that in 2018, when I was in Bern, it was great to share the knowledge with the evaluators, because I am not an evaluator, but I am for evaluations. Of course, that influenced me, that evaluation is very important for decision- making. And 2021 was specially designed for parliamentarians, I was able to focus on that, how parliamentarians would benefit from that. The take-away message from these trainings and the one now in Sri Lanka is: “We must meet on-site!”. Of course, all the online trainings are great regarding content, but you cannot talk to and get to know each other the same way, especially when it comes to share best practices.

What I did realize after the trainings is the following: First, we as participants can influence our/other parliamentarians with our new evaluation knowledge. Second, we can influence the institutions about evaluation. For example, for Sri Lanka we developed three major recommendations: 1. Enact the bill on evaluation. 2. Set up a standing committee on evaluation within the parliament. 3. Enhance the parliament research division as an evaluation unit. Third, most discussions on evaluation are done in English. To get more parliamentarians involved we need to transmit to local languages. There is no language barrier per se, but it would support spreading the word about the topic.

What are you planning to apply from the training in your work?

I am not a member of the Parliament anymore, but working for example for the Sri Lankan Evaluation Association (SLEvA) as a resource person and project manager now. What I do in this context is managing evaluations with the government institutions. We have selected three pilot evaluations, one from education, one from health, and one from agriculture. I am enjoying that, being a former parliamentarian becoming an evaluation consultant through the benefits of the all the trainings I attended.

What I am going to do after this training now, is to further share all the materials and knowledge with others. With all these stakeholders and the network, we can move forward and create policies or programs on evaluation for evidence-based decision-making.

My final message is the following: We need to serve our people and be accountable to them regarding what happened to the tax money we collected from them. Through evaluation we can find a way to take right decision on how to serve people. This is the scientific method to make decisions. We can understand through the experts and professionals and we can implement evaluation for the betterment of the country and the people.