IPDET’s Impact
The Tracer Study, conducted in 2021/22, covering the years 2014-2020, provides insights about the impact IPDET is making.
The Tracer Study was meant to represent equal number of program years to represent both Canada and Switzerland as venues of the IPDET programs: 2014 to 2016 were three On-Site Program years conducted in Canada. 2018 and 2019 were both conducted On-Site in Switzerland while 2020 was offered only online, the base of operation and management was in Switzerland. This equals to a total number of individuals attending IPDET between 2014-2020 at least one time of 1,018.
Design and Methods
The Tracer Study followed an ex-post facto research design. This means that only data from persons who participated in IPDET (and organizations, which sent participants) in former years, were collected.
A multi-step mixed-methods approach, using qualitative and quantitative data collection and analysis methods, was applied for this Tracer Study.
This involved at first level a review of IPDET’s publications and evaluation reports of the years 2014 till 2020, as well as a review of related literature of tracer studies.
Afterwards an alumni database covering the years 2014-2020 was compiled using participants background data received from Canada and merged with the existing database in Switzerland. This database served as the basis for the selection of alumni and organizations for the interviews as well as for the online survey.
In a further step, semi-structured interviews with alumni served mainly as exploratory to support the development of the online survey. After the survey was developed and programmed into the software SoSciSurvey, it underwent several testing. Response from these testing was used to refine and finalize the online survey. The survey was then published, and invitations were sent out on 18 May 2022 and the survey was administered until 6 July 2022.
The population for the online survey came from the alumni database from 2014 to 2020. Invitation to respond to the survey was sent out to 908 participants, however 137 mails failed to deliver; 771 people represent therefore the total population for the online survey. A total of 228 alumni responded to the survey. In the end, 221 cases were included in the analysis, as seven cases only finished one page of the survey, with a response rate of 28.7%.
For the data analysis, qualitative content analysis and descriptive statistics were mainly employed. Data sources and methods used for data analysis are outlined in the analysis grid. Furthermore, quantitative data were triangulated with qualitative data whenever possible.
Objectives
The overall objectives of the Tracer Study were to provide the IPDET management with information on the achievement of its goals, formulated in its Program Theory. Additionally, it was supposed to support a development process, as it promotes internal and external communication on the results and with this, in the end, further improvement. Lastly, it aimed to create transparency on the programs implementation and be accountable towards its stakeholders.
Based on these objectives, IPDET defined the following categories guiding the Tracer Study: demographics of alumni and organizations, increased knowledge and skills and commitment to produce and use evaluative evidence, and grown community that is actively sharing learning and collaborating. The timeline defined was 2014-2020, to represent equal number of program years to represent both Canada and Switzerland as venues of the IPDET programs. For the implementation a multi-step mixed-methods approach, using qualitative and quantitative data collection and analysis methods, was applied for this Tracer Study.
Results
Result 1: IPDET is reaching a diverse target audience in the international community | Results of the demographic analysis of the alumni database and survey respondents have shown that IPDET attracts and reaches a diverse target audience in terms of gender balance, regional distribution, income classification and type of organization.
- In terms of gender distribution, the participation trend from 2014 to 2020 showcased a distribution that favors participation from women with an average of 58% female participation.
- Most participants at the time of their IPDET attendance were coming – based on work country – from Europe and Central Asia (29%), Sub-Saharan Africa (23%) and North America (15%). Comparing the regional distribution between the two venues revealed that proximity to the venue seems to play a role in the regional distribution of participants: participation from Europe and Central Asia significantly increased in 2018 and 2019 compared to 2014, 2015 and 2016. Also, though smallest in percentage, participants working in the Middle East and North African region doubled 2018 to 2019 from an average of 2.7% to 5.4%. On the other hand, participants from the North American region have significantly reduced from an average of 21.1% (2014-2016) to 12.4% (2018 to 2019).
- An average of 45% were awarded an IPDET scholarship between 2014 and 2020. The availability of IPDET scholarship helped facilitate participation to IPDET trainings for participants coming from low, lower-middle-, and upper-middle-income countries. Furthermore, the online format offered by IPDET since 2020 seems to have helped increase participation and accessibility for participants coming from low- and lower-middle-income countries.
“Without scholarship I would have probably participated only on the Core Course. With the scholarship I was able to add one week workshop to my training, which was very good.” – Participant from 2019
- In terms of type of organization, about a third of alumni were working in government organizations during their IPDET participation. The most striking difference between the two venues is reduction to more than half in the number of participants working in a government agency. The reduction of government agency participants also means increase of participation from other types of organization from 2018 to 2020, namely from the UN (specialized agencies), NGO and foundation, other international/regional or multilateral organizations, and from the World Bank Group.
- Majority of alumni are still currently working in the same type of organization they were working at when they participated at IPDET. However, it is worth noting that there has been a 5% increase of self-employed among the survey respondents.
- While IPDET is reaching a diverse target it seems to be difficult so far to convince most sending organization to send more than one participant or attract past participants to attend other (more advanced) workshops in succeeding years.
“The IPDET program was for me an important step in my education as an evaluator since there is no official higher level education program available in my country nor region. It provided a structured overview of the topic, contributed to the development of my evaluation skills and use of evaluation methodologies. In terms of intercultural cooperation, I am deeply thankful in getting the opportunity to meet and learn from peers from all over the world. What a wonderful experience and opportunity to share knowledge.” – Participant from 2018
Result 3: IPDET alumni are able to apply their IPDET training in their work and boost their personal and professional development | The majority of alumni were able to apply their acquired knowledge and skills in their work. From the point of view of the organizations interviewed, familiarization of terminologies used in evaluation, uniform use of methodology, and understanding technical proposals and reports are some of the useful knowledge and skills acquired by their staff, especially for beginners in the field.
Their IPDET participation was also useful in terms of personal and professional development. Almost all respondents (strongly) agreed that attending IPDET was useful for their personal development. A majority claimed that their evaluation related tasks/responsibilities were extended and their job opportunities in the field of evaluation increased. Moreover, majority of respondents who currently work as consultants claimed to have received more consulting jobs after participating in an IPDET course. Finally, qualitative data revealed that the program was useful not just for those working in the field of evaluation but also provided transferrable skills for those who are now working in other field of profession.
“I think IPDET is a tremendous initiative that has filled a real need. I hope it continues to grow and prosper and reach even more people – particularly in developing regions. I have benefited enormously from being part of the IPDET family, and I look forward to attending more of the specialized sessions, and doing whatever I can to support to its future development for my organization and my professional background.” – Participant from 2016
Result 5: The networking opportunity and IPDET’s community-building format is a unique and an important aspect of IPDET | Through its different community building formats, IPDET offers participants and alumni several opportunities to network, collaborate and share knowledge during the training as well as different online platforms to keep the alumni community active. Results from both quantitative and qualitative analyses provided evidence to the extent that IPDET has successfully contributed to fostering an active evaluation community during and after its trainings. This was verified by the results of the survey where two-thirds of respondents (strongly) agree that their intercultural awareness was raised and understanding of different fields of evaluation practice was broadened through their exchange with other participants. A majority of the survey respondents claimed they still have contact with other alumni members, some regularly exchange (also on evaluation topics), and some have worked on joint evaluation activities with other alumni members.
“What distinguishes the IPDET 2014 training program is the face-to-face coexistence with the various participants from all over the world. The discussions that took place in the evenings on the university campus after the formal training in the halls had a very prominent role in exchanging knowledge and exchanging cultures, and this in turn developed the personality of the evaluators to understand the reality of the evaluation in other countries around the world. […] Eternal friendships are built from coexistence face to face with each other. […] To this day, we exchange experience and knowledge in the field of evaluation.”– Participant from 2014
Recommendations
Relating to:- IPDET should consider designing appropriate follow-on courses, both, online and on-site to encourage alumni to enroll again in IPDET courses.
- This goes hand-in-hand with the identification and testing of different course formats for different audiences, e. g. shorter courses, hybrid-formats, flipped classroom, etc.
- Moreover, IPDET should take further advantage of online workshops as it offers the important opportunity to target individuals, more specifically those who are normally unable to attend the On-site Program, to enroll.
- To further strengthen the position of IPDET as a high-quality training provider, IPDET should investigate offering micro-credentials or other types of university certification.
- IPDET should continue to offer fundamentals in evaluation, including a solid foundation in methodology and theory, as well as cutting-edge topics in the field of evaluation.
- To keep its program up-to-date, IPDET should stay flexible and able to adapt to the changing needs of different audiences (public, private, non-profit, academic sector).
- To respect different socio-economic backgrounds, partial and full scholarships need to be considered, to grant individuals with insufficient financial means access to IPDET.
- Applicants who are rejected during the selection process, should be encouraged to join other IPDET formats (i. e., virtually, free of charge).
- In order to further assure a professional implementation of the online format, IPDET should continue investing in digital didactics.
- To offer more intense opportunity to connect with other participants, IPDET should strengthen networking opportunities for online participants.
- To offer them more ownership of the IPDET community, IPDET should make alumni more engaged and increase their role in the activities and events offered by the community.
- IPDET should look into new organizations who have not yet sent participants to IPDET courses and investigate how to best market IPDET to these organizations.
- In order to keep tracing alumni pathways, IPDET could benefit from conducting regular tracer studies, e. g. every three years. Further tracer elements can be included in regular communication with its alumni, e. g. via data retrievals or status reports.
- Future tracer studies should put more focus on the extent of acquired knowledge and skills, and the usefulness and impact of IPDET in alumni’s career.
Read the executive summary and the complete Tracer Study 2014-2020 here:
