Skip to content
Menu Close

5. Accounting for Covid-19 as a confounding factor during impact evaluations

Accounting for COVID-19 as a confounding factor during impact evaluations

 

We know from numerous studies that exogeneous factors, such as economic crises, natural disasters or rising conflicts can superimpose societal developments that are subject to development interventions – which again are subject to evaluations. Pests, for instance, occasionally thwart the impacts of rural development measures, which impedes their measurement and attribution or at least makes it much more difficult. [1]

As evaluators we are familiar with such challenges and – most of us – know how to deal with them adequately. We have statistical methods available for controlling confounding factors in experimental and quasi-experimental designs, for instance instrumental variable estimation or propensity score matching. And we know about hermeneutic and discursive approaches for identifying bias in our qualitative data, such as expert panels or the Delphi technique.

However, we also know that things are getting trickier when the magnitude of influence of a confounding factor exceeds the expected size of an intervention’s impact. This is even more true, if there is more than one of such nasty factors.

So, what to do if technically speaking the ‘signal-to-noise ratio’ between our subject of investigation and the unwanted interference is becoming too small for such a corrective maneuver? How can we attribute an observable change to an intervention if this change is influenced significantly more by something else than the intervention under investigation itself? Well if you know, then it is high time that you tell us how! Because this is exactly the situation the virus spreading across the globe brings us into in many areas.

Apparently, CoViD19 does not only infect humans but basically every sphere of our lives from our health systems, general infrastructure, education systems and labor markets to national economies and the global development at large.

For instance, imagine evaluating now a further training program and finding out that none of its graduates found a job afterwards; not because it was such a bad program but simply because the labor market in the sector the program was implemented in broke down due to the crisis. Not realistic you think? Hint: tourism.

Or what about evaluating the impact of a value chain support project when due to the crisis the chain just ripped at one point, e.g. because goods could not be shipped from the producers to the consumers. How would you evaluate an intervention aiming at improving the resilience of vulnerable groups at the very moment their resilience is severely tested by a collapsing regional economy?

[1] Cf. CEval (2018): Follow up Study – Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development (https://www.fairtrade-deutschland.de/fileadmin/DE/01_was_ist_fairtrade/05_wirkung/studien/2018_ceval_studie-fairtrade-und-laendliche-entwicklung_komplett.pdf)

The Task

Provide us with an evaluation subject, a suitable evaluation design and a methodological concept for its implementation. As regards the subject, the choice is yours. You can give a general example from your working area or a type of project or program you always wanted to evaluate. The evaluation design should be geared to measure the impact of that project or program as rigorous as possible. Scientific methodological standards do apply!

Please structure your ideas along the following questions:

  • What is your subject of investigation? Please briefly outline the project or program you would evaluate, i.e. its overall implementing framework, target groups, objectives, Theory of Change and at least a rudimental results model.
  • What would be the evaluation design? Which data would you collect when from whom and which are your objects of comparison? Make clear how the evaluation design accounts for intervening factors.
  • Which methods would you apply and how would you combine them in order to control for the bias introduced by an epidemy such as CoViD19 on your subject of investigation, respectively your data? Please make your suggestions as concrete as possible in terms of methods for data collection and analysis.
  • How would you draw your data collection instruments? Which tools would you apply and how? How would they look like in terms of content, items, format etc.?
  • Which precautions would you take to provide for maximum validity and reliability of your data (beyond the usual ones)? What would be the corrective measures?
  • How would you overcome logistical constraints as regards accessibility of information sources, travel restrictions and the like? Outline briefly the organizational setup of your evaluation.

Tell us how you keep the wavelet of your data from the unwanted background noise. We – and surely myriads of fellow evaluators out there – are curious to hear about your ideas!