I’m going to let you in on a little secret. My life as an economist would be much easier if I had the ability to observe an infinite number of parallel universes. Think about it: if I want to understand how X affects Y—say, the impact of a peacebuilding programme on the perceptions of those who received the ‘treatment’—all I need to do is to find a universe identical to our own in every other way, and compare the outcomes in our universe to the outcomes in theirs. Unfortunately, such technologies are, as yet, beyond the reach of economists. We therefore need to be more creative in our research methodologies, if not in our flights of imagination.
For many, the obvious next best thing to a parallel universe is the use of randomised control trials (RCTs). Here, individuals and locations are randomly selected to receive a treatment, while others do not. If this selection process is truly random then, on average, there should be no other difference between the two groups. Thus, convoluting factors are accounted for and the true impact can be isolated. However, despite the benefits of both being an actualreal thing and a tool that isolates causality, use of RCTs, particularly in the kind of scenario that necessitates a peacebuilding programme is, to put it mildly, controversial.
Ethical dilemmas
It’s impossible to get very far in a debate about RCTs in disaster contexts without confronting ethics. On a very human level, ethical dilemmas are inherent to RCTs—only certain individuals, even if randomly chosen, can receive a ‘treatment’, while others are explicitly prohibited from benefitting. Understandably, studying swathes of people who are ineligible to receive treatment is controversial.
However, this argument may be something of a misnomer and, in many cases, is certainly something of a matter of perspective. In principle, ethical concern arises with an assumption that a treatment could be given to everyone. More realistically, budgetary limitations suggest that this assumption will be strong to excess in a majority of scenarios. The question, therefore, really boils down to one about the selection process. How should agencies choose who receives a treatment and who does not?
Take the example of the 2000–2006 European Union Programme for Peace and Reconciliation in Northern Ireland and the Border Region of the Republic of Ireland (PEACE II). At its core, PEACE II solicited applications from individuals, groups and communities that were then evaluated based on quality and expected impact. Thus, individuals, groups and communities self-selected their potential participation, whilst the success of their proposals further relied on assessment against a set of objective criteria.
As a result, from a researcher’s point of view, evaluating the impact of PEACE II is problematic. Of course, the purpose of PEACE II was to improve the lives of individuals in Northern Ireland, not to make it easy for researchers to quantify the programme’s impact. In this respect, I have no desire to gripe about such research complexities. Rather, I seek to contextualise how this approach seems to have influenced the achievements of a programme that spent €500 per person in the areas of its domain.
Money can’t buy love but can it buy PEACE?
In the case of the PEACE II programme, in asking individuals, groups and communities to come together to create applications that could, in turn, help build a better local future, a critical assumption was made. This assumption was that individuals in the areas deemed ‘weakest’ were equally capable of creating successful proposals as those in areas that were deemed ‘stronger’. To put it another way, there was no guarantee that, should this assumption fail, PEACE II spending would make it to the areas that needed it most.
One could argue that, in identifying the strongest proposals, opportunity of progress was maximized. However, this still relies on the money getting to where it was most needed. Our recent research does not show this to be the case. Although individuals in the areas that received funding exhibit higher perceptions of neighbourhood quality than individuals in those that did not, deeper analysis showed that these inflated perceptions were pre-existing.
This suggests that areas with the weakest social capital appear to have been less successful at submitting fruitful applications. The effects of this process are astounding—the correlation between deprivation and receipt of spending, for example, is in the low range, yet we know that the areas in Northern Ireland most affected by violence in the past remain the most deprived today. Such deprived neighbourhoods are the areas where PEACE II was hoped to have the greatest impact, yet it seems that spending did not reach them.
It is, of course, hard to argue against the theory of well-targeted spending, especially when resources are limited. That said, the practicalities and outcomes in Northern Ireland do not match such theoretical expectations. In part, this relates to the funding mechanism itself but a deeper, and more concerning, failing may lurk in the background—a failure to effectively identify the areas that most required assistance; or, at least, a failure to ensure that spending made it to the areas identified as most needy.
Scarcity and limitations of RCTs
At the heart of both RCTs and the open competition model of PEACE II is the notion of scarcity. At some stage, budgetary limitations become important in any decision-making process. Despite a budget of almost €1 billion, PEACE II funded only half of all applications that were received. Some communities received funding of €4.5m, while other locations received none. Here, the major difference between PEACE II and an RCT is not that some individuals who could have benefitted went without, but that PEACE II aimed to specifically target the areas most in need. Although, on the surface, this may overcome the ethical dilemma of choosing who receives a treatment, the process by which such decisions were reached in Northern Ireland seems to have suffered its own major and serious limitations.
More generally, these limitations require significant thought. Northern Ireland is a developed country with a wealth of reliable community-level data. If needy communities in Northern Ireland cannot be targeted accurately, what does this say about the processes by which the most needy communities in areas with less reliable data in the developing world are identified? While it is therefore noble to target spending where it is most needed, it is a path fraught with difficulties and its own ethical dilemmas.
The feasibility of successfully targeting spending is an incredibly important consideration. In cases where it is difficult to identify the most needy communities, the benefits of a targeted spend must be leveraged against the probability of the spending finding its way to the communities most in need. And in efforts to build peace and improve inter-ethnic cohesion, making such an identification remains complex, if not impossible. In this context, although perhaps still imperfect, RCTs may not in fact pose the moral dilemma they appear to at first sight.
I’m going to let you in on a little secret. My life as an economist would be much easier if I had the ability to observe an infinite number of parallel universes. Think about it: if I want to understand how X affects Y—say, the impact of a peacebuilding programme on the perceptions of those who received the ‘treatment’—all I need to do is to find a universe identical to our own in every other way, and compare the outcomes in our universe to the outcomes in theirs. Unfortunately, such technologies are, as yet, beyond the reach of economists. We therefore need to be more creative in our research methodologies, if not in our flights of imagination.
For many, the obvious next best thing to a parallel universe is the use of randomised control trials (RCTs). Here, individuals and locations are randomly selected to receive a treatment, while others do not. If this selection process is truly random then, on average, there should be no other difference between the two groups. Thus, convoluting factors are accounted for and the true impact can be isolated. However, despite the benefits of both being an actual real thing and a tool that isolates causality, use of RCTs, particularly in the kind of scenario that necessitates a peacebuilding programme is, to put it mildly, controversial.
Ethical dilemmas
It’s impossible to get very far in a debate about RCTs in disaster contexts without confronting ethics. On a very human level, ethical dilemmas are inherent to RCTs—only certain individuals, even if randomly chosen, can receive a ‘treatment’, while others are explicitly prohibited from benefitting. Understandably, studying swathes of people who are ineligible to receive treatment is controversial.
However, this argument may be something of a misnomer and, in many cases, is certainly something of a matter of perspective. In principle, ethical concern arises with an assumption that a treatment could be given to everyone. More realistically, budgetary limitations suggest that this assumption will be strong to excess in a majority of scenarios. The question, therefore, really boils down to one about the selection process. How should agencies choose who receives a treatment and who does not?
Take the example of the 2000–2006 European Union Programme for Peace and Reconciliation in Northern Ireland and the Border Region of the Republic of Ireland (PEACE II). At its core, PEACE II solicited applications from individuals, groups and communities that were then evaluated based on quality and expected impact. Thus, individuals, groups and communities self-selected their potential participation, whilst the success of their proposals further relied on assessment against a set of objective criteria.
As a result, from a researcher’s point of view, evaluating the impact of PEACE II is problematic. Of course, the purpose of PEACE II was to improve the lives of individuals in Northern Ireland, not to make it easy for researchers to quantify the programme’s impact. In this respect, I have no desire to gripe about such research complexities. Rather, I seek to contextualise how this approach seems to have influenced the achievements of a programme that spent €500 per person in the areas of its domain.
Money can’t buy love but can it buy PEACE?
In the case of the PEACE II programme, in asking individuals, groups and communities to come together to create applications that could, in turn, help build a better local future, a critical assumption was made. This assumption was that individuals in the areas deemed ‘weakest’ were equally capable of creating successful proposals as those in areas that were deemed ‘stronger’. To put it another way, there was no guarantee that, should this assumption fail, PEACE II spending would make it to the areas that needed it most.
One could argue that, in identifying the strongest proposals, opportunity of progress was maximized. However, this still relies on the money getting to where it was most needed. Our recent research does not show this to be the case. Although individuals in the areas that received funding exhibit higher perceptions of neighbourhood quality than individuals in those that did not, deeper analysis showed that these inflated perceptions were pre-existing.
This suggests that areas with the weakest social capital appear to have been less successful at submitting fruitful applications. The effects of this process are astounding—the correlation between deprivation and receipt of spending, for example, is in the low range, yet we know that the areas in Northern Ireland most affected by violence in the past remain the most deprived today. Such deprived neighbourhoods are the areas where PEACE II was hoped to have the greatest impact, yet it seems that spending did not reach them.
It is, of course, hard to argue against the theory of well-targeted spending, especially when resources are limited. That said, the practicalities and outcomes in Northern Ireland do not match such theoretical expectations. In part, this relates to the funding mechanism itself but a deeper, and more concerning, failing may lurk in the background—a failure to effectively identify the areas that most required assistance; or, at least, a failure to ensure that spending made it to the areas identified as most needy.
Scarcity and limitations of RCTs
At the heart of both RCTs and the open competition model of PEACE II is the notion of scarcity. At some stage, budgetary limitations become important in any decision-making process. Despite a budget of almost €1 billion, PEACE II funded only half of all applications that were received. Some communities received funding of €4.5m, while other locations received none. Here, the major difference between PEACE II and an RCT is not that some individuals who could have benefitted went without, but that PEACE II aimed to specifically target the areas most in need. Although, on the surface, this may overcome the ethical dilemma of choosing who receives a treatment, the process by which such decisions were reached in Northern Ireland seems to have suffered its own major and serious limitations.
More generally, these limitations require significant thought. Northern Ireland is a developed country with a wealth of reliable community-level data. If needy communities in Northern Ireland cannot be targeted accurately, what does this say about the processes by which the most needy communities in areas with less reliable data in the developing world are identified? While it is therefore noble to target spending where it is most needed, it is a path fraught with difficulties and its own ethical dilemmas.
The feasibility of successfully targeting spending is an incredibly important consideration. In cases where it is difficult to identify the most needy communities, the benefits of a targeted spend must be leveraged against the probability of the spending finding its way to the communities most in need. And in efforts to build peace and improve inter-ethnic cohesion, making such an identification remains complex, if not impossible. In this context, although perhaps still imperfect, RCTs may not in fact pose the moral dilemma they appear to at first sight.
This blog post is published as part of a collaborative partnership between SIPRI and Economists for Peace and Security (EPS).
ABOUT THE AUTHOR(S)