In itself, every word is neutral. But we feed them, we give them meaning, in the debate we seek to convince and come out convinced. And we also act.
Based on the speeches, individual decisions are made, and at the public level, policies are designed. Many times there are words that become fashionable, to the point that they lose their limits, or - diplomatically speaking - begin to obtain multiple meanings. So many that it is no longer possible to know what they define, except by investigating case by case.
For those who closely follow the debates on the formulation of public policies, a word - or in any case, a phrase - that has become fashionable, and that is gaining more and more strength, is the notion of Impact evaluation.
In very general terms, political decision makers are interested in determining what the impact of this or that measure will be.
After all, the effectiveness in the application of a public policy is expected to leave a mark on the electoral base of each candidate and, obviously, on the general well-being of the population affected by the measure.
However, there does not seem to be a general consensus that – for the moment – goes beyond the diffuse idea about the need to know the impact of “something” in socio-economic terms.
Fortunately, unlike other words that became fashionable, as happened with “globalization”; Impact evaluation is a set of techniques with a well-defined objective.
Furthermore, the knowledge that can be derived from its application is extremely interesting, both for policy makers and citizens. Let's see what this is about.
Why evaluate?
The impact evaluation seeks to establish the causal relationship which is established between the application of a policy, with a series of results. Since the latter are quantifiable, the application of this technique provides information on how effective a measure has been in terms of well-being.
In other words, it is a technique that allows us to establish whether the applied program is actually the cause of the changes observed in the population that participates in it, and not any other factor in the environment, or the behavior of the agents themselves under study.
In general, there is a misreading of the results produced by the application of a program or, more specifically, a public policy.
The most commonly used fallacy when interpreting data is to believe that correlation implies causation. What do you mean by that?
Take for example the relationship between education and income level. The graph shows the income level of the population with a level of completed university education for the Gran Rosario Agglomerate, separated by deciles.
By convention, the lowest decile shows the lowest level of income, while the X decile shows the highest level of income.
Although the concentration of university professionals in the highest income deciles is evident, in the analysis, a serious error would be incurred if one concludes, without further evidence than that presented, that “the more education, the better the income.”
Even worse would be the design of public policies that, no matter how well-intentioned, are based on false premises. Now, why could the above be a wrong conclusion?
The main reason is the following: that two phenomena develop in parallel does not imply that one is the cause of the other. It could well be the case that there is a third cause that explains both.
For example, it could be postulated that intelligent individuals (with all the difficulties that come with the use and application of this term) manage to achieve high levels of education, as well as high levels of income. So, in this case, intelligence would be the cause of the other two.
But another hypothesis could also be used to explain the same thing: individuals who come from rich homes achieve a higher level of future income, due to the social capital that belonging to that family implies, which also makes it easier for them to finish their university studies.
In general, the evidence indicates that the relationship between education and income is only a reflection of other causes, which cause both phenomena to run in parallel.
Impact evaluation techniques deal with this type of problems, organizing the information derived from the application of a program, trying to statistically emulate laboratory conditions. The same ones that are required, for example, to test the effectiveness of a medication.
How do you act in general terms? A minimum of two groups of individuals are generated, a control group and another treatment group, and they are compared. The control group is given a placebo; and to the treatment group, the policy measure (the medication).
The difference in the average of both groups of an identifiable variable provides a measure of the effectiveness of the treatment.
Of course the story is more complex. For example, you must ensure that all participants have an equal chance of participating in the program, regardless of the group for which they were selected (for example, through a lottery).
Likewise, the analysis will be more complex when it is observed that people change groups, or simply disappear from the sample.
The list of difficulties is more complete, and includes not only econometric analysis, but also analysis of the behavior of agents in behavioral terms.
However, the core of the problem remains, and is easy to understand: being able to compare two statistically similar groups, which differ in only one thing, whether they were beneficiaries - or not - of a measure.
In the case of the application of social policies, an experimental design can be generated in the implementation of a program, such that it allows generating an information structure from which its effectiveness can be derived.
Of course, beyond the different econometric techniques that may exist to attack a type of problem, each evaluation must be carried out according to the needs of the target population and the control, logistics, measurement, monitoring and audit capabilities of the agency that applies it.
Several points arise from the above:
- That impact evaluation can be expensive.
- That not every program is worth evaluating.
- That these types of studies are intensive in the use of disaggregated statistical information. That is, administrative statistical information, such as that provided by organizations such as INDEC, can rarely be used.
On the other hand, in the positive aspects, the impact evaluation allows:
- Determine if a program is having the desired effects on the target population.
- Identify which components of a program are most important to produce an impact.
- Test alternative programs to establish which one has the greatest impact on the target population.
- Determine if the results of a program can be replicated in different contexts.
- Efficiently implement the use of the resources allocated to the program, especially if the financing comes from agencies that tend to manage their organization through results.
In this way, with the use of this technique we seek to answer questions such as:
- Does the intervention have the expected effects on the beneficiaries?
- Are some beneficiary groups more affected by the intervention than others?
- To what extent did the problem improve?
- Are there unplanned adverse effects?
- Are the effects maintained over time?
An example: NGO TECHO
The NGO CEILING works in precarious settlements, to generate concrete solutions to the problem of poverty. Specifically, the community development processes undertaken by the NGO consist of the participatory construction of emergency housing, in which both volunteers and families from the community are involved.
The idea behind TECHO is to provide housing that constitutes a concrete and achievable solution in the short term, with an impact on the quality of life of families.
Recently, the Abdul Latif Jameel Poverty Action Lab (J-PAL) measured the effects of these actions on their beneficiaries. The study analyzed 896 families in Mexico, 698 in El Salvador and 779 in Uruguay.
Two groups were considered, a control group that maintained its precarious housing (given that there was excess demand) and a treatment group, which was able to obtain emergency housing.
The research extended between 18 and 27 months per country and considered two survey periods and other tests, to identify the impact it could have on the following aspects: materiality of the home, satisfaction with the quality of life and perception of security, as well as such as health, access to goods, income.
Thus, the objective was to measure the impact on aspects such as satisfaction with quality of life and perception of security, as well as health, access to goods and improved income.
In the first case, satisfaction with quality of life increased significantly, according to the surveys carried out, between 21,1% and 41,7% compared to the control group.
However, according to the same study, no effect was detected on family income or asset accumulation, nor could it be robustly concluded that there were effects on health improvements.
What are the practical advantages of evaluation?
In the previous point, it is more than clear that a good impact study allows us to generate extremely useful knowledge in social terms. But what motivates its application for the formulator of a program, in addition to obtaining knowledge?
Currently, much of the applied economic literature is based on the use of these techniques, either through the identification of natural experiments or by their design to test research hypotheses.
Basically, the use of these econometric tools has become the international standard of good practices. Reason why, the application of a project that contemplates these characteristics would prioritize it by placing it within the map of academic production.
That is, it draws the attention of researchers who, with the same data, could make a novel contribution.
From the point of view of state, having this type of studies would allow us to accumulate knowledge about what works and what does not, in areas of public interest.
Especially in those considered sensitive and that require continuous and persevering work, but also results. An example of the above are anti-poverty programs.
In this regard, at the local level, much progress has been made in a multidisciplinary conception of the problem, but without it being very clear which of all the measures taken from multiple approaches are those that work and which do not.
From the point of view of citizenship, the contribution of these studies undoubtedly favors the improvement of the transparency of the democratic process, by providing a robust measure of the application of public treasury resources. Citizens would have access to an improvement in the quality of information.
Using the previous example, a management could publish an advertisement that does not just say something like: “10.000 homes were built.” Alternatively, it could be published: "the construction of the homes has represented a % improvement in the quality of life of the awarded families, although other measures are still required to achieve improvements in their health."
Finally, these practices represent an opportunity. Although impact evaluation programs are one of the most relevant areas in the public policy monitoring and evaluation systems in Latin American countries, the deficit is still large.
To know a bit more…
The list of publications that use this type of techniques is simply abundant. An extremely fun introduction to the econometric techniques on which these works are based (something that is not easy to achieve with these topics) is the popular book by Angrist and Pischke, not yet translated from English, “Mastering Metrics. “The Path From Cause to Effect”.
Another popular book, which aims to explain the results obtained in impact evaluation work applied to issues of the fight against poverty, is the book by Banerjee and Duflo, “Rethinking Poverty. “A radical turn in the fight against global inequality.” In it, the authors show the results of applying this approach to topics such as education, health, micro-savings, micro-credits and entrepreneurship.
To listen to Dufló herself talking about these topics, you can consult the following TED talk:
A local level, we also have our references, for example, Walter Sosa Escudero explains the foundations for thinking about causality in statistics.
Finally, there are a myriad of academic works that test their hypotheses using non-experimental techniques.
We are going to cite two works that have the characteristic of having been made by Argentine authors, and that deal with Argentine problems. Unfortunately, the lack of statistical information means that this is not so common.
The first study seeks to answer a question with a far from obvious answer: does police presence on the street contribute to reducing the crime rate? He second, has been the subject of debate a couple of years ago, and answers the following question: has compulsory military service favored higher levels of crime?
- Rafael Di Tella and Ernesto Schargrodsky (2004). “Do Police Reduce Crime? Estimates Using the Allocation of Police Forces after a Terrorist Attack.” American Economic Review 94, 115-133.
- Sebastián Galiani, Martín Rossi, and Ernesto Schargrodsky (2011). “Conscription and Crime: Evidence from the Argentine Draft Lottery.” American Economic Journal: Applied Economics 3, 119-136.