Evaluation strategy
A randomized experiment
Understanding if a given policy, or programme, has improved the welfare of its participants is key to determine whether it was effective or not. However, such evaluation faces a fundamental problem which is independent of the context under investigation and the data available. The quantitative evaluation of policy effects requires the comparison between outcomes for participants and outcomes that these same individuals would have experienced had they not been exposed to the policy. The latter quantity is called a counterfactual and, as such, can never be observed: hence a strategy has to be found to overcome this issue.
The most widely used approach to accomplish this goal is to identify a comparison, or control, group that did not participate in the intervention. The best way to construct a comparison group, and the one we follow in FORWORK, consists in randomly denying access to the services offered. A carefully planned and administered randomization ensures the absence of any systematic selection, meaning that participants and non-participants are statistically the same except that only one group has benefitted from the intervention. The absence of any systematic differences in the composition of “treatment” and “control” individuals allows the researcher to measure causal effects by a simple comparison of outcomes for the two groups.
Design trumps the complexity of the analysis. In other words, if the research design produces randomized treatment and control groups, one can argue that differences in unobservable characteristics have been taken care of, compared to a situation where two groups who are initially very different in terms of observed covariates get balanced via statistical methods.
The quantitative evaluation of the FORWORK project elaborated by the Fondazione Ing.
Rodolfo Debenedetti builds upon this very intuitive idea.
The unit of intervention
Interactions among asylum seekers are likely within the same centre, more than are across centres. On the one hand this may generate externalities among individuals in the same centre, and particularly so if its size is small (and this is the case in our setting). On the other hand, randomly denying access to individuals within the same centre may be politically sensitive and ethically not acceptable, and fraught with operational problems exactly because of social interactions.
Because of this, it is wise to consider reception centres, and not individuals, as the statistical unit of intervention. Put differently, we will implement a clustered design in which reception centres (and not single individuals) are randomized into “treatment” and “control” groups. All individuals within the centre will benefit from the intervention, but not all CAS participate in the experimentation. Interactions or spill-over effects from treated to control centres are of course possible. The evaluation design can somewhat control for these effects if, for example, sampling is conducted considering statistical units that are sufficiently far apart (see our discussion below on how CAS will be selected).
The outcomes
One of the main outcomes of interest is the probability of finding a job. We include in the definition of employment also apprenticeships and fixed-term contracts. This seems appropriate in the context of the Italian labour market, in which new entrants typically go through several temporary contracts before attaining some stability. We will measure employment status from administrative data. The latter have the advantage of being available both for the treated and for the controls.
At the same time, we recognize that labour market integration is a multidimensional concept, not limited to actual employment status; it also extends to knowledge of the local labour market, motivation, job search effort, and so on. We also plan to collect qualitative information along these additional dimensions through baseline and follow-up surveys. Even in this case, we will administer the survey both to the treated and to the controls.
