DUBLIN

De partida para Dublin, para defender um paper na Conferência da European Evaluation Society, centrado nas implicações para a avaliação decorrentes da crescente complexidade dos programas operacionais regionais com cofinanciamento comunitário, não deu tempo para rever algumas leituras das atmosferas dublinescas. O intrigante DUBLINESCA de Enrique Vila-Matas ou o turbulento e sempre mutável ULISSES de Joyce seriam leituras recomendáveis. Não deu. Deixo-vos com as conclusões do paper, malfadadamente em inglês, pois o conhecimento tem uma língua, seja ela de terminologia americana ou britânica e não adianta muito esperar por outra alternativa.

Some policy and evaluation-oriented conclusions

Closing the paper, the conclusions also put forward policy and evaluation-oriented recommendations, based on the need to develop more intensive links between the design and evaluation of programmes. Difficulties in achieving credible and rigorous evaluations tend to reduce the effectiveness of programmes and increase the risks of failure faced by policymakers.

Albeit the relevant contribution theory-oriented evaluations has made to minimizing evaluation risks and challenges generated by the increasing complexity of ROPs, as discussed in this paper, it seems clear that it is more and more difficult to build a sound evaluation of ROPs as a whole, encompassing the wide diversity of actions and policy instruments employed to achieve the targeted outcomes. To address this problem, the main contribution of theory-based evaluations, particularly rooted in the strong involvement of stakeholders participating in the programme’s design, is the validation of a theory of change for the ROPs, complemented by the identification of the mechanisms linking the actions and policy instruments and targeted outcomes, considered as the supporting conditions required for these mechanisms and the implementation context to work. The validation of a programme theory does not mean that the evaluation of the whole programme, encompassing all its complexity, is feasible. In our experience, the validation of a programme theory is always combined with the identification of critical programming factors (CPF). CPF are critical not only from the perspective of the strategic monitoring of the ROPs, but also because of the need of policymakers and programme managers to have an in-depth knowledge of the main mechanisms through which the ROPs’ strategic outcomes could be achieved.

The increasing complexity of the rationale underlying the ROPs is not only endogenous, derived from the complexity of targeted outcomes. In the new 2014-2020 programming period, complexity is also strongly determined by the high number (eleven) of strategic thematic objectives of the Europe 2020 strategy. An in-depth analysis of how national administrative bodies have reacted to the 2020 thematic objectives and the negotiation of the Partnership Agreement shows that a huge number of investment priorities and policy instruments were channelled towards the ROPs in order to increase the probability of accessing the ESIF. As we know, to tackle the complexity of regional weaknesses through an increase in the number and diversity of policy instruments is not necessarily the most appropriate answer.

Given the fact that a vast number of policy instruments do not have the same weight in terms of funding, the likely trend will probably be the deconstruction of the idea of evaluating the ROPs as a whole. The evaluation object will tend to be divided into blocks and probably focused on the strategic dimensions of programming. The identification of the CPF that theory-oriented evaluations tend in our experience to achieve could be very helpful in guiding policymakers and programme managers to select the evaluation object blocks.

This means that, in the future, ROPs cannot really be seen as integrated programmes. As a corollary of this trend, the division of evaluation objects into blocks requires policymakers and programme managers to lead meta-evaluation exercises in order to not only try to compensate for the stratification of the evaluation object, but also to allow for a more comprehensive view of the effects of the ESIF in the regions, which are not limited to the ROPs’ effects.

Finally, the paper concludes that the combination of quantitative analyses (namely counterfactual exercises) with the qualitative dimensions of identifying and reconstructing mechanisms linking actions and targeted outcomes seems to be a very powerful means to address complexity in evaluating ROPs. As far as counterfactual exercises are concerned, our experience shows that a selective strategy to mobilize them is necessary. The good practice here is to carry out these exercises only if feasible information conditions are met. There is no experience in Portugal of carrying out experimental evaluations before the implementation of policies. In this context, conducting ex-post counterfactual exercises is very time and resource-consuming.