De partida para Dublin, para defender um paper na Conferência da European Evaluation Society, centrado
nas implicações para a avaliação decorrentes da crescente complexidade dos programas
operacionais regionais com cofinanciamento comunitário, não deu tempo para
rever algumas leituras das atmosferas dublinescas. O intrigante DUBLINESCA de
Enrique Vila-Matas ou o turbulento e sempre mutável ULISSES de Joyce seriam
leituras recomendáveis. Não deu. Deixo-vos com as conclusões do paper, malfadadamente em inglês, pois o
conhecimento tem uma língua, seja ela de terminologia americana ou britânica e
não adianta muito esperar por outra alternativa.
Some policy and evaluation-oriented conclusions
Closing the
paper, the conclusions also put forward policy and evaluation-oriented
recommendations, based on the need to develop more intensive links between the
design and evaluation of programmes. Difficulties in achieving credible and
rigorous evaluations tend to reduce the effectiveness of programmes and
increase the risks of failure faced by policymakers.
Albeit the
relevant contribution theory-oriented evaluations has made to minimizing
evaluation risks and challenges generated by the increasing complexity of ROPs,
as discussed in this paper, it seems clear that it is more and more difficult
to build a sound evaluation of ROPs as a whole, encompassing the wide diversity
of actions and policy instruments employed to achieve the targeted outcomes. To
address this problem, the main contribution of theory-based evaluations,
particularly rooted in the strong involvement of stakeholders participating in
the programme’s design, is the validation of a theory of change for the ROPs,
complemented by the identification of the mechanisms linking the actions and
policy instruments and targeted outcomes, considered as the supporting
conditions required for these mechanisms and the implementation context to
work. The validation of a programme theory does not mean that the evaluation of
the whole programme, encompassing all its complexity, is feasible. In our
experience, the validation of a programme theory is always combined with the
identification of critical programming factors (CPF). CPF are critical not only
from the perspective of the strategic monitoring of the ROPs, but also because
of the need of policymakers and programme managers to have an in-depth
knowledge of the main mechanisms through which the ROPs’ strategic outcomes
could be achieved.
The
increasing complexity of the rationale underlying the ROPs is not only
endogenous, derived from the complexity of targeted outcomes. In the new
2014-2020 programming period, complexity is also strongly determined by the
high number (eleven) of strategic thematic objectives of the Europe 2020
strategy. An in-depth analysis of how national administrative bodies have
reacted to the 2020 thematic objectives and the negotiation of the Partnership
Agreement shows that a huge number of investment priorities and policy instruments
were channelled towards the ROPs in order to increase the probability of
accessing the ESIF. As we know, to tackle the complexity of regional weaknesses
through an increase in the number and diversity of policy instruments is not
necessarily the most appropriate answer.
Given the
fact that a vast number of policy instruments do not have the same weight in
terms of funding, the likely trend will probably be the deconstruction of the
idea of evaluating the ROPs as a whole. The evaluation object will tend to be
divided into blocks and probably focused on the strategic dimensions of
programming. The identification of the CPF that theory-oriented evaluations
tend in our experience to achieve could be very helpful in guiding policymakers
and programme managers to select the evaluation object blocks.
This means
that, in the future, ROPs cannot really be seen as integrated programmes. As a
corollary of this trend, the division of evaluation objects into blocks
requires policymakers and programme managers to lead meta-evaluation exercises
in order to not only try to compensate for the stratification of the evaluation
object, but also to allow for a more comprehensive view of the effects of the
ESIF in the regions, which are not limited to the ROPs’ effects.
Finally,
the paper concludes that the combination of quantitative analyses (namely
counterfactual exercises) with the qualitative dimensions of identifying and
reconstructing mechanisms linking actions and targeted outcomes seems to be a
very powerful means to address complexity in evaluating ROPs. As far as
counterfactual exercises are concerned, our experience shows that a selective
strategy to mobilize them is necessary. The good practice here is to carry out
these exercises only if feasible information conditions are met. There is no
experience in Portugal of carrying out experimental evaluations before the
implementation of policies. In this context, conducting ex-post counterfactual
exercises is very time and resource-consuming.