A conferência da European Evaluation Society começa a ser uma realidade gigantesca,
com uma organização já bastante complexa e envolvendo um número prodigioso de
sessões paralelas, para além das plenárias dedicadas aos grandes expoentes da
ciência da avaliação. Tendo em conta esta dimensão de conferência e o facto de
em Portugal continuar a não existir uma sociedade de avaliação (o que muito se
deve a manobras de poder de gente proveniente do ISCTE que não entendeu o que é
uma comunidade de práticas), compreende-se o estado incipiente da prática e das
ciências da avaliação em Portugal.
Regra geral, à medida que as conferências
aumentam de dimensão e complexidade, o número de participantes (assistentes)
nas sessões paralelas tende drasticamente a diminuir e o risco é frequentemente os apresentadores de papers o fazerem quase em circuito
fechado. Ontem, no meu caso, isso não aconteceu e com 3 intervenientes e 20
minutos para cada apresentação tivemos cerca de 25 assistentes o que é
excelente.
Deixo-vos os 12 slides que apresentei, destacando
que na discussão realizada, a ideia chave que ficou e que mais interessou os
intervenientes foi a ideia da identificação dos Fatores Críticos da Programação
como o grande produto da aplicação das avaliações baseadas na teoria dos
programas (theory-driven evaluations).
E sobretudo a ideia de constituir um material chave para a interação entre as
equipas de avaliação, as autoridades de gestão dos programas e os organismos
responsáveis pela conceção dos programas. Por isso valeu a pena.
Slide 1 – Estrutura da apresentação
1. Context
2. The
increasing complexity of the evaluation object
3.
Complexity of ROPs and the appeal of theory-oriented evaluations
4. Some
policy and evaluation-oriented conclusions
Slide 2 – O contexto da avaliação em Portugal
- EU-driven evolution path for evaluation in Portugal
- Public policies strongly dependent on ESIF support
- No other relevant sources for evaluation needs: education as an exception
- Incipient research on evaluation
- Ambivalent relationship
- Potential lock-in effects induced by EU framework: EES paper in 2004
- Regional development policy dominated by Regional Operational Programmes (ROPs)
Slide 3 – As razões da complexidade
- Twofold reasons for the increasing complexity
- Targeted situations: stakeholders, rationale and policy instruments
- Logical and bureaucratic programming framework
- Cohesion interventions: broad and comprehensive
- Stronger role of competitiveness policy instruments
- Scarcity of national funding
- A tendency to invoke all the EU 2020 11 thematic objectives
Slide 4 – Continuação
- Matching difficulties between bottom-up strategies led by regional bodies ...
- And top-down policy instruments led by ministry bodies
- Strong segmentation of ROPs
- A black box complexity problem
- Not always understood by management authorities …
- From the perspective of evaluation implications
- Rogers (2008): complication and complexity problems
Slide 5 – Súmula das razões da crescente complexidade
In sum, increasing complexity due to:
- Context in which stakeholders and target publics are embedded
- Design and bureaucratic framework of programmes
- Governance models
- Changing programming paradigms: evolution towards a results-based programming, for example
- Complexity problems different from those generated by integrated programmes
Slide 6 – As avaliações baseadas na teoria
- Practically 20 years of dealing with complexity problems in evaluation ...
- Capable of renewal in keeping with the complexity challenges
- The case of contribution analysis
- An explicit theory of change in programmes
- Well beyond a results chain or logical framework …
- More probabilistic and less deterministic models of causality
Slide 7 – Continuação
- How programmes (increasingly complex) can have a (hidden) theory...
- A challenge but also an opportunity regarding communication and interaction between the evaluation team ...
- And stakeholders, including the management authorities
- “Nothing as practical as a good theory” (Pawson)…
- “Not whether programs work but how they work” (Rogers)
- Ways to increase the receptiveness of stakeholders to discuss the theory of the ROP
Slide 8 – Teoria da programação
- The first step in building it: the role of the evaluation team (ET)
- Validation of the explicit theory of change : a product of the interaction between the ET, management bodies and policymakers responsible for the programme’s design
- Identification of Critical Programming Factors (CPF)
- Programme dimensions and instruments …
- Which strongly influence the expected results estimated by the explicit or implicit theory of change
Slide 9 – Exemplos
- Dealing with complexity
- Competitiveness dimensions of ROP targeted at SME
- In-depth qualitative analysis of mechanisms
- Quantitative (counterfactual) analysis
- Competitiveness dimensions targeted at territories…
- • More complex mechanisms
- • Mechanisms + supporting factors + context of implementation
Slide 10 – Conclusões 1
- Albeit the contribution of theory-driven evaluations ...
- ROPs hardly to evaluate as a whole
- Validation of a theory of change ...
- + mechanisms linking actions and policy instruments and targeted outcomes …
- + Critical Programming Factors
- Evaluation object divided into blocks and focused on stratgeic dimensions of programming
Slide 11 – Conclusões 2
- ROPs are not integrated programmes anymore...
- Meta-evaluation exercises are needed to compensate for the stratification of the evaluation object
- Critical Programming Factors as powerful instruments of strategic monitoring...
- Combination of selective quantitative (counterfactual) evaluations…
- With strategic theory-driven qualitative evaluations
Sem comentários:
Enviar um comentário