quinta-feira, 2 de outubro de 2014

AINDA DUBLIN



A conferência da European Evaluation Society começa a ser uma realidade gigantesca, com uma organização já bastante complexa e envolvendo um número prodigioso de sessões paralelas, para além das plenárias dedicadas aos grandes expoentes da ciência da avaliação. Tendo em conta esta dimensão de conferência e o facto de em Portugal continuar a não existir uma sociedade de avaliação (o que muito se deve a manobras de poder de gente proveniente do ISCTE que não entendeu o que é uma comunidade de práticas), compreende-se o estado incipiente da prática e das ciências da avaliação em Portugal.
Regra geral, à medida que as conferências aumentam de dimensão e complexidade, o número de participantes (assistentes) nas sessões paralelas tende drasticamente a diminuir e o risco é frequentemente os apresentadores de papers o fazerem quase em circuito fechado. Ontem, no meu caso, isso não aconteceu e com 3 intervenientes e 20 minutos para cada apresentação tivemos cerca de 25 assistentes o que é excelente.
Deixo-vos os 12 slides que apresentei, destacando que na discussão realizada, a ideia chave que ficou e que mais interessou os intervenientes foi a ideia da identificação dos Fatores Críticos da Programação como o grande produto da aplicação das avaliações baseadas na teoria dos programas (theory-driven evaluations). E sobretudo a ideia de constituir um material chave para a interação entre as equipas de avaliação, as autoridades de gestão dos programas e os organismos responsáveis pela conceção dos programas. Por isso valeu a pena.
Slide 1 – Estrutura da apresentação
1. Context
 2. The increasing complexity of the evaluation object
 3. Complexity of ROPs and the appeal of theory-oriented evaluations
 4. Some policy and evaluation-oriented conclusions
Slide 2 – O contexto da avaliação em Portugal

  • EU-driven evolution path for evaluation in Portugal
  • Public policies strongly dependent on ESIF support
  • No other relevant sources for evaluation needs: education as an exception
  •   Incipient research on evaluation
  • Ambivalent relationship
  • Potential lock-in effects induced by EU framework: EES paper in 2004
  • Regional development policy dominated by Regional Operational Programmes (ROPs)

Slide 3 – As razões da complexidade

  • Twofold reasons for the increasing complexity
  • Targeted situations: stakeholders, rationale and policy instruments
  •  Logical and bureaucratic programming framework
  • Cohesion interventions: broad and comprehensive
  • Stronger role of competitiveness policy instruments
  • Scarcity of national funding
  • A tendency to invoke all the EU 2020 11 thematic objectives

Slide 4 – Continuação

  • Matching difficulties between bottom-up strategies led by regional bodies ...
  • And top-down policy instruments led by ministry bodies
  • Strong segmentation of ROPs
  • A black box complexity problem
  • Not always understood by management authorities …
  • From the perspective of evaluation implications
  • Rogers (2008): complication and complexity problems

Slide 5 – Súmula das razões da crescente complexidade
In sum, increasing complexity due to:

  • Context in which stakeholders and target publics are embedded
  • Design and bureaucratic framework of programmes
  • Governance models
  • Changing programming paradigms: evolution towards a results-based programming, for example

  • Complexity problems different from those generated by integrated programmes

Slide 6 – As avaliações baseadas na teoria

  • Practically 20 years of dealing with complexity problems in evaluation ...
  • Capable of renewal in keeping with the complexity challenges
  • The case of contribution analysis
  • An explicit theory of change in programmes
  • Well beyond a results chain or logical framework …
  • More probabilistic and less deterministic models of causality

Slide 7 – Continuação

  • How programmes (increasingly complex) can have a (hidden) theory...
  • A challenge but also an opportunity regarding communication and interaction between the evaluation team ...
  • And stakeholders, including the management authorities
  • “Nothing as practical as a good theory” (Pawson)…
  • “Not whether programs work but how they work” (Rogers)
  • Ways to increase the receptiveness of stakeholders to discuss the theory of the ROP

Slide 8 – Teoria da programação

  • The first step in building it: the role of the evaluation team (ET)
  • Validation of the explicit theory of change : a product of the interaction between the ET, management bodies and policymakers responsible for the programme’s design
  • Identification of Critical Programming Factors (CPF)
  • Programme dimensions and instruments …
  • Which strongly influence the expected results estimated by the explicit or implicit theory of change

Slide 9 – Exemplos

  • Dealing with complexity
  • Competitiveness dimensions of ROP targeted at SME
  • In-depth qualitative analysis of mechanisms 
  • Quantitative (counterfactual) analysis
  • Competitiveness dimensions targeted at territories…
  •    More complex mechanisms
  •    Mechanisms + supporting factors + context of implementation

Slide 10 – Conclusões 1

  • Albeit the contribution of theory-driven evaluations ...
  •  ROPs hardly to evaluate as a whole
  •  Validation of a theory of change ...
  • + mechanisms linking actions and policy instruments and targeted outcomes …
  •  + Critical Programming Factors
  •  Evaluation object divided into blocks and focused on stratgeic dimensions of programming

Slide 11 – Conclusões 2

  • ROPs are not integrated programmes anymore...
  • Meta-evaluation exercises are needed to compensate for the stratification of the evaluation object
  •  Critical Programming Factors as powerful instruments of strategic monitoring...
  • Combination of selective quantitative (counterfactual) evaluations…
  • With strategic theory-driven qualitative evaluations

Sem comentários:

Enviar um comentário