Authors: Thomas Aston; Alix Wadeson
There is increasing interest in learning about rigorous methods which can help to effectively assess complex change processes, such as Process Tracing and Realist Evaluation. However, these methods are often considered inaccessible to practitioners. We will present an innovative combination of Process Tracing, Outcome Harvesting, and evaluation rubrics called Contribution Rubrics.
This innovative method is designed to help teams assess their own outcomes, with limited training and support, by using SMART outcome statements, Process Tracing evidence tests and a structure for evaluative judgements through rubrics. It arose out of the real-world challenges from a learning project at CARE International in building capacity for applying Contribution Tracing in Ghana and Bangladesh and was recently tested in Côte d’Ivoire.
This session is underpinned by the paper Process Tracing as a Practical Evaluation Method: Comparative Learning from Six Evaluations in which we detail practical lessons from applying different forms of Process Tracing with project teams from several global organisations. The practical recommendations and tips provided offer substantial material that can help improve evaluators’ capabilities and demonstrate the value of more participatory evaluations.
We will share our experience of combining different theory-based evaluation tools to assess level of contribution and strength of evidence, through presenting a new method – Contribution Rubrics. We aim to offer practical advice and promote discussion on how theory-based tools can not only be better used, but also more accessible and participatory, without substantially diminishing rigour or potential causal inference.
The Centre of Excellence for Development Impact and Learning (CEDIL) recently identified Process Tracing as a method that has been insufficiently tested (Masset and White, 2019). Outcome Harvesting sessions at the American Evaluation Association Conference in 2018 were extremely well attended as were Rubrics sessions at the Australian Evaluation Society Conferences in 2018 and 2019. Therefore, we are confident of the session’s relevance and potential to promote cross-cultural exchange and a discussion of diverse perspectives on theory-based evaluation tools.
The session for the format “tools in use, changing tools” will focus on sharing practical skills, tools, and tips relevant for evaluators considering the application of process tracing but struggling to find practical ways to do so. This session on Contribution Rubrics can help equip evaluators conducting participatory and theory-based evaluators with more robust and accessible tools for their toolbox.
Link to theme
As Contribution Rubrics is an adaptation of three known methods it speaks well to the methodological challenges of adapting the evaluator’s toolbox. In particular, our focus is on the efficacy of the method in assessing complex change processes and “hard to measure” programming such as governance and advocacy interventions. This concern thus also matches the overarching theme of the conference Evaluation in an Uncertain World - Complexity, Legitimacy and Ethics.Contribution Rubrics available here: https://static1.squarespace.com/static/5555daace4b0bd68287c4b64/t/5e2f05199109395792b52938/1580139803081/Contribution+Rubrics.pdfProcess Tracing as a Practical Evaluation Method available here: https://www.linkedin.com/posts/tom-aston-consulting_process-tracing-as-a-practical-evaluation-activity-6648128535292583937-zGSw