Evaluation Design

Lead

David Shanks (University College London)

Background and scope

The Programme aim is to accelerate the uptake of high quality open research practices. The Center for Open Science theory of change highlights five types of action that can promote change. This is a supporting project, that is focused on enabling institutions and UKRN to monitor changes in open research practice over time and between groups, to better target support and other interventions at all levels in the COS theory of change.

Aim

This project will build institutional and sectoral capacity in evaluating changes in the uptake of open research practice over time, by both providing evaluation tools and supporting their use.

Outputs

• WG4 Project Plan
• ORP Evaluation Design Report July 2023 (pdf format)
• Open and Transparent Research Practices survey
• Open Research Indicators

Goals

Intervention outputs and outcomes

1. By Mar 24, there is a reliable quantitative measurement of the baseline level of open research practice, leading to UKRN and institutions being able to measure progress against it by Aug 27 using reporting tools and dashboards.
2. By Sep 26, there is an agreed survey instrument and method, leading to UKRN and partner institutions being able to measure progress by Aug 27.
3. By Sep 25, indicators are used in at least 18 institutions that show levels for an agreed set of open research practices and so those institutions can identify and learn from those with high levels of practice, leading to them and UKRN being able to measure change related to their interventions by Aug 27.
4. By Sep 24, a framework published for testing efficacy of structured interventions in the context of institutional research improvement, leading to measurable improvements in the evaluation of interventions by Aug 27.

Evaluation objective and output

5. By Aug 27, report published providing evidence on the extent to which the project aim was met.

Description of Activities and their contribution to outputs and outcomes

1 Programme theories of change and reporting tools

Description of how each project and the programme overall will deliver outputs (the ‘intervention’ and its protocol), how these outputs lead to outcomes, and the relevant context and assumptions. They specify how each project and the programme as a whole will evaluate its outcomes, likely impact and value for money [A]. They are updated in a managed way through the programme as part of process evaluation. They are the basis for dashboards / reporting tools. Associated with them is a glossary.

2 Open research indicators pilots

Identification, testing and roll-out of a targeted and agreed set of indicators of open research practices [C], and their use to inform sharing of practice and to evaluate progress including via dashboards / reporting tools [A].

3 Open research indicators accountability

Identification of interests and responsibilities in the provision of indicators and underpinning data, to promote their transparency as an open research practice in the long term interests of the sector [C].

4 Survey

Development and refinement of best practice in the design and delivery of a survey of open research practices, agreement on a relevant instrument and approach to sampling and analysis, preregistration and deployment of this to measure progress [B], integration of results into dashboards / reporting tools [A].

5 Evaluation framework

Creation, testing and promotion of a framework by which institutions can test the efficacy of structured interventions in the context of institutional research improvement [D] [A].

Communications and engagement

Work on open research indicators will engage with institutions, funders, publishers and others to ensure that good practice is defined, agreed and adopted.

Page top