Recently Diane Hird met with LNL Dr Alexis Makin to learn more about his work and open research at Liverpool. This blog is the result of that conversation, huge thanks to Alexis for his time.
The Liverpool Open Research Team are based in the Library and provide services to support the incorporation of open research principles as early as possible within the research process. This can include data management plans, data sharing, protocols, methodology, code, open access and preprints. The team have run a highly successful Open Research Week for several years, this year’s event is in late February and will feature several LNLs and a UKRN Roadshow as well as many other events and activities.
The route into reproducibility
Alexis is a researcher and lecturer in the Psychology Department within the Institute of Population Health at the University of Liverpool. He first became interested in reproducibility at the start of COVID-19 when research was paused and he was unable to do lab work. Instead he spent time learning about metascience and scientific methodology, for example Dorothy Bishop’s commentary on the four horsemen of irreproducibility – publication bias, low statistical power, p-hacking and HARKing (hypothesising after results are known). In this influential piece, Dorothy states that ‘many researchers persist in doing research in a way virtually guaranteed not to deliver meaningful results’ and Alexis wondered…could this apply to me? How good is my lab? To find out, his group made their project data open and used it to audit their own scientific performance. His approach to research has changed in light of the replication crisis. He now publishes fewer papers because he wants every paper to be much more rigorous.
The LNL role
Alexis took on the LNL role a couple of years ago. He likes the freedom associated with the LNL role, that he can take the spirit of UKRN and do what he can with it. There isn’t a Local Network mailing list, instead he shares information with the whole institute and within departmental newsletters.
His main job is running the ReproducibiliTea journal club. Slides from the club are available on OSF. ReproducibiliTea events have around 7 – 20 attendees, depending on the topic. Sometimes they pick a paper and discuss it, sometimes they invite a speaker on a particular topic. Alexis has run a half day tournament called ‘‘If you could do just one thing to fix research what would it be?’ sponsored by Methods NorthWest involving researchers across the institute and other local universities (more on this later). He also attends the annual LNL Retreats.
Liverpool is part of the UKRN Open Research Programme and has both an Institutional Lead (IL), Professor Bill Greenhalf, and an Open Research Coordinator and Administrator (ORCA), Dr Alice Howarth. Alexis meets with them both and also with the Open Research Team. He is also involved in the Open Research Week. He has also a taken on a (workloaded) role of Research Integrity Lead within the Institute.
All new researchers are encouraged to complete mandatory online training units in for example EDI, and GDPR. At Liverpool these are provided Epigeum. The research integrity unit includes links to open research. However mandatory training doesn’t necessarily mean people buy into it, Alexis wants researchers to embrace the spirit of open research not just follow the letter of it.
How to increase research trustworthiness – gather more, publish less
It’s relatively easy to put data on OSF. Alexis worries that if we simply focus on open research rather than experimental design we risk continuing to create untrustworthy literature where people think the issue has been addressed simply because every paper has a link to a dataset on OSF. The key to increasing trustworthiness is to gather far larger samples and publish far fewer papers.
Harking and low statistical power and p-hacking and publication bias massively undermine the trustworthiness of the published scientific record. As he puts it, people p-hack because it’s a performance enhancing drug that offsets the cost of doing underpowered experiments. Alexis believes that 90% of the problems in reproducibility would be removed if researchers use highly powered statistics and prioritise doing research in a way that is more likely to find real effects. For example if every cell in an experimental design requires 100 participants. Researchers don’t need to fully understand the technical arguments about priors and positive predictive values, they can simply use a much larger number of participants than is usually considered OK.
Teaching
Alexis teaches a Masters module in Research Methods and has embedded more open research content into his teaching including two classes on research design and open research. His aim is to promote trustworthy research rather than open research. For example students are set a grant proposal assignment with a large component on research design. Students enjoy the idea that they are going to be the first generation doing research ‘properly’.
His final year undergraduates preregister their projects using As Predicted although he doesn’t think this is widely done at Liverpool. Other supervisors can see it as just another thing for students to do. He would like to see preregistration included in the mark scheme for final year projects – is there a preregistration? Does the analysis follow the preregistration?
Harnessing the power of UKRN grassroot networks
Alexis feels that one of the most powerful things UKRN can do is make people aware of the replication crisis within new academic disciplines aside from Psychology. To sow the seeds of discontent. For example, UKRN could sponsor work to replicate individual but very influential papers in a field, or large scale replication projects on 100 accepted findings. Disciplines wake up when a paper says ‘we can’t replicate your key texts’. This forces disciplines to change and improve their research practices. We could harness LNL expertise here. LNLs could work with their Local Networks to identify textbook claims and secure knowledge which, if found to be unreplicable, would shake their field.
Dream big
Asked to dream big, Alexis would like to see a pivot away from publication with a small number of high quality studies instead. For example a 10 fold reduction in the number of papers published each year where each paper involves 10 times more participants. This in itself isn’t open research although the data and analysis would be made available. He acknowledges that the career incentives and mindsets would need to change, but that’s dreaming big!
An activity you could do – run a Reproducibility Tournament
Like Alexis, you could run your own Reproducibility Tournament. This half day event is an engaging way to talk about open research. Invite 10 people to talk about a research challenge that is close to their heart. If anyone else wants to join/speak on the day they can, they don’t need to send an abstract ahead of time. Encourage persuasive arguments and solutions rather than just a whinge. Examples at Alexis’s event included researchers describing their own activities – this is what I do and why I think others should do it too. That researchers should preregister exploratory analyses of secondary datasets. Alexis’s presentation was entitled ‘Fewer papers – more participants’. After each talk there is a general discussion. Often people would give reasons why something couldn’t be done, and that creates opportunities to discuss those obstacles. Then everyone votes for the most persuasive case. There can be prizes and photos, and it’s an opportunity to promote open research across the local institution or regionally.