On 5th December 2024 over 50 researchers from Bangladesh, China, Denmark, Mexico, Spain, Switzerland and across the UK took part in UKRN Replication Games. Led by Abel Brodeur, Derek Mikola, Juan Aparicio and Bruno Barbarioli from the Institute for Replication (I4R) the researchers were organized into 15 teams and worked to replicate 13 papers published in Psychological Science, Nature Human Behaviour, the American Economic Review or the American Journal of Political Science.
The event was held online with 2 groups working together in person. I4R were delighted with the turnout: this was their largest online Replication Games event in 2024. Following brief introductions, teams worked with discipline-specific templates to structure the work on their chosen paper, with the organisers checking in with them throughout the day. In the final session each team briefly fed back on their experiences. These varied widely with some teams able to access data and adequate code to reproduce the findings, and others unable to access data or code. Challenges included maps created using the data were similar but not the same as the originals, unlabelled data with collapsed values (for example, 7 data points listed but only 6 present in the corresponding chart), raw and clean data accessible but no code for how to clean the raw data, no code for a paper’s figures, and a lack of explanation within the code so having to troubleshoot.
The next step for each team is to complete and submit their report to I4R who will then work with the authors of the original paper, inviting them to respond to the replication reports. I4R also plan to include the reports in 3 meta-papers. The researchers can choose to be named on the initial reports and subsequent meta-papers. The author response and the I4R papers are released at the same time on the I4R Reports and Meta-papers webpage.
Following up with the five UKRN Local Network Leads (LNLs) involved, they all found the event an enjoyable or good experience although some were keen to point out that there is a significant time commitment both before (to meet as a team, choose and review a paper) and after (to write the report for I4R) the one day event. The LNLs said:
‘I greatly enjoyed learning how to do computational replications and robustness checks first hand, as this is not something I have done in great detail before.’
‘It was interesting to see how difficult it is to follow the logic of someone else’s code. It made me reflect on my own code.’
‘It showed me the value of commenting code and arranging things better.’
‘It was great working and collaborating with a team, and I took away key lessons on how to handle my own research.’
‘I think it is very valuable both for research and researchers.’
‘These events are fun, good for science, and a great educational opportunity.’
‘Gaining knowledge, collaboration, and even a publication is excellent.’
Thanks to everyone who participated in the event, we hope you found it useful. And demonstrating the power and reach of the LNL network, thanks also to everyone who promoted this event across their networks, the I4R team were amazed and very pleased by how many researchers signed up and were involved in the December event.