̽��ֱ�� of Cambridge - Danny Kingsley

Opinion: ̽��ֱ��science ‘reproducibility crisis’ – and what can be done about it

cjb250 — Mon, 20 Mar 2017 09:57:15 +0000

A survey by Nature revealed that 52% of researchers believed there was a “significant reproducibility crisis” and 38% said there was a “slight crisis”.

We asked three experts how they think the situation could be improved.

Open Research is the answer

Danny Kingsley, head of the Office of Scholarly Communication, ̽��ֱ�� of Cambridge

̽��ֱ��solution to the scientific reproducibility crisis is to move towards Open Research – the idea that scientific knowledge of all kinds should be openly shared as early as it is practical in the discovery process. We need to reward the publication of research outputs along the entire process, rather than just each journal article as it is published.

As well as other research outputs – such as data sets – we should reward research productivity itself as well as the thought process and planning behind the study. This is why Registered Reports was launched in 2013, where researchers register the proposal and how the research will be conducted, before any experimental work commences. It allows editorial decisions to be based on the rigour of the experimental design and increases the likelihood that the findings could be replicated.

In the UK there is now a requirement from most funders that the data underpinning a research publication is made available. However, although there are moves towards open research, many argue against the sharing of data among the research community.

Questionable findings are often hidden. Shutterstock

Researchers often write multiple papers from a single data set and many fear that if this data is released with the first publication then the researcher will be “scooped” by another research group, who will publish findings from similar data sets before the original authors get the chance to publish follow up articles – to gain maximum credit for the work. If the publication of data itself could be recorded as a “research output”, then being scooped would no longer be such an issue, as such credit will have been given.

One benefit of sharing data could be an improvement in its quality – as previous research has shown. And there have been small steps towards this goal, such as a standard method of citing data.

We also need to publish “null” results – those that do not support the hypothesis – to prevent other researchers wasting time repeating work. There are a few publication outlets for this, and a recent press release from ResearchGate indicated that it supports the sharing of failed experiments through its “project” offering. It lets users upload and track experiments as they are happening – meaning no one knows how they will turn out.

Psychology is leading the way out of crisis

Jim Grange, senior lecturer in psychology, Keele ̽��ֱ��

To me, it is clear that there is a reproducibility crisis in psychological science, and across all sciences. Murmurings of low reproducibility began in 2011 – the “year of horrors” for psychology – with a high profile fraud case. But since then, ̽��ֱ��Open Science Collaboration has published the findings of a large-scale effort to closely replicate 100 studies in psychology. Only 36% of them could be replicated.

̽��ֱ��incentive structures in universities and the attitude that you “publish or perish” means that researchers prioritise “getting it published” over “getting it right”. It also means that some, implicitly or explicitly, use questionable research practices to achieve publication. These may include failing to report parts of data sets or trying different analytical approaches to make the data fit what you want to say. It could also mean presenting exploratory research as though it was originally confirmatory (designed to test a specific hypothesis).

However, many psychology journals now recommend or require the preregistration of studies which allow researchers to detail their predictions, experimental protocols, and planned analytical strategy before data collection. This provides confidence to readers that no questionable research practices have occurred.

Erasing data: a questionable research practice. Shutterstock

Registered Reports has taken this further. But of course, once results are produced, isolated findings don’t mean much until they have been replicated.

I make efforts to replicate results before trying to publish and you’d be forgiven for thinking that replication attempts are common in science, but this is simply not the case. Journals seek novel theories and findings, and view replications as treading over old ground which offers little incentive for career-minded academics to conduct replications.

This has also led to the introduction of Registered Replication Reports in Perspectives on Psychological Science. This is where teams of researchers each follow identical procedures independently and aim to replicate important findings from the literature. A single paper then collates and analyses them to establish the size and reproducibility of the original study.

Although psychology is leading the way for improvements with these pioneering initiatives, it is certainly not out of the woods. But it has started to move beyond a crisis and make impressive strides – more disciplines need to follow suit.

This is a publication bias crisis

Ottoline Leyser, director of the Sainsbury Laboratory, ̽��ֱ�� of Cambridge

Reproducibility is a fundamental building block of science. If two people do the same experiment, they should get the same result. But there are many good reasons why two “identical” experiments might not give the same result such as unknown differences that have not been considered – and some exciting discoveries have been made this way.

So if a lack of reproducibility is itself not necessarily a problem, why is everybody talking about a crisis? In some cases poor practice and corner cutting have contributed to lack of reproducibility, and there have been some high profile cases of out and out fraud. It’s a major concern, but what is causing it?

In 2014 I chaired a project on the research culture in Britain for the Nuffield Council on bioethics, which was motivated by concerns about research integrity including over-claiming, rushing prematurely to publication and incorrect use of statistics. ̽��ֱ��main conclusions were that poor practice is incentivised by hyper-competition with overly narrow rules for winning.

There is an excessive focus on the publication of groundbreaking results in prestigious journals. But science cannot only be groundbreaking, as there is a lot of important digging to do after new discoveries – but there is not enough credit in the system for this work and it may remain unpublished because researchers prioritise their time on the eye-catching papers, hurriedly put together.

̽��ֱ��reproducibility crisis is actually a publication bias crisis which is driven by the reward structures in the research system. Various approaches have been suggested to address problems, such as pre-registration of experiments. However, the research landscape is highly diverse and this type of solution is only sensible for some research types. ̽��ֱ��most widely relevant solution is to change the reward structures. In the UK there is a major opportunity to do this by reforming the Research Excellence Framework (REF). Through the REF, public money is allocated to universities based on the “quality” of the four best research outputs, usually papers, produced by each of their principal investigators over approximately six years and it disproportionately rewards groundbreaking research.

We need reward for a portfolio of research outputs, including not only the headline grabbing results, but also confirmatory work and community data sharing, which are the hallmarks of a truly high quality research endeavour. This would go a long way to shifting the current destructive culture.

Ottoline Leyser, Director of the Sainsbury Laboratory, ̽��ֱ�� of Cambridge; Danny Kingsley, Head, Office of Scholarly Communication, ̽��ֱ�� of Cambridge, ̽��ֱ�� of Cambridge, and Jim Grange, Senior Lecturer in psychology, Keele ̽��ֱ��

This article was originally published on ̽��ֱ��Conversation. Read the original article.

Reproducibility is the idea that an experiment can be repeated by another scientist and they will get the same result. It is important to show that the claims of any experiment are true and for them to be useful for any further research. However, science appears to have an issue with reproducibility.

NIAID

Study of Human Immune Response to HIV

̽��ֱ��text in this work is licensed under a Creative Commons Attribution 4.0 International License. For image use please see separate credits above.

Yes

Licence type:

Attribution

Your Questions Answered on open access

Anonymous — Fri, 23 Oct 2015 14:18:08 +0000

Open access means making peer reviewed works freely available in digital form, so that anyone with internet access can use them, without financial, legal or technical barriers. It allows users to download, copy, print and distribute works, without the need to ask for permission or to pay.

To the mark the eighth annual Open Access Week, we asked what readers wanted to know about the initiative.

Why do we need open access? How can I use it? Is it better for the sciences or the humanities?

Lucy Montgomery: Open access is a powerful mechanism for widening access to knowledge and for increasing the impact of research beyond universities. Because it makes peer-reviewed scholarship free at the point of use, open access helps ensure people who need knowledge can access it, even if they can’t afford to pay for it.

Patients scouring the internet for the latest information about rare medical conditions, scholars in the developing world, and practitioners who want to apply evidence-based research to challenges they face every day, are just a few examples of groups who benefit from open access.

̽��ֱ��global shift to open access is being driven by a consensus that the public has a right to access publicly funded research outputs. Closed publishing models rely on recovering the costs of publishing research by selling access to it. This made sense in a print-dominated world, when the marginal costs associated with making and distributing physical copies of books and journals was high; it makes much less sense in digital landscapes where the costs of making additional copies of a work once it’s been published are very low.

̽��ֱ��global shift to open access is being driven by a consensus that the public has a right to access publicly funded research outputs. Gideon Burton/Flickr, CC BY-SA

Once a work has been made open access, it’s free for anyone in the world to read or download. This is a boon for anyone who has ever been frustrated by a pay wall, for teachers looking for resources that can be shared easily with students, and for scholars who hope their work will contribute to a wider body of knowledge.

Although open access has been faster to take off in the sciences, it also has important benefits for scholars working in the humanities: helping authors to share their work with the communities that they write both for and about, and making knowledge and ideas available to new audiences.

How can journals meet the costs of editing, typesetting, proofreading, website construction and management if they move from subscriptions to open access?

Keyan Tomaselli: One of the key blind spots in open access discussions is the cost it poses to publishers. Journals that are not funded by foundations or universities are financially vulnerable in an open access environment unless they start charging for publishing articles. This is because their “permissions income stream”, which are paid to journals through national copyright agencies when their articles are reproduced in student course packs, will dry up.

In this model, the burden of payment will shift from reader or library payment for downloads or subscriptions, to author or institution for articles to be published. ̽��ֱ��assumption that open access is free – after data charges are paid – is wrong because though readers can access articles for free, authors and their institutions will end up paying so journals can recoup their costs. Data charges relate to the cost of internet access and downloading.

Too often one forgets that such accessing of the internet has cost implications too. And then there are journal post-production costs, including online platform hosting, marketing, discoverability, and archiving, among other things.

Open scholarship includes open notebook, open data and open review as well as open access. What are more systematic and rigorous treatments of open scholarship?

It’s now possible to put a digital ‘stamp’ on different scholarly outputs. Gideon Burton/Flickr, CC BY-SA

Danny Kingsley: There’s an increasing amount of research and discussion about open scholarship about integrity and researcher support; research management; assumptions and challenges; and about how we capture what’s being produced in repositories.

But although the nature of research is changing profoundly, the current system still only rewards and recognises traditional publication. Opening up scholarship has multiple benefits: research claims can be verified, work doesn’t have to be repeated to recreate the data, and data can be analysed from other perspectives.

It’s now possible to put a digital “stamp” on different scholarly outputs, called digital object identifiers (or DOIs). This means a researcher can be cited when another uses their work, and receive recognition.

By having an “open process” in research, we can put digital stamps on all aspects of research, such as progress in thinking through an online discussion paper, for instance; new techniques; and approaches and experiments. These can themselves be cited and therefore rewarded, rather than only recognising traditional published outputs.

How do we ensure research published under open access continues to have a system of rigorous quality checks, such as peer review, that can cope with the enormous load of research looking for publication?

James Bradley: We can’t ensure rigorous peer review of research will be undertaken under open access. Not only that, we know for sure that the explosion of open access journals has allowed for the publication of not just bogus work, but also work that’s irrelevant or useless for scientific or the whole academic enterprise.

How do we know this? For starters, there was an infamous sting in late 2013 that revealed a nonsensical piece of research was accepted for publication by a large number of open access journals. Then, there’s the research showing the huge numbers of “predatory” journals, which are basically in it for the money. ̽��ֱ��academic or the academic’s institution pays for publication and the piece gets in, regardless of quality. That’s why so many researchers often get emails from start-up journals soliciting our work — for a fee. It’s all about profit.

There’s another form of quality control that transcends peer review and lies in the after-life of a publication. Gideon Burton/Flickr, CC BY-SA

To mitigate this situation, there’s the Directory of Open Access Journals, which is supposed to act as quality control. If you make it on to the list, then you are supposed to be reputable. But some of the journals that have made it to the list are, in fact, “predatory”.

But it’s false to assume that all research that makes it into a front-rank publication is great or that all work in pay-for-publication journals is junk. ̽��ֱ��peer review system has always had flaws. Ultimately, there’s another form of quality control that transcends peer review and lies in the after-life of a publication — the opinion of your peers.

And this can, to some extent, be measured by metrics through citation databases. But it’s also reflected in the status and reputation accorded by your peers. It was ever thus, and most definitely remains the best form of quality control.

To what extent does this issue go beyond the machinations of open access versus the nuances of what’s free and not free, to the problem of the role of the university in a world where capitalism and the internet frame much of what we do?

Tom Cochrane: Open access has three points of origin. These, in no particular order, are the interests of the researcher in greater exposure and readership; the distorted economics of the price of scholarly communication (as distinct from the true cost of academic publishing); and the fact that the internet has made open access possible in the first place.

Openness in access to research outputs, research data and research processes, enhances replication capability, and allows review. Gideon Burton/Flickr, CC BY-SA

As the debate about open access has matured, it has also become clear that greater openness can also provide protection against research fraud or dishonesty. Openness in access to research outputs, research data and research processes, enhances replication capability, and allows review.

Open access has no particular correlation or causal relationship with the broader role of universities, other than to improve the efficiency and integrity of research and to increase the likelihood of greater integration with their various communities. It’s certainly true that we wouldn’t have seen it develop without the internet and, as such, the movement is another case of innovation and disruption of legacy models.

Where are we getting with the movement, year to year? How much concrete progress has there been as opposed to awareness raising?

Virginia Babour: There’s no doubt that the open access has come a long way. There are now mandates for open access in many countries and institutions globally.

These mandates vary in what they require. Some, like the one in the United Kingdom, are primarily supported through publication in open access journals. Others, like Australia’s funding councils' mandates, are via deposition of an author’s research in university repositories.

There’s also been an explosion of different technologies around open access, including new ideas on what can be published - just parts of articles, such as figures, fir instance – and new models for publishing open access books.

Finally, the infrastructure to support open access is developing with licenses for publishing, which lay out clearly how articles can be used. And identifiers for people and documents (even parts of documents), so there can be better linking of scholarly literature.

Open access is an evolving ecosystem. There will be different models to fit different specialities and probably different countries. But that’s fine if it works.

Virginia Barbour, Executive Officer, Australasian Open Access Support Group, Australian National ̽��ֱ��; Danny Kingsley, Executive Officer for the Australian Open Access Support Group, ̽��ֱ�� of Cambridge; James Bradley, Lecturer in History of Medicine/Life Science, ̽��ֱ�� of Melbourne; Keyan Tomaselli, Distinguished Professor, ̽��ֱ�� of Johannesburg; Lucy Montgomery, Director, Centre for Culture and Technology, Curtin ̽��ֱ��, and Tom Cochrane, Adjunct Professor Faculty of Law, Queensland ̽��ֱ�� of Technology

This article was originally published on ̽��ֱ��Conversation. Read the original article.

̽��ֱ��opinions expressed in this article are those of the individual author(s) and do not represent the views of the ̽��ֱ�� of Cambridge.

Meredith Kahn

Open access

̽��ֱ��text in this work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Images, including our videos, are Copyright © ̽��ֱ�� of Cambridge and licensors/contributors as identified. All rights reserved. We make our image and video content available in a number of ways – as here, on our main website under its Terms and conditions, and on a range of channels including social media that permit your use and sharing of our content under their respective Terms.

For image use please see separate credits above.

Yes

Licence type:

Attribution-Noncommerical