Harvey et al. paper becomes the poster child of the reproducibility crisis in science

Josh’s ‘stupidest paper ever’ cartoon is a reminder of the depths to which some people will descend to silence the voices of rational scientific debate just as the authors of the 2018 Harvey et al. BioScience paper qualifies it for the honour of poster child of the reproducibility crisis in science.

By insisting — repeatedly and as recently as this month — that the information they released as supplementary data (find it here) was all that was used to construct the two figures in the paper (see Rajan and Tol 2018), the authors tacitly admit:

1) the data used to construct the figures were not derived from the methods they said they used

2) the work is not replicable

In other words, it is not science. That surely qualifies the Harvey et al. (2018) paper for the honour of poster child of the reproducibility crisis in science, discussed here and here (Baker 2016; Nosek et al. 2015). It’s an example the public will readily grasp, since the data said to have been used are easily accessible (and understood) by anyone who uses the Internet.

Below is the letter outlining the data release issue sent more than a week ago (18 May 2018) by Dr. Richard Tol, University of Sussex, to the president of the university highest academic authority in the Netherlands, where Jeff Harvey, lead author of the Harvey et al. 2018 BioScience paper, is employed as an academic [h/t R. Tol for the correction]. While Koninklijke Nederlandse Akademie van Wetenschappen [KNAW] in Amsterdam has a clearly stated data sharing policy, it is clearly not being upheld for the Harvey et al. paper — which nullifies the point of having a policy at all.

Professor Dr Jose F.T.M. van Dijck, Koninklijke Nederlandse Akademie van Wetenschappen [KNAW]
Postbus 19121
1000 GC Amsterdam
The Netherlands
president@knaw.nl

Falmer, 18 May 2018

Re: KNAW data policy

Dear Professor van Dijck,

In November 2017, Professor Jeffrey Harvey of the Netherlands Institute of Ecology, a KNAW institute, was the lead author on a paper published in the journal BioScience.

I requested the data behind the paper, and was pointed to the data archive. Unfortunately, the data released was incomplete.

The paper classifies blogs as “yellow” or “blue”. This classification is hardwired into the R code used to analyse the data. The paper only vaguely describes the procedure for classification: The content of the blogs is compared to the reports of the Intergovernmental Panel on Climate Change (IPCC). Unfortunately, the data released only contain the home page of the blogs investigated, rather than the specific blog posts examined. Some of these blogs consists of hundreds if not thousands of posts. Similarly, the IPCC has published many reports running to thousands of pages, with positions that change over time and chapters that contradict one another. Agreement or disagreement with the IPCC is therefore a meaningless concept, unless chapter and verse are specified.

I have repeatedly asked Professor Harvey and his co-authors to be more explicit, but to no avail. Specifically, I have asked, in vain, for the list of blogposts examined, for the IPCC statements assessed, and for the score per post and statement.

This may appear to be a detail, but it is a key contribution of the paper. Professor Harvey argues that a blog’s position on the IPCC predicts its position on polar bears. However, not knowing how the IPCC position is derived, it cannot be excluded that Harvey fitted the data to match the conclusion.

In the same paper, Professor Harvey and colleagues analyse the contents of a list of published papers. The released data specify the list of papers, but replicating their query to the Web of Science obtains a different list of papers. I have asked Professor Harvey, in vain, to specify the criteria by which certain papers were excluded; and to specify the query through which papers were added.

This point may seem mundane, but it is the other main conclusion of the paper. Professor Harvey claims that blue blogs agree with “the literature”, and yellow blogs do not.

Unfortunately, as Professor Harvey will not explain how “the literature” was delineated, the validity of this conclusion is unknown.

As indicated above, I have repeatedly asked Professor Harvey and his co-authors to release the full dataset. They have refused to do, suggesting that they have something to hide.

I have contacted Mr Melle de Vries, the head of information policy at the KNAW. KNAW data policy is clear: All data should be released, and exceptions specified. My experience with Mr de Vries was frustrating. He kept blandly repeating that all data had been released, which is not true as described above and repeatedly explained to Mr de Vries.

I apologize to have to bring this to your attention. KNAW data policy sets a good example: Scientific data should be open to outside scrutiny.

Unfortunately, KNAW researchers do not follow this policy, Ilj KNAW officers do not enforce it.

I would be grateful for your intervention in this specific case, and for your efforts to make sure that KWAW data policy is maintained.

Yours sincerely,

Richard Tol,
Professor, Department of Economics, University of Sussex

Professor, Institute for Environmental Studies and Department of Spatial Economics, Vrije Universiteit

Member, Academia Europaea

Research fellow, Tinbergen Institute

Research fellow, CESifo

Research fellow, Payne Institute for Earth Resources, Colorado School of Mines

Editor-in-chief, Energy Economics

There has not yet been a response (pdf here). Tol has posted the letter on his own blog as well.

In summary, by insisting that the information they released as supplementary data was all that was used to construct the two figures in the paper, the Harvey et al co-authors essentially admitted the so-called “science” component of their paper was bogus (see my April 2018 blog summary of the issues surrounding the paper here, in pdf format here).

Without a full release of the data, we must assume the figures presented in the paper were a smoke-screen. Such a fake veneer of science allowed the authors to enter into the scientific literature an unprofessional attack against my professional reputation and tar dozens of web-based writers with the same brush. Silencing legitimate criticisms of polar bear research and advocacy was surely the primary objective of this paper, although censoring this blog and others, including the most widely read climate-related website worldwide, WUWT might have been preferred.

Poster child of the reproducibility crisis in science seems a fitting legacy for this paper and a lasting honour for all 14 co-authors and the editors of the BioScience journal.

Get your commemorative mousepad!
Note: I have withdrawn the mousepads and shopping totes for sale over a dispute with Zazzle about my account setup. It wasn’t worth the hassle.