Trial by public, open peer review and the power of attention.

There are two very interesting recent examples of review by community on highly exciting results. They both share one very important characteristic, but stand in stark contrast to one another in almost every other regard.

The first is the [paper on making STAP cells via use of an acid bath] lemon juice.  If confirmed, the result is transformative. The result was published after peer review to tremendous fanfare in Nature. Very quickly the community tried to replicate the result with no confirming replication happening. A community effort has spring up to document evidence, and irregularities have been found in the paper. It’s starting to look like there is going to be a retraction.

The second recent example was the announcement, ahead of peer review, of the BICEP2 result. This is the first claimed observation that could provide evidence for a period of cosmological inflation in the early universe. As far as can be stated, if confirmed, a Nobel prize is up for grabs. Big news.

The BICEP2 group announced their results before their paper was peer reviewed, but the way they have done this looks like it is going to ensure that this paper will be reviewed in the open far more rigorously than via traditional means. At the same time they have gotten both the result and all of the underlying data out much faster than had they waited for formal peer review. They announced their result via press conference, and at the moment of announcement made the paper and underlying data freely available.

What both of these papers have in common is the ability to attract attention. As a result of this they are both receiving review in an open fashion in full view of the public. The difference however is that one was released before formal review and one after. The one that was released before formal review was done in a way to encourage further examination by the community. They did the following:

• making the data available at the time of announcement
• having people online at the moment of the announcement, ready to engage with questions, and in particular on facebook.

The authors of the other paper have been less forthcoming. It might be said that they have hidden behind the reputation of the journal that they published in.

Does this trial by public represent a new model of peer review? I think it certainly speaks towards how very highly contentious claims may no longer be able to rely on the existing format of peer review for confirmation. News travels instantly, and those who make such claims are well advised to be prepared for attention at scale. However when trying to argue that this will be a new model for peer review, the question of attention is critical. Most claims struggle to get the attention of even three reviewers Simply posting such items on to an archive will not ensure that they will be reviewed by more people than would have reviewed them in the traditional way.

If you do have a paper that is going to get a lot attention I think it’s also fair to say that you now should be highly confident in your result prior to publication. In the case of the BICEP2 paper I would expect that the preprint went through many rounds of internal review in preparation for this step. It seems that with the other paper even some of the coauthors were not fully up on all of the contents if the paper.

These two example stand in fascinating contrast to each other, and show how, and how not to, break news of extraordinary importance. What lessons we can take for the reviewing of the bulk of the literature is, in my kind, less clear.