Some Thoughts on Peer Review and Altmetrics

The upcoming altmetrics meeting, and a submitted abstract by Kelli Barr prompted me to note down some of my own thoughts on peer review and altmetrics. I would love to make it over to the meeting, but with just a few days now before my first child is born, it ain’t gonna happen.

I’ve not read Kelly’s paper, but after reading the abstract my take home message from it would be something along the lines of “don’t replace peer review with altmetrics because you will just replace one bias with another, and at least with peer review the bias is contained within the academic community”

I personally don’t see any conflict between altmetrics and peer review. If anything, in a nutshell, I believe that altmetrics can help to create an augmented peer review, placing better information into the hands of those in the community who are making judgements on impact, on likelihood of future productivity, on who should get money.

The abstract starts with

The altmetrics community, according to its manifesto, has grown around the assumption that the use of peer review as a filtering mechanism for quality scholarship has outlived its usefulness in the changing landscape of scholarly communication (Priem et al. 2010).

I would classify myself as part of this community, but my interest has not grown out of the assumption listed here. The idea that strongly motivates me is the following

The web, and the processing power associated with it, should provide better information to those interested, about the impact of research, and the existence of relevant and related research to the question at hand to the researcher. In this way the web should be a tool to reduce informational asymmetry.

The best current example of how this does not happen is the continued use of the impact factor as a proxy for impact. I think that it’s a case of since we can do this, and we can make systems that support the research process, we damn well should. There are many branches of thought that follow on from this: supporting peer review, recommendation of articles, graphing the flow of ideas within the literature, extending information about relevance to communities outside of the main research community for example patients.

At it’s heart though, I’m a believer that as in other areas of human interaction, the web has the potential to help with some of the core activities of research, and not just by making it possible to read pdfs online.

The abstract continues

I argue that the altmetrics community should resist the attempt to supplant peer review with a host of altmetrics, no matter how diverse.

I totally agree with this. I think though, that peer review is not a single thing. It’s a description for a very variated process. When I was managing a number of journals I saw different types of peer review in process. Sometimes for small journals it consisted of getting the nod from the editor in chief. Sometimes it consisted of polling a large number of researchers where the work was felt to be potentially very problematic, leading in the end to the dismissal of the editor in chief (I’m talking here about different people). For some instantiations of what we could call peer review, I’m fully confident that a quite different approach could work just as well. PeerJ may be a move in this direction.

I’d make the analogy to code review. The reason for code review, within a programming organisation, is that humans are fallible. There are lots of different ways of doing code review, but at it’s heart, it’s always used as a sense check against human infallibility. I think this is one of the important aspects of peer review. Altmetrics can only help with this as by providing more information to the person doing the peer reviewing, there is a bigger chance that that person may be alerted either to potential errors, conflicts, or opportunities for interesting cross-pollination.

My feeling on the following point

Any evaluation scheme is simultaneously a system of incentives, and so assessing the impact of research according to a suite of altmetrics will inevitable steer research in particular directions, as peer review has done.

is that the current data metrics: citations and the impact factor, are simply just very crude. The impact factor is a shit sandwich, and the only reason we continue to swallow it is that we don’t have open reproducible metrics. The impact factor itself if neither of these things. I think that projects like the open citations are hugely important for helping to resolve this, and altmetrics itself. It’s early days, we have been talking about altmetrics since about the mid-2000’s, and early systems like post-genomic gave an early taste, but I think it’s fair to say that systems that could really start to do something interesting have only been in existence for a little under five years now.

but the judgments rendered will not be immune to the promotion of conventionality (i.e. groupthink, or cultural exclusivity on a larger scale), nor would it limit the volume of research published.

This is a fair comment, but my hope will be that by promoting a variety of metrics people will be able to pick the groupthink that they most like, rather than all having to stick with one. One could then imagine applying meta-analysis to application of altmetrics to uncover these biases.