Soft peer review
Sun Dec 21, 2008
The following paper "Soft peer review: Social software and distributed
scientific evaluation" was passed along to me by alf today. I think
another copy has been haunting my file system for a few days, but this
seemed like a good reason to sit down again with it.
It's by Dario
Taraborelli and the abstract is as follows:
Abstract: The debate on the prospects of peer-review in the Internet age and the
increasing criticism leveled against the dominant role of impact factor
indicators are calling for new measurable criteria to assess
Usage-based metrics offer a new avenue to scientific quality assessment but
face the same risks as first generation search engines that used unreliable
metrics (such as raw traffic data) to estimate content quality. In
this article I
analyze the contribution that social bookmarking systems can provide to the
problem of usage-based metrics for scientific evaluation. I suggest that
collaboratively aggregated metadata may help fill the gap between traditional
citation-based criteria and raw usage factors. I submit that bottom-up,
distributed evaluation models such as those afforded by social bookmarking
will challenge more traditional quality assessment models in terms of coverage,
efficiency and scalability. Services aggregating user-related quality indicators
for online scientific content will come to occupy a key function in
and I get a mention in the acknowledgments, which is cool.
It is a very nice essay on the potential of social bookmarking as a tool for ran
king academic articles, in addition to adding metadata to scientific articles. D
ario discusses the issue of ranking the expertese of people who are bookmarking
and proposes a really nice method to get over the scaling problem that is inherr
ent when we try to intoduce manual methods to rank people. He suggestes that a u
sers notes and annotations could be made available about a bookmark on an anonom
ous basis. Others would have the option to copy these annotations, or rate them.
This would be a form of soft peer review on the annotations, which would in tur
n effect the standing of the person creating these annotations.
There would be ways to cheat this system, but with enough signal, one hopes that
such noise could be drowned out.
The paper also pointed out http://www.naboj.com/ which I'd not
seen before and which is
I really like this paper. Thanks Dario!