Goodreads, Peer-Review, and Bibliometrics

Do you know of Goodreads?

It is strongly linked in my mind with the peer-review process due to a historical accident. I was at a conference a while ago and saw a presentation by a very enthusiastic, articulate, intelligent student early in their PhD. Unfortunately, the main characterisation method they were using was totally unsuitable for the system they were investigating, and the mechanism they were using to explain the chemistry that was happening (previously published in their group) was completely bogus. I made the deduction that their supervisor was clueless. It didn't seem either kind or appropriate to point these things out in the question period, so I thought I would try and catch up with them later at the conference. I missed them there, so when I got home I thought I would try to find them on the interwebz. The only trace of the student I could find was on Goodreads, so I made a Goodreads account and tried to contact them that way. But they obviously weren't paying attention to their Goodreads account, or else had a policy of not replying to private messages from people whose avatar was a picture of a bunny with a pancake on its head. So that didn't work either. But I ended up with a Goodreads account sitting there not doing anything.

There may be a happy ending to this story, in that I ended up writing to their supervisor with my concerns. I say may, because in the understandable reaction of most people to smart-aleck random pedants, he never wrote me back.

Eventually, I can't remember why, I wandered idly back to Goodreads and had a look around. There is a wealth of information about any particular book in Goodreads: you can see how many people have rated it, their average rating from 1 to 5 stars, and read their individual reviews on the book, many of which are very detailed and intelligent; you can in turn 'like' the reviews if you find them helpful, and explore other reviews the same reviewers have made to let you discount whatever biases they will invariably have. So, vastly more information than the reader of an academic paper can ever get from Bibliometrics. And vastly more information - often - than the writer of an academic paper can get from Peer-Review.

I don't have any ironclad mappings between Goodreads and Peer-Review, or Goodreads and the Bibliometrics Circus, but peering at them pairwise reveals similar pitfalls.

I'll start out with something fairly positive. I read this book not knowing it was an internet phenomenon. My wife recommended it, and she shies away from anything that is too popular, so she wouldn't have recommended it if she had known it was an internet phenomenon. It has a few rough edges, a few slow bits, but it is really pretty good. However, since it was an internet phenomenon, it has more reviews, and more positive reviews, than a lot of really top-notch stuff. Why is it an internet phenomenon? It hit a niche that had just opened. High-flying sci-fi authors of the 90s and 00s sell their e-books for practically the same price as the dead tree ones. Older high-flying sci-fi authors have pathetically limited e-book back catalogues. There is a huge mass of cheap self-published sci-fi that ... needs work. The critical niche, what sci-fi readers wanted, was at least halfway decent cheap new sci-fi e-books. This book was one of very few of those: so off it went. The same sort of thing happens with scientific publications all the time: it's not the fundamental importance of the research, or how well-argued it is, but how many other researchers are looking for something halfway decent in that niche at that particular moment in time.

Conversely, this book was very heavily excerpted and promoted on a website that gets a huge number of hits and is politically influential in the Old Country. Yet the only review of it is by - er - me. Sure, we can say, the demographic of Goodreads users must not overlap much with the demographic of 'National Review Online' readers. Maybe, pulling a number off of Goodreads isn't the best way to compare the 'influence' of a book. That seems pretty obvious.

But... isn't that the way bean-counters approach journal impact when they try to pull out one number to compare how oncologists use oncology journals, art historians use art history journals, organic chemists use organic chemistry journals? There are more things you can do with a book than write a Goodreads review about it, and there are more things you can do with a journal article than cite it.

Now for an ugly Peer Review analogy.

Take this book. The top-rated community review, with 20 likes, is from someone who admits that they haven't read it. But they have a claque of mates who resent other things the author has written, and they are all out to get him.

I don't think I've ever had an academic reviewer anything like this. But if I did, they would be anonymous. And they wouldn't admit that they hadn't actually read my manuscript.

Or maybe they would admit it, but no one would care. This example isn't anywhere near as bad, but it did happen to me. A colleague and I submitted a review article to a journal with an impact factor in the 1.5-2.0 range. It was rejected out of hand on the basis of one reviewer, who quite rightly caned us over some serious mistakes we made in the introductory section. Which is good, that's how it's supposed to work. Except, this reviewer then said - before they got to the bit where (I hope) we actually knew what we were talking about: 'I can't be bothered reading any more of this' and gave up. We patched the paper up and sent it to another journal and it has had 13 citations in the past year. So the first journal kind of missed out.

[UPDATE: As of January 2015, it had 112 citations, and was the most cited paper over the period 2008-2013 of any academic in my School.]

I am sure I could peer into the murky tea-leaves of Goodreads some more and come up with all sorts of other similarities. But that's enough for now. Maybe instead I should run some numbers from Goodreads through a spreadsheet and add some educational/bibliometric jargon and write a paper about it. Hmm.