Google scholar is a wonderful way to measure the quality of scientific papers based on the number of citations. I actually find the number of citations so interesting that I display it in my list of publications (the yellow bars on the left correspond to citations).
I learned, for example, that one of my papers (which I thought was one of my better ones) never got cited (I'm guessing that's because there’s not too many people working on organizing drum sample libraries). In addition, I believe it might be useful for someone browsing my list of publications to identify which papers are probably more readable. But I'm not sure about the usability in its current form.
I also just learned that somehow my PhD thesis has made it into the top 30 results for the search term music information retrieval! :-)
Anyway, the point is that Google scholar as a form of evaluating the quality of papers is highly insufficient: the delay between having a final version of the paper and the point where reliable quality estimates can be made is way too large (often taking several years).
Given the limitations of Google scholar I’ve been thinking about what the future of the paper industry should be like (and in particular how the quality of a paper could be measured more quickly), and here are some things I’d love to see:
1. Papers should be publicly reviewed similar to the book reviews by Amazon’s customers. (Including the option to rate the usefulness of reviews.)
2. Researchers, research projects, and research teams should have blogs in which they present their findings and encourage open discussions and criticism of their work.
3. Researchers (and in particular students) should post their ideas on public sites to get instant feedback from peers (and document who came up with the idea first).
And here’s how I plan to contribute to the future:
I plan to publish some comments from reviewers, and my response for the ISMIR'07 paper for which I’m first author of (including a link to the paper and the demonstration video).
I plan to write reviews on papers that I read in the future. Maybe I’ll do so for some ISMIR'07 papers that are already online.
However, I’m also planning to spend most of my time in the next weeks helping improve how Last.fm's radio stations listen to their listeners. So it might take me a while to get my "Paper Industry 2.0" contributions started :-)
Btw, using A/B tests to evaluate algorithms on radio stations beats number of citations any time :-)
(And if you like A/B tests, you’re probably also a fan of Greg Linden’s blog.)
UPDATE:
I just read the two links posted in the comments and they are great!
The first link posted by Paul basically talks about how the Nature community is still very old school. The second link posted by Chris talks about how things will need to change in the future and why so many researchers are still very old school.
Subscribe to:
Post Comments (Atom)
4 comments:
It's funny that there's all this innovation going on right now, yet distribution, reviewing, etc. of papers hasn't really changed much at all. It seems strange that at a place like Google, where everyone comes has an academic background and must have run into the same nuisances you describe here, they haven't dealt with this type of thing. While the rest of the web is ever so 2.0, the paper industry (good term) probably hasn't reached 0.5.
And it's not just measurements of quality - even searching for papers could be much more advanced than it is.
I like the idea of Amazon like paper reviewing. Could make research a lot easier. "People who liked this paper also liked.."
Or are we going to see "The last paper repository you'll ever need" someday? ;)
I like the idea of open reviews... it would help me as a paper reader find better papers to read, and as a paper writer to improve my papers and as a paper reviewer, to improve my reviews.
Nature magazine had a trial open peer review. Their conclusions: Despite the significant interest in the trial, only a small proportion of authors opted to participate. There was a significant level of expressed interest in open peer review among those authors who opted to post their manuscripts openly and who responded after the event, in contrast to the views of the editors. A small majority of those authors who did participate received comments, but typically very few, despite significant web traffic. Most comments were not technically substantive. Feedback suggests that there is a marked reluctance among researchers to offer open comments.
Read more here:http://www.nature.com/nature/peerreview/debate/nature05535.html
There's an interesting article that touches on some of this here:
http://www.popsci.com/popsci/technology/f8a1780809ed3110vgnvcm1000004eecbccdrcrd.html
Christian D. and Paul, thanks for the links! I'll check them out as soon as I have some time :-)
Post a Comment