Late Night Thoughts

Last night I went to bed and fell asleep listening to a podcast that Paul recommended. I didn't stay awake for long but I remember that the interviewer seemed skeptical about some of the ideas of open research.

I guess it's natural to be skeptical when something is radically different to what we are used to. I wonder what Newton would have said if someone would have told him to publish or perish and that there are plenty of good journals that publish articles within a year of submission (including the review process). It took Newton almost 22 years to publish his findings, and that was not so unusual back in 1687.

Using those two data points (22 years in 1687 and 1 year in 2008) a simple linear model would suggest that a submit-publish cycle might take less than 3 months in 2020. While I can't see how that linear model is a good fit I could easily see how pushing a publication out within 3 months could be achieved with tools similar to what we know as blogs today.

Furthermore, looking at historic data it's also not too hard to see that unlike some might expect we will see fewer disputes with respect to who discovered what first.

Btw, tonight I'll try to stay awake as long as possible with this podcast about earworms.

Open Research

In response to what Paul just blogged about...

I'm curious how in the long run blogging about ongoing research will transform communication within the research community. I've already been curiously following blogs by some PhD students such as Mark or Yves who are very open about their ongoing work and ideas.

I don't think blogs can replace publishing papers at research conferences or journals. However, I wouldn't be surprised to see more and more references to blog posts in conference papers in the future.

Blogs would be the perfect communication platform for researchers if:

1. there would be a guarantee that a blog post will be around for ever (i.e., that researchers in 20 years from now can go back and look at it)
   
2. if it would not be possible to alter any information published on a blog (or at least to detect if something has been altered), this includes not being able to change the date when an idea or result was first published
   

It seems that one way to overcome both limitations would be to have authorities frequently crawl, store, and index blog posts related to research. Another option might be to have something like a "research mode" on popular blogging service providers such as blogger.com: if the blogger opts into this mode, then the researcher wont be able to ever change his blog posts (including deletion) and the blog posts will be indexed by search engines.

However, even as publishing preliminary research results on blogs becomes an accepted standard I wouldn't be surprised if some unfortunate researchers without ideas of their own consider an idea published on a blog to be not published at all and try to publish it at a conference with their own names on it without referencing the source. I'm sure though, that such attempts would ultimately fail as the blogging research community would point them out quickly.

I wouldn't be surprised if a few researchers starting to use blogs to communicate ongoing results will trigger a snowball effect. For example, if Paul starts blogging about ongoing research that someone else is currently working on, wouldn't that other researcher feel urged to publicly state that he or she is also working on the same topic? Otherwise, by the time this other person publishes results, everyone might think that those ideas were just copied from Paul's blog?

Looking at how research has developed over the past centuries, the direction we have consistently been heading in seems very obvious: more openness and getting results out faster. Research blogs seem like a very natural next step in the evolution of communication in the science community.

The Future of the Paper Industry

Google scholar is a wonderful way to measure the quality of scientific papers based on the number of citations. I actually find the number of citations so interesting that I display it in my list of publications (the yellow bars on the left correspond to citations).

I learned, for example, that one of my papers (which I thought was one of my better ones) never got cited (I'm guessing that's because there’s not too many people working on organizing drum sample libraries). In addition, I believe it might be useful for someone browsing my list of publications to identify which papers are probably more readable. But I'm not sure about the usability in its current form.

I also just learned that somehow my PhD thesis has made it into the top 30 results for the search term music information retrieval! :-)

Anyway, the point is that Google scholar as a form of evaluating the quality of papers is highly insufficient: the delay between having a final version of the paper and the point where reliable quality estimates can be made is way too large (often taking several years).

Given the limitations of Google scholar I’ve been thinking about what the future of the paper industry should be like (and in particular how the quality of a paper could be measured more quickly), and here are some things I’d love to see:

1. Papers should be publicly reviewed similar to the book reviews by Amazon’s customers. (Including the option to rate the usefulness of reviews.)

2. Researchers, research projects, and research teams should have blogs in which they present their findings and encourage open discussions and criticism of their work.

3. Researchers (and in particular students) should post their ideas on public sites to get instant feedback from peers (and document who came up with the idea first).

And here’s how I plan to contribute to the future:

I plan to publish some comments from reviewers, and my response for the ISMIR'07 paper for which I’m first author of (including a link to the paper and the demonstration video).

I plan to write reviews on papers that I read in the future. Maybe I’ll do so for some ISMIR'07 papers that are already online.

However, I’m also planning to spend most of my time in the next weeks helping improve how Last.fm's radio stations listen to their listeners. So it might take me a while to get my "Paper Industry 2.0" contributions started :-)

Btw, using A/B tests to evaluate algorithms on radio stations beats number of citations any time :-)
(And if you like A/B tests, you’re probably also a fan of Greg Linden’s blog.)

UPDATE:
I just read the two links posted in the comments and they are great!
The first link posted by Paul basically talks about how the Nature community is still very old school. The second link posted by Chris talks about how things will need to change in the future and why so many researchers are still very old school.