MIREX 2008

This year's MIREX evaluation task has been one of my personal ISMIR 2008 highlights. Stephen Downie and his team computed more numbers than I could possible keep track of for lots of different algorithms in 12 different MIR tasks. That's a lot more than in any of the previous years, and that's a lot of interesting data to dig into.

I've been particularly interested in the auto-tagging task. It's the first time MIREX ran this form of task, and there have been only few research papers in the MIR community on the subject. As far as I understood there is no agreement yet as to how to exactly evaluate the algorithms, which is also reflected on the result page. Kris West has added information on the statistical significant of the results which show that none of the submissions was consistently and significantly better than others. Nevertheless, there's a lot to learn from the evaluation and I hope we'll see many more participants next year.

Paul has a good summary of the discussion of this year's MIREX panel.

Web Services for Researchers

It just occurred to me how soon every research lab might be offering a long list of web services. Bandwidth is not a limiting factor. Building a web service is not that hard as it was 5 years ago. It's a great way to share without giving away code (and IP). It's also user friendlier as it doesn't require installing someone else's most likely buggy code on your own system. And it's potentially a great way to make money, too!

I wonder if I'm the last one to realize this? :-)

Anyway, what has helped me realize this was Thomas Lidy's announcement of his teams new web service, and The Echo Nest's web services that I recently found out about through Paul. Both allow you to upload music, extract features from the audio signal, and send them back to you.

I just gave both a try and they worked very smoothly. The two pictures below show results for the same track. The first one is created with the processing music visualization tool provided by The Echno Nest, the second one using Matlab to analyze the fluctuation pattern that Tom's tool extracts.





I wonder if the Echo Nest's service would crunch 100k tracks. (I believe there are at least a few research groups already dealing with collections beyond 100k tracks.) The service Tom announced is limited to 100 tracks/day and a maximum of 300 total per voucher (which requires you to sign up with your email address). Anyway it's a great start. And it seems that Tom will soon be making more announcements on further services that allow anyone to visually organize their music collections using a metaphor of geographic maps. Nice!

Btw, the Last.fm web services also seem to be very popular amongst researchers, at least some have been hitting them very hard ;-)

And one of the most eagerly anticipated web services is probably the MIREX DIY web service which was announced at ISMIR 2007 by Stephen Downie's team. The service will allow researchers to upload their implementations and receive evaluation results in return. Which will make it very easy for researchers to test if they are heading in the right direction.

One Llama, IMIRSEL, MIREX

One of the interesting things I learned in the recommendation tutorial today is that IMIRSEL launched a startup called One Llama. Seems like they have some ideas on how to make money with MIR technologies. I wonder how many of the MIREX participants were aware of this before submitting their latests implementations to IMIRSEL.

One evening and no testing

It’s been a long and busy day, and it’s taken me a while to go through the flood of emails that landed in my inbox today. Several of those were related to a singing microwave which seems to be at the height of its career.

Another interesting email I found in my inbox explains how the system that scored highest in several MIREX 2007 tasks was built:

“The system was not tuned - in fact it was not tested on any dataset (from the competition or otherwise) beyond making sure it was outputting feature values into its feature files and was in fact cobbled together in one evening.”

Something that has never been tested before, and sounds like a preliminary prototype outperformed them all. Since it hasn't been tweaked yet, the system probably has a very good potential to generalize, and probably can easily be tweaked to add at least another 1 or 2 percentage points accuracy to the genre classification results. That's pretty impressive.

Talking about MIREX I would like to add the following to clarify things I have written in a previous blog post:

I highly value MIREX, it's a driving force behind advances in MIR. I've personally learned a lot from it.

I understand that IMIRSEL has sacrificed a lot to make MIREX happen. It's been an amazing effort organized by Stephen Downie and his team.

I'm sorry my comments on the conflicts of interest issues have been perceived as personal attacks. That was not my intention.

I realize that my previous blog post on the topic should have clearly stated that: I'm fully (and always have been) convinced that no one at IMIRSEL had the intention to cheat. I have absolutely no doubts about that.

However, I'm still fully convinced that IMIRSEL submissions should not be listed in the same ranking as the submission of others.
 

Overfitting and MIREX

IMIRSEL (the organizer of MIREX) hasn't officially responded yet to the conflicts of interest of organizing a non transparent evaluation and at the same time participating in it. What I've heard from others is that they don't see any problems with it.

Btw, has anyone else noticed that they won in every classification category where overfitting is a big issue? However, in a very related category (mood classification) where overfitting isn't an issue (thanks the a human component in the evaluation) they were outperformed by several others.

Furthermore, IMIRSEL never had their name put down on the list of potential candidates. Given the lack of transparency of the respective MIREX tasks I think this is something every participant should have known before submitting their work. Btw, so far it isn't even known who the researchers are who actually did the work. AFAIK, no entry so far in the history of ISMIR evaluations has been submitted without mentioning who the authors are.

Btw, as to now, IMIRSEL are the only ones in the genre classification task who haven't published an abstract (describing what their algorithm does and how it was optimized) yet. (They also haven't submitted one yet for the other tasks they won in.)

UPDATE:
Regarding anonymous MIREX submissions I just remembered that at the ISMIR 2004 evaluation hosted by MTG allowed anonymous submissions... and some authors did choose to do so. (However, as I already mentioned in the comments of this post: MTG clearly stated that they did not participate in the tasks they organized to avoid any conflict of interest.)
 

MIREX Results Online!

The MIREX results just got posted by Stephen Downie. Interestingly the organizers scored highest in a number of categories. To be honest, if I were a participant in a task like genre classification I’d be a bit suspicious. (Knowing the distribution of the genres before hand can be a huge advantage when designing an algorithm.)

Congratulations to Tim Pohle and Dominik Schnitzer (two very clever PhD students I once worked together with in Vienna) who scored highest in the audio similarity task. I wouldn’t be surprised if they also had one of the fastest implementations. Tim also scored second highest last year in the same task. And Dominik recently made the results of his Master’s thesis available (open source playlist generation).

Congratulations also to Joan Serrà and Emilia Gomez (a former SIMAC colleague) who scored highest in the cover song identification task.

And congratulations to everyone who participated and the organizers for managing to complete all the tasks before ISMIR!

Music Similarity: G1C Implementation

I’ve been planning to do this for over a year: The MA (Music Analysis) Toolbox for Matlab now finally includes the G1C implementation which I described in my thesis (btw, the code probably also runs in the freely available Scilab in case you don’t have Matlab). The code is packaged as required for the MIREX’06 evaluation, where the implementation was overall fastest and scored highest (but not significantly better than other submissions).

The code might be useful for those who are new to the field and just want a quick start. Btw, last October I held a presentation on music similarity which might also be helpful for starters and the best documentation and explanation of the code I can offer is my thesis.

I also hope the implementation is somehow useful for those interested in comparing their work on computational models of music similarity to work by others. I believe the best option to do so is to conduct perceptual tests similar to those I conducted for my thesis and those done for MIREX’06 (btw, I wrote some comments about the MIREX’06 evaluation here).

A much easier approach to evaluate many different algorithms is to use a genre classification scenario (assuming that pieces from the same genre are generally more similar to each other than pieces from different genres). However, this doesn’t replace perceptual tests it just helps pre-select the algorithms (and their parameters). Btw, I think it would even be interesting for those working directly on genre classification to compare G1C (combined with a NN classifier) against their genre classification algorithms.

There are lots of things to be careful about when running evaluations based on genre classes (or other tags associated with music). Most of all I think everyone should be using an artist filter: The test set and the training set shouldn’t contain music from the same artists. Some previous work reported accuracies of up to 80% for genre classification. I wouldn’t be surprised to see some of those numbers drop to 30% if an artist filter had been applied.

I first noticed the impact of an artist filter when I was doing some work on playlist generation. In particular, I noticed that songs from the same artist appeared very frequently in the top 20 most similar lists for each song, which makes sense (because usually pieces by the same artists are somehow similar). However, some algorithms which were better than others in identifying songs from the same artists did not necessarily perform better in finding similar songs from other artists. I reported the differences in the evaluation at ISMIR’05, discussed them again in my MIREX'05 submission, and later in my thesis. An artist filter was also used for the MIREX’06 evaluation. Btw, I’m thankful to Jean-Julien Aucouturier (who was one of the reviewers of that ISMIR’05 paper) for some very useful comments on that. His thesis is highly relevant for anyone working on computation models of music similarity.

Another thing to consider when running evaluations based on genre classes is to use different music collections with different taxonomies to measure overfitting. For example, one collection could be the Magnatune ISMIR 2004 training set and one could be the researcher’s private collection. It can easily happen that a similarity algorithm is overfitted to a specific music collection (I demonstrated this in my thesis using a very small collection). Although I was careful to avoid overfitting, G1C is slightly overfitted to the Magnatune collection. Thus, even if G1C outperforms an algorithm on Magnatune, the other algorithm might still be much better in general.

There’s some room for improvements of this G1C implementation in terms of numerical issues, and some parts can be coded a lot more efficiently. However, I’d recommend trying something very different. Btw, I recently noticed how much easier it is to find something that works much better when having lots of great data. I highly recommend using Last.fm’s tag data for evaluations, there’s even an API.

Recent Events

The MIREX preparation meeting in Vienna organized by Stephen Downie was a big success. Having the chance to discus the various issues in person is definitely a much more efficient approach than the endless flood of emails we had last year. One of the most discussed topics was audio-based music similarity. Respective discussions ranged from the need for a more specific definition of the task to technical details of large similarity matrices. Andreas Rauber’s team and in particular Thomas Lidy did a great job hosting the meeting. MIREX was also mentioned on the Boing Boing blog. Below is a picture of the participants taken by Rainer Typke. More of his pictures can be found here.



Almost at the same time the ICASSP conference took place in Hawaii which was also well attended by MIR researchers. The next get together seems to be the AES122 workshop on Music 2.0 in Vienna which is organized by Oscar Celma and Mark Sandler. (Update: see also this post by Paul Lamere.)
Btw, 214 papers were submitted to ISMIR this year, which is a 17% increase of the previous record which was set last year (183 submissions).

One a side note, Paul Lamere somehow already guessed it: I’m joining last.fm. I’m starting on Monday will try my best to support the social music revolution :-)

MIREX 2007 Preparation Meeting

MIREX is all about evaluating MIR research. Evaluations are needed to measure improvements, and thus enable improvements. Even only talking about how to conduct proper evaluations (which is what happens all the time at MIREX) is already beneficial for MIR.

Next week the MIREX 2007 preparation meeting will take place in beautiful Vienna. 25 people from all over the world are on the list of people who will attend the MIREX planing meeting. Every major European MIR lab seems to be sending someone, and there's lots of people from North America, and even from Japan and New Zealand! And I heard that a company working on MIR topics might be sending someone. And of course I'll be there, too :-)

When the evaluations started in 2004 by some fine researchers at MTG no one could have ever foreseen this success. And without Stephen and his IMIRSEL team at UIUC it would never have happened. Even a week ago I never would have thought that Stephen would manage to bring together so many researchers from all over the world for this preparation meeting.

Of course, everyone will now argue that the large number of participants is just a typical Vienna effect (everyone wants the chance to get to see the city, which I fully understand). Btw, as most of us know, this is also the main reason why the number of submitted papers to this year's ISMIR will be much higher than ever before. I hope my papers don't get rejected :-/

Anyway, this year's MIREX success is also based on the success of the previous MIREX evaluations. I know many have been very skeptical about MIREX. But many of the issues we've seen have been fixed, and with some luck this year everything will be really smooth. There's still time to jump on the train! (If you haven't done so already) :-)