Fading Out

I first started learning about MIR around spring 2001. Back then I never would have guessed what a wonderful journey I was about to embark on. The journey had way more highlights than I ever deserved and I'm very grateful to so many people I've met on the journey.

I recently returned to Vienna to be closer to my family and friends again, and to settle down and start a new chapter in Gwen's and my life. Part of this transition has also been switching jobs. I'm in telecommunications now and uncertain if I'll ever return to MIR research.

I'll always continue following all the great stuff that's going on in the community from the side lines - but my already very infrequent blogging frequency is very likely to continue to decrease. When I started writing there was basically just Paul's blog. Now there's still Paul's blog, but also lots of other great MIR related blogs. Some of which are listed in the side bar.

Updated List of MIR PhDs

There's a long-overdue updated version of the list of MIR PhDs. The new entries I've added are Stéphane Rossignol (2000), Juan José Burred (2008), Iman S. H. Suyoto (2008), Matti Ryynänen (2008), Enric Guaus (2009), Claudio Baccigalupo (2009), Klaas Bosteels (2009), Katy Noland (2009), Tim Pohle (2010), and Jouni Paulus (2010). I've also updated some other links. Thanks to everyone who sent me information!!

I've had the pleasure of reading the thesis of Claudio and I can highly recommend it. His enthusiasm is inspiring. It's a great point to start for anyone researching playlist generation.

Another pleasure I've had was to attend the defense of Klaas' thesis. I've been closely following his work for a very long time. I've had the honor to collaborate with him on several occasions, and I'm very pleased that it's him who has taken over some of my previous responsibilities at Last.fm.

I've also been following Katy's and Tim's work for many years. Which makes me wonder how soon until all MIR PhD students I know are done. Until that happens I'll infrequently continue updating this blog and the list.

10 Reasons to Work for Last.fm

I've moved to Vienna, and I'm about to leave Last.fm. Looking back, here are 10 reasons why I love to work for Last.fm:

1. Staff
My colleagues turn work into play and have created an environment that feels like a family. I've learned a lot from many of them. Some have become very close friends. All of them have inspired me with their passion, enthusiasm, motivation and curiosity for everything related to technology and music. Also they've always been very patient with me and always very helpful. I admire their intelligence, creativity, and how determined they are to make music more enjoyable.

2. Users
There is a constant flow of very valuable feedback from our users. Many are as passionate about Last.fm as we are. They have enabled us to run countless experiments (A/B tests and similar) - from which we learned more than we could have ever learned in a standard academic environment.

3. Data
If you love data mining and working with large data sets, you'll love working at Last.fm. There are a lot of fascinating things that can be learned about music from Last.fm's data. And there is a lot of fascinating things to learn about dealing with data at such a scale.

4. Technology
Last.fm has been on the front-line of some very interesting developments. Always using the tools that work best and developing new ones when needed. Always trying to push scalability limits and trying to make things work faster and better. Before joining Last.fm I didn't realize how fascinating technology can be that solves large scale problems. I've learned new programming languages, things like Hadoop, and many algorithms I had never heard of before. I caught a glimpse of the amazing technology (both software and hardware) behind serving tens of millions every month. If you love technology, you'll love working at Last.fm.

5. Openness
The data is easily available through APIs. Last.fm staffers constantly contribute to open source projects and parts of Last.fm are open source. Every staff member (even I) can post anything they want on the company blog. We are constantly attending events where we talk about the things we do and share our experiences. Also the communication within Last.fm is very open.

5. Fun
We have a table tennis table standing around in the office, a foosball table, there was always a skateboard around, and for a while we even turned one of the meeting rooms into a ball pit (and had meetings in there). There were remote controlled helicopters flying around in the office, XBoxes, we have colored bears telling us about the state of our code, highlight words on IRC include "pub", on IRC we also constantly share the newest coolest stuff we find on the Internet, servers have names like "badger", ...

6. Lunch Discussions & Techmosis
One of the things I've already started missing since I moved to Vienna are the daily (informal) lunch discussions. In particular the discussions in which Norman Casagrande was involved - he's like a living encyclopedia. We talked about why C++ templates are so useful, how to solve the problems in the middle east, inconsistencies in religious beliefs, objectivity as a concept, world history, politics, the daily show, ...

We also have something we call Techmosis, where we teach each other things. Usually Friday evenings we reserve a one hour slot to learn about something cool. Sometimes we'd also have non-staffers teach us. Today it will be a presentation on manga by Japanese manga artists. Generally I found the atmosphere of teaching each other extremely inspiring.

7. London
London is an amazing city. There is so much happening, there is more stuff to check out every day than I could absorb in a year. I'd love to move Last.fm to Vienna, but I can see why East London is the perfect place for Last.fm. As one of my colleagues says: it rubs off on you. If you haven't lived in London yet, you should definitely give it a try.

8. Free Stuff
My favorites are my Last.fm t-shirts (one of which I'm wearing right now) and the fruit. (I herewith admit that every week I probably eat about half of our weekly supplies all on my own.) Pizza lunches and the occasional pub visits with a company credit card were nice to have, too.

9. Rooftop Barbecues
We made sure our neighbors knew they lived (or worked) next to us. But I think the best part always was how the parties just seemed to spontaneously happen.

10. Music
Last on this list, but not least: If you love music, you'll have a hard time finding a better place to work than Last.fm where you are surrounded by music, and people who love music. You might also like the band practice room.

Wittgenstein Award: Gerhard Widmer (!!)

The Wittgenstein award is kind of like the Austrian version of the Nobel prize. It's worth €1.5M and the most prestigious research award I'm aware of. Yesterday Gerhard Widmer received the prize for his outstanding research work.

Gerhard is one of the hardest working I know, he has an amazing talent of communicating research to non-researchers, creating very productive work environments, and bringing the right people together. Also he has conducted, supervised, and inspired lots of great research. He also gave me my first (paid) job in MIR and supervised my PhD. I've learned a lot from him and I'm very grateful. So I'm particularly happy to see Gerhard received this award he well deserves.

My favorite daily newspaper wrote about it. And here's a quick, shortened, and far from accurate translation of what they wrote:

Frustrations and Fortunes with Ludwig van Beethoven

Among Gerhard Widmer's favorite music you find Beethoven's piano sonatas. He doesn't have a preferred interpret. For some phrases he prefers Friedrich Gulda, for others Alfred Brendel, or other pianists.

It is also Beethoven who is to blame that Gerhard did not pursue a career as musician and instead became a internationally renowned researcher on algorithms to study music. Work for which he was now awarded with the Wittgenstein prize.

Gerhard was an early talent but gave up his career as pianist after frustrations with Beethoven's sonatas as a teenager. Instead he took a quick dip in Jazz and more or less randomly ended up studying computer science.

He received an MSc in Vienna and at the University of Wisconsin. It was also in Wisconsin that Gerhard briefly returned to the Jazz piano. Back in Vienna he completed his PhD in computer science which finalized his career as researcher.

Gerhard's research started with analyzing the performances of a single artist and as a side effect he tried to teach computers to interpret music. Later his group started focusing on developing algorithms that enable organizing and retrieving content from very large music collections. The work of his group can also be found in the newest devices by Bang & Olufsen.

Music has always been more than just a research subject to Gerhard. And he says that the scientific analysis of music does not take away any of its magic - instead it makes the music even more beautiful when you start to understand its structure.

Updated MIR PhD Theses List

I've updated the list of MIR PhD theses to include the recent work by Antti Eronen on Signal Processing Methods for Audio Classification and Music Content Analysis. Antti covers a broad range of fascinating audio content processing topics such as: instrument classification, classification of ambient environments/backgrounds (libraries, cars, ...), chorus detection, and meter analysis.

Seems like I'm the list is still far from complete. As the following two theses indicate. If you know of any more that I'm missing please let me know. (Thank you Arijit Biswas for your help!)

One that I missed is the work of Michael J. Bruderer on Perception and Modeling of Segment Boundaries in Popular Music. Which describes an interesting series of experiments in which Michael explores which cues listeners use when segmenting music and how they can be modeled.

Another older one I've added is the work of Olivier Gillet Transcription des signaux percussifs. Application à l'analyse de scènes musicales audiovisuelles. Olivier joined Google after his PhD where he worked on the optimization of Google's ad product and on technologies now being used by Google China Music. Olivier is now joining us at Last.fm's to work on fun next generation MIR technologies.

I would also like to mention the Master thesis Implementing a scalable music recommender by Erik Bernhardsson. I really like how Spotify has given him access to their data and let him publish the results. Seems like Spotify was also very happy with his work as they have hired him.

Bang & Olufsen and OFAI

This is probably very old news to some, but I'm a bit behind in blogging.

Yet another really cool product my former colleagues at OFAI have been working on over the last years: integration of content-based playlist generation technologies into the most awesome music hardware out there.

Read more about it here (English) and here (German).

If you can't read German, the perhaps most interest bits: retail price per unit €4.765, 500 pre-orders before the device was available, OFAI and B&O will continue the collaboration, size of the features representing each song: 3.2KB.

FM4 Soundpark and OFAI

My former colleagues at OFAI have been experimenting with audio content based music recommendation for FM4 Soundpark for a while now. (I blogged about it here.) Now they (and in particular Martin Gasser) have taken it a few steps further and implemented a 3D exploration interface using the islands of music metaphor.

It's all in Java and requires special privileges because it needs access to low level graphics card functions. But if you get passed those hurdles here's what you'd see:



(Navigation: space bar, A,W,S,D and mouse)

There is more information about this and other recommendation tools they have developed for Soundpark here (in German).

Last.fm Artist Connections

Playing with Last.fm similar artist data, trying to connect one artist to another through similar artists:



The Beatles -> John Lennon -> Sean Lennon -> Joseph Arthur -> Howie Day -> Teddy Geiger -> Jonas Brothers



ABBA -> Cher -> Madonna -> Róisín Murphy -> Björk -> Radiohead



Metallica -> Iron Maiden -> Paul Di'Anno -> Numbers From The Beast -> Scott Lavender -> Ark Sano -> Frédéric Chopin -> Wolfgang Amadeus Mozart



Brian Eno -> Roxy Music -> Bryan Ferry -> Eurythmics -> Annie Lennox -> Madonna -> Britney Spears

MIR PhD Thesis: Luís Gustavo Martins

Luís Gustavo Martins recently completed his PhD thesis titled A computational Framework for Sound Segregation in Music Signals.

From the abstract: "[...] This dissertation proposes a flexible and extensible Computational Auditory Scene Analysis framework for modeling perceptual grouping in music listening. The goal of the proposed framework is to partition a monaural acoustical mixture into a perceptually motivated topological description of the sound scene (similar to the way a naive listener would perceive it) instead of attempting to accurately separate the mixture into its original and physical sources. [...]"

Luis is probably best known in the MIR community for his contributions to Marsyas (the awesome open source software framework for audio processing with specific emphasis on MIR applications).

If you know of any other dissertations missing in the list of MIR PhDs please let me know.

Visual Listening Charts (Part 2)

Here's a follow-up to my previous attempt to visualize my listening history differently.

Basically, I've been looking for a fun project that would give me plenty of reasons to play with matplotlib.

My preliminary conclusion is that matplotlib is awesome. They couldn't have made it much easier to use for someone already familiar with Matlab.

Recent MIR PhDs

I'm slowly catching up. The following dissertations were added to the list of MIR PhDs:

"Real Time Automatic Harmonisation" by Giordano Cabra. I couldn't find a link to the thesis, but I found a video of the defense. I'd love to see more defense videos (preferably in English).

"Modeling musical anticipation: From the time of music to the music of time" by Arshia Cont.

"Music Recommendation and Discovery in the Long Tail" by Oscar Celma. I highly recommend it!

"From Sparse Models to Timbre Learning: New Methods for Musical Source Separation" by Juan Jose Burred.

UPDATE: I totally missed Yves' announcement on the Music-IR list on Friday. I've added him now too:

"A Distributed Music Information System" by Yves Raimond.

UPDATE 2: Almost forgot that Kris finished recently too:

"Novel Techniques for Audio Music Classification and Search" by Kris West.

Please send me any I might have missed - thanks!

Since ISMIR I've been listening to...

I'm currently a bit fascinated with different ways of representing my listening history.

I really like LastGraph which was inspired by Lee Byron. The graphs show how my listening preferences (and in particular how often I listen to my favorite artists) change over time.

I also like these visualizations by Martin. They show how artists move up or down over time in my chart.

Recently I tried to visualize my top artist chart using artist images, and readjusting their sizes so they correspond to how often I've listened to each artist respectively. Here's what I've been listening to since my last blogpost at ISMIR 2008:



I think this montage says it all: I like black and white music from solo artists playing guitars.

C++ Software Engineer, Data & Recommendations

If you are interested in data structures, algorithms, and scalability, you might also be interested in joining the data and recommendations team at Last.fm.

You'd be working with Norman Casagrande, Mark Levy, me and other highly motivated colleagues trying to solve lots of fun challenges in music information retrieval.

ISMIR 2008 Demos

One of the best parts about ISMIR was the demo session.

Paul Lamere and Francois Maillet demonstrated Explaura which enables users to directly interact with tag clouds by resizing individual tags for tag-based recommendations. Increasing the size of a tag puts more emphasis on it, shrinking it reduces the impact to the point where it's ignored. Increasing the "negative size" of a tag filters results by the respective tag. They also allow combining artists with tags in the search query. Paul blogged about it. There's also a short ISMIR abstract. I like the idea of interacting with the individual tags in a tag cloud and found it very intuitive to use.

Another demo which was presented (and I already previously blogged about) was the work of Martin Gasser and his colleagues on how they integrated audio-similarity into FM4 Soundpark (a platform for independent Austrian artists). The numbers they presented show that using audio-similarity helped Soundpark users find older and more obscure items in the catalogue. It would be nice to see more real world applications using audio-similarity.

Òscar Celma and Marcelo Nunes presented GeoMuzik which allows drawing a route on a world map. Their system then generates a playlist according to this route. They implemented genre/tag filters. They can also visualize the artists in my Last.fm profile on top of a map. There's an ISMIR abstract. I liked the demo very much and would like to play a bit more with it. The picture below shows a screenshot of a playlist generated from the interface.



Luke Barrington and his colleagues demonstrated their new tagging game Herd it which they implemented in the Facebook application framework. If I understood them correctly they still want to do some testing before releasing it publicly, so I'll write more when they are ready. Anyway, I've heard from others how difficult it can be to get something to work seamlessly on Facebook. I'm impressed that Luke and his colleagues are doing this. It would be nice to see researchers use Facebook and similar platforms more frequently for their work.

One demo I unfortunately didn't have enough time to see (but at least I got the handout) was the work Anita Lillie presented. There's also a video. In the video Anita shows different ways to visualize the same music collection from different perspectives using a PCA (principal component analysis, a linear projection of a high-dimensional space onto 2-dimensions). Seems like Anita just finished her MSc thesis on MusicBox: Navigating the space of your music. I've been told the demonstration was implemented in processing. Very nice! One thing I didn't see were playcounts. It would be great if playcounts (or ratings per track) could also be visualized (size of the circles?). The picture below shows a screenshot of her MusicBox.



Another demo I really liked was Claudio Baccigalupo's work on Poolcasting. The idea is to have several people tune into the same radio station at the same time, and to have them rate the songs, and use that rating to optimize the overall listening experience. I could easily see Gwen and myself sharing the same radio stream frequently. What I found particularly interesting is how Claudio tries to maximize happiness for everyone in a situation where compromises are unavoidable and where he does not want the majority to completely ruin the experience for an individual.

Btw, if you like demos, you might also be interested in Last.fm's playground.

MIREX 2008

This year's MIREX evaluation task has been one of my personal ISMIR 2008 highlights. Stephen Downie and his team computed more numbers than I could possible keep track of for lots of different algorithms in 12 different MIR tasks. That's a lot more than in any of the previous years, and that's a lot of interesting data to dig into.

I've been particularly interested in the auto-tagging task. It's the first time MIREX ran this form of task, and there have been only few research papers in the MIR community on the subject. As far as I understood there is no agreement yet as to how to exactly evaluate the algorithms, which is also reflected on the result page. Kris West has added information on the statistical significant of the results which show that none of the submissions was consistently and significantly better than others. Nevertheless, there's a lot to learn from the evaluation and I hope we'll see many more participants next year.

Paul has a good summary of the discussion of this year's MIREX panel.

1000 years of music to listen to

Youngmoo Kim had asked everyone on the ISMIR recommendation panel to briefly summarize what they think will happen in the next 5 years of music recommendation. However, it's really hard to do so in less than 4 minutes.

From my perspective the most interesting development in the next 5 years will be the increase in the amount of data we will be working with. We will have a lot more of the same and we will have additional sources. Combining different sources is an interesting challenge, but the main challenge will be to scale things up.

All of this additional information will lead to much better recommendations overall, and in particular in the long tail. We'll be able to detect new trends such as an up-and-coming artists or the emergence of a new subgenre much sooner. We'll be able to localize recommendations a lot more.

At the same time there'll obviously be a lot more music to choose from. I'd roughly estimate about 200 million tracks in Last.fm's recommendation engine in the next 5 years. That's more than 1000 years of continuous listening. Subcultures and genres will emerge faster.

In 5 years recommendation engines will have a much better understanding of listeners. While Last.fm, Pandora, and others already do a lot to understand what listeners are interested in, I'm sure there is room for a lot more improvements.

Another interesting development I'm looking forward to is data portability and openness. In particular, I'm looking forward to users being able to move freely with their personal data from one site to another. Similar to how Last.fm users can already today allow other sites to access their data.

I'm also expecting to see a lot more artists and labels embrace recommendation engines. Similar to SEO (search engine optimization) more artist and labels will try to do a lot more REO (recommendation engine optimization).

Obviously mobile applications will be very important, and so will mobile music recommendations. And I have no doubts that human-to-human recommendations (which are strongly supported by Last.fm) will continue to be very important, maybe even more than they are today.

Anthony Volodkin made a great point that we'll see a lot happen in terms of user interfaces, how recommendations are represented, how recommendations are explained. I believe Paul Lamere would call that steerable and transparent recommendations. I like how Last.fm explains recommendations by explaining a recommendation in terms of a bunch of similar artists I'm familiar with. However, there's obviously room for a lot more. On the other side, I wouldn't mind no explanation at all, as long as every recommendation is spot on. Anthony also made a great point by pointing to playful discovery systems.

I believe it was Brian Whitman who said that recommendations will be a commodity. Every music site will have recommendations. Just like almost every web 2.0 site out there supports tagging. I believe Etienne Handman made a similar point when he previously explained to me why he expects the word "personalization" to fade away. Everything will be personalized, it will be the default option.

ISMIR Thoughts

Inspired by Paul's constant flow of ISMIR blog posts I thought I should give it a try and post some thoughts as well.

First of all, the organizers did a wonderful job organizing ISMIR. I already wrote about what I think about the electronic proceedings. I was also very happy to see that they did not waste unnecessary resources and skipped the silly conference bag thing.

I very much enjoyed giving the social tags tutorial with Paul. It worked out really well, and although I knew Paul's slides since weeks, I found it fascinating to listen to Paul talk about them.

So far ISMIR has by far exceeded my expectations. I've had the pleasure to meet many in person that I previously hadn't had the opportunity to meet. I've also had the pleasure to see many very interesting posters. Unfortunately I've also managed to miss many that I wanted to see. I guess there's never enough time to see everything.

Last night's banquet was great too. In particular I enjoyed the conversations with Etienne from Pandora.

The recommendation panel today was fun, too.

ISMIR 2008 Tutorial: Social Tags and Music Information Retrieval

When Paul originally asked me if I'd be interested in helping him put together a tutorial proposal for ISMIR I was a bit reluctant. I did an ISMIR tutorial a long time ago, and I didn't forget how much work it was (although it was only a one hour mini tutorial). In fact, only last year Paul and Oscar told me how much work it was to put together their recommendation tutorial (which I liked a lot).

Anyway, somehow I couldn't say no and looking back I don't regret it at all. I'm totally fascinated by social tags. There's no tutorial topic I'd rather talk about. Working together with Paul has been a great pleasure, and I've learned a lot. However, I'm very much looking forward to have a free weekend or even just a free evening again. (Free as in not-MIR-related.)

Paul and I have put together about 200 slides for our tutorial. I think we still need to figure out our time budget. Hopefully we'll have plenty of time for interesting discussions.

Btw, as part of the tutorial we started compiling a list of relevant papers. That list started to grow. Then we thought it would be a good idea to group papers into topics (e.g. autotagging). Then we realized that several papers were in several categories... which is when we moved everything to delicious. In particular, we started using the tag "SocialMusicResearch" to mark interesting things we found in the Internet. Here's a list of some of the items we have tagged. We hope that others will start using that tag as well.

Btw, if you are going to ISMIR, please say hi! If you don't know what I look like: try to spot the guy wearing a Last.fm t-shirt.

ISMIR Proceedings 2008! Wow!

I'm extremely impressed. The ISMIR proceedings are online. Whoever wants a printed copy can organize it themselves (it couldn't be much easier). Some might also want to only print the papers they are interested in. And some might be happy to have only an electronic version.

It's always been a pain to drag the heavy ISMIR proceedings home. And it always felt like a huge waste of paper.

I heard Juan and Youngmoo talk about this idea a year ago in Vienna (at last year's ISMIR). I'm very happy to see that they found a solution that should make everyone happy.

Juan writes in his email:
We hope that you will like this new approach to printing the proceedings which we intend to be more cost effective, more convenient, and, with luck, more environmentally friendly than mass printing of proceedings for all attendees who may not wish to carry a printed copy around.

Wonderful! :-)

Librarians and Tags

Yves pointed me to this really nice presentation by a librarian who seems to have a really good understanding of tagging. The only part missing in that presentation is music.

Libraries and the Hive Mind: Folksonomies and Tagging

View SlideShare presentation or Upload your own. (tags: tagging tags)