ISMIR Proceedings 2008! Wow!

I'm extremely impressed. The ISMIR proceedings are online. Whoever wants a printed copy can organize it themselves (it couldn't be much easier). Some might also want to only print the papers they are interested in. And some might be happy to have only an electronic version.

It's always been a pain to drag the heavy ISMIR proceedings home. And it always felt like a huge waste of paper.

I heard Juan and Youngmoo talk about this idea a year ago in Vienna (at last year's ISMIR). I'm very happy to see that they found a solution that should make everyone happy.

Juan writes in his email:
We hope that you will like this new approach to printing the proceedings which we intend to be more cost effective, more convenient, and, with luck, more environmentally friendly than mass printing of proceedings for all attendees who may not wish to carry a printed copy around.

Wonderful! :-)

Librarians and Tags

Yves pointed me to this really nice presentation by a librarian who seems to have a really good understanding of tagging. The only part missing in that presentation is music.

Libraries and the Hive Mind: Folksonomies and Tagging

View SlideShare presentation or Upload your own. (tags: tagging tags)

Getting Last.fm Tags for MP3s with Python

Paul (who is already distributing a large chunk of Last.fm tags) and I are planing to include a few slides in our ISMIR tutorial on how to obtain tag data.

Below is some Python code that basically takes an MP3 file as input and outputs a list of Last.fm tags (for both artist and track). The MP3s don't need correct ID3 tags, but they need to be full length (clips won't work).

The Python code uses Norman's command line finger printing client to find the correct artist and track name. The path to the executable needs to be set in the code. Norman supports Win32, OSX Intel, Linux - 32.

The output is written to a file. For each MP3 file passed as argument there are up to two rows in the output file: one for the artist tags, and one for the track tags. Each row has the format: "<mp3filename> <encoded artist or artist/track name> <tag> <score> [<tag> <score> ...]". Tabs are used as delimiters.

The data from the Last.fm API is available under the Creative Commons Attribution-Non-Commercial-Share Alike License.

Btw, special thanks to Eric Casteleijn for various Python recommendations (lxml etc). (Which reminds me that I still need to fix the other Python code I posted.) As usual any feedback is much appreciated.

import subprocess, sys, re, time, urllib
from lxml import etree

FP_CLIENT_PATH = '"C:\\fpclient\\lastfmfpclient.exe"'
MAX_RETRIES_URL_OPEN = 5

def getArtistTrack(mp3FileName): # ret: (artist, track)
    command = FP_CLIENT_PATH + ' ' + mp3FileName
    pipe = subprocess.Popen(command, \
                            stdout=subprocess.PIPE).stdout
    for line in pipe:
        mo = re.search('<url>.*/([^/]+)/_/(.+)<',line)
        if mo:
            return urllib.quote(mo.group(1)), \
                   urllib.quote(mo.group(2))
    print "ERROR: failed to get artist/track for: " + \
          mp3FileName

def crawlTags(url): # ret: [(tag, count), ...]
    for i in xrange(MAX_RETRIES_URL_OPEN):
        tagCounts = []
        time.sleep(1) # be nice!
        try:
            root = etree.parse(
                urllib.urlopen(url)).getroot()
        except IOError:
            print "(%d/%d) Failed trying to get: %s." % \
                (i, MAX_RETRIES_URL_OPEN, url)
        else:
            for tag in root.iter('tag'):
                tagCounts.append(
                    (tag.find('name').text, \
                     tag.find('count').text))
            return tagCounts

def tags(prefix, items, outStream): # crawl and write
    for mp3FileName, item in items:
        url = prefix + item + '/toptags.xml'
        print url
        tagCounts = crawlTags(url)
        outStream.write('%s\t%s\t%s\n' %
                (mp3FileName, item, '\t'.join(
                    tag + '\t' + str(count)
                    for tag, count in tagCounts)))    

def main():
    if len(sys.argv)<3:
        print 'USAGE: python getTags.py ' + \
              '<outFile> <f1.mp3> [<f2.mp3> ...]'
        sys.exit(2)
    outFile = sys.argv[1]
    mp3FileNames = sys.argv[2:]
    artists = set()
    artistTracks = set()
    for mp3FileName in mp3FileNames:
        print 'Fingerprinting: ' + mp3FileName
        artist,track = getArtistTrack(mp3FileName)
        artists.add((mp3FileName, artist))
        artistTracks.add((mp3FileName,
                          artist + '/' + track))

    print 'start crawling tags'
    o = open(outFile,'w');
    tags('http://ws.audioscrobbler.com/1.0/artist/', \
         artists, o)
    tags('http://ws.audioscrobbler.com/1.0/track/', \
         artistTracks, o)    
    o.close()    

if __name__ == "__main__":
    main()

Tagging Critics

I was doing some research for the ISMIR tag tutorial when I stumbled upon (via this interesting paper Playing Tag: An Analysis of Vocabulary Patterns and Relationships Within a Popular Music Folksonomy by Abbey E. Thompson):

The following expert from this paper:

[...] "tags are often ambiguous, overly personalised and inexact" [...] "The result is an uncontrolled and chaotic set of tagging terms that do not support searching as effectively as more controlled vocabularies do." [...]

This was published in the D-Lib magazine in early 2006. I wouldn't be surprised if by now the authors realized they were wrong.

But why would anyone ever want to control the vocabulary people use when describing something so extremely multifaceted and something that evolves so fast like the content on the web (delicious), or snapshots of life (flickr), or music (Last.fm)? I guess I'd need to think more like an old-skool librarian to understand that.

Tagging Games (Callabio on Facebook)

Did Microsoft researchers clone the scoring system MajorMiner uses for a tagging game on Facebook? Well, maybe it's not identical, but it seems kind of similar. And I guess it could be lots of fun.

Btw, if scoring points is what drives people to play that game, what would stop them from entering a whole dictionary?

UK Hadoop User Group Meeting

Last Tuesday the first UK Hadoop user group meeting took place in London. Johan did a great job in organizing it and speakers included Doug Cutting from Yahoo! (who leads the Hadoop project).

Here's are some links:

• Yahoo! developer network blog mentioning the event. (Includes a video of interviews with some of the presenters.)
  
• Doug Cutting: Hadoop overview
  
• Tom White: Hadoop Web Services on Amazon S3/EC2
  
• Steve Loughran: Deploying Apache Hadoop with Smartfrog
  
• Mark Butler: Distributed Lucene for Hadoop
  
• Last.fm related talks:
  • Martin Dittus: Hadoop usage at Last.fm
    
  • Elias Pampalk: Radio Log Analysis for A/B Tests
    
  • Johan Oskarsson: Hadoop: Lessons learned at Last.fm
    
  • Klaas Bosteels: Dumbo: Hadoop streaming made elegant and easy
    
  • Tim Sell: PostgreSQL to HBase Replication
    
  
  

Seems like the interesting talk by Miles Osborne on "Using Nutch and Hadoop for Natural Language Processing" is still missing on skills matter's website. I'll update this blog post when the talk is added.

More MIR related PhDs

I've added 5 dissertations to the incomplete list of MIR related PhDs. I'm particularly happy that Markus, Tomoyasu, and Kazuyoshi (all of them are former colleagues of mine) finished their thesis so successfully.

Tomoyasu Nakano recently finished his thesis on "A Study on Developing Applications Systems Based on Singing Understanding and Singing Expression" (in Japanese). He has now joined Masataka Goto's research group as a postdoc and is working on the CrestMuse project. He recently received a lot of attention for his work on optimizing Vocaloid parameters.

Kazuyoshi Yoshii recently finished his thesis on "Studies on Hybrid Music Recommendation Using Timbral and Rhythmic Features". Kazuyoshi was awarded a tenure position at the AIST which is pretty impressive. He joined Masataka's group and is working on CrestMuse.

Markus Schedl recently finished his thesis on "Automatically Extracting, Analyzing, and Visualizing Information on Music Artists from the World Wide Web". As part of his thesis he crawled a very impressive number of web pages to build a retrieval system for 600,000 artists. In the next months he'll be finishing his business studies.

Steven Travis Pope's thesis is already a bit older (2005) and serves as perfect example of how incomplete the list of PhDs is which I maintain. His work was on "Software Models and Frameworks for Sound Composition, Synthesis, and Analysis: The Siren, CSL, and MAK Music Languages".

The 5th PhD thesis I added to the list is Matt Wright's work on "Computer-Based Music Theory and Acoustics" which he completed at CCRMA, Stanford University.