Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Monday, 25 August 2008

Getting Last.fm Tags for MP3s with Python

Paul (who is already distributing a large chunk of Last.fm tags) and I are planing to include a few slides in our ISMIR tutorial on how to obtain tag data.

Below is some Python code that basically takes an MP3 file as input and outputs a list of Last.fm tags (for both artist and track). The MP3s don't need correct ID3 tags, but they need to be full length (clips won't work).

The Python code uses Norman's command line finger printing client to find the correct artist and track name. The path to the executable needs to be set in the code. Norman supports Win32, OSX Intel, Linux - 32.

The output is written to a file. For each MP3 file passed as argument there are up to two rows in the output file: one for the artist tags, and one for the track tags. Each row has the format: "<mp3filename> <encoded artist or artist/track name> <tag> <score> [<tag> <score> ...]". Tabs are used as delimiters.

The data from the Last.fm API is available under the Creative Commons Attribution-Non-Commercial-Share Alike License.

Btw, special thanks to Eric Casteleijn for various Python recommendations (lxml etc). (Which reminds me that I still need to fix the other Python code I posted.) As usual any feedback is much appreciated.

import subprocess, sys, re, time, urllib
from lxml import etree

FP_CLIENT_PATH = '"C:\\fpclient\\lastfmfpclient.exe"'
MAX_RETRIES_URL_OPEN = 5

def getArtistTrack(mp3FileName): # ret: (artist, track)
command = FP_CLIENT_PATH + ' ' + mp3FileName
pipe = subprocess.Popen(command, \
stdout=subprocess.PIPE).stdout
for line in pipe:
mo = re.search('<url>.*/([^/]+)/_/(.+)<',line)
if mo:
return urllib.quote(mo.group(1)), \
urllib.quote(mo.group(2))
print "ERROR: failed to get artist/track for: " + \
mp3FileName

def crawlTags(url): # ret: [(tag, count), ...]
for i in xrange(MAX_RETRIES_URL_OPEN):
tagCounts = []
time.sleep(1) # be nice!
try:
root = etree.parse(
urllib.urlopen(url)).getroot()
except IOError:
print "(%d/%d) Failed trying to get: %s." % \
(i, MAX_RETRIES_URL_OPEN, url)
else:
for tag in root.iter('tag'):
tagCounts.append(
(tag.find('name').text, \
tag.find('count').text))
return tagCounts

def tags(prefix, items, outStream): # crawl and write
for mp3FileName, item in items:
url = prefix + item + '/toptags.xml'
print url
tagCounts = crawlTags(url)
outStream.write('%s\t%s\t%s\n' %
(mp3FileName, item, '\t'.join(
tag + '\t' + str(count)
for tag, count in tagCounts)))

def main():
if len(sys.argv)<3:
print 'USAGE: python getTags.py ' + \
'<outFile> <f1.mp3> [<f2.mp3> ...]'
sys.exit(2)
outFile = sys.argv[1]
mp3FileNames = sys.argv[2:]
artists = set()
artistTracks = set()
for mp3FileName in mp3FileNames:
print 'Fingerprinting: ' + mp3FileName
artist,track = getArtistTrack(mp3FileName)
artists.add((mp3FileName, artist))
artistTracks.add((mp3FileName,
artist + '/' + track))

print 'start crawling tags'
o = open(outFile,'w');
tags('http://ws.audioscrobbler.com/1.0/artist/', \
artists, o)
tags('http://ws.audioscrobbler.com/1.0/track/', \
artistTracks, o)
o.close()

if __name__ == "__main__":
main()

Sunday, 29 June 2008

Last.fm's API, Python, and tagging behaviour (Part 2)

Update: (2008/08/25) I fixed the same kind of bug I had in the previous post on this topic. While fixing it I decided to rerun it with a sample of 5000 instead of 2000 users. The code is fixed and the data is updated.

Last night I was a bit tired and quickly concluded "the numbers show...". But do they really?

To find out I use Python to do some simple statistical analysis to add some weight to the claim that older Last.fm users have a larger vocabulary and tag items more frequently.

First, I compute 95% confidence intervals for the percentage of non-taggers in each age group. Seeing the large margins (in the table below) helps explain why the age group 25-30 has a higher percentage than the age group 19-22. However, it doesn’t help explain why the age group 22-25 has a lower percentage. (I’d blame that on the relatively small and skewed sample, and I’d argue that they are still reasonably similar, both in average age and deviations of the percentages). (With the larger sample size this is not the case any longer.)

Computing the confidence interval is very easy. A user is either a tagger or not. The probability within an age group can thus be modeled with a Bernoulli distribution. The 95% confidence intervals for a Bernoulli distribution can be computed with:
z = 1.96 # norminv(0.975) for a 95% confidence interval
def binom_confidence(p,n):
if n*p*(1-p) >= 5:
return z*(p*(1-p)/n)**0.5

Btw, I couldn’t find the equivalent to the Matlab norminv function in Python. Any pointers would be appreciated!

To test the hypothesis that the vocabulary size of a tagger depends on her or his age I test the following: Given my observations, are all age groups likely to have the same vocabulary size, i.e, are the differences I observed just random noise? Since the distributions within each age group are far from Gaussian I can’t use a standard ANOVA. Instead I use the non-parametric version of a one-way ANOVA which is the Kruskal-Wallis test. In particular, I use the test to compute a p-value. The p-value is the probability that I would have made the same observation if the hypothesis that there is no difference between age groups would be true. (Thus smaller p-values are better. Usually one would expect at least a value below 0.05 before accepting an alternative hypothesis.) In this case the resulting p-value is nice and low indicating that it's extremely unlikely that older users don't have larger vocabularies.

Here are the results, and below is the Python code.

age || % non taggers || tagger's median
vocabulary size
14-19 || 41.3-48.1 || 6
19-22 || 37.5-43.6 || 7
22-25 || 40.6-47.3 || 9
25-30 || 34.8-41.4 || 8
30-60 || 28.4-36.0 || 13
Kruskal-Wallis p-value: 1.04e-008


from scipy.stats.stats import kruskal
from numpy import asarray

def print_stats(age_tags):
age_groups = (14,19,22,25,30,60)
ll = []
print "age || % non taggers || " + \
"tagger's median \n" + \
" vocabulary size"
for i in xrange(0,len(age_groups)-1):
nonzeros = [];
zero_count = 0
for j in xrange(age_groups[i],age_groups[i+1]):
for item in age_tags[j]:
if item!=0:
nonzeros.append(item)
else:
zero_count += 1
conf = binom_confidence(
zero_count/float(zero_count+len(nonzeros)),
zero_count+len(nonzeros))
ll.append(nonzeros);
print \
"%d-%d || %8.1f-%.1f || %2d" % \
(age_groups[i],age_groups[i+1],
(zero_count/float(max((1,len(nonzeros)+\
zero_count)))-conf)*100,
(zero_count/float(max((1,len(nonzeros)+\
zero_count)))+conf)*100,
median(nonzeros))
p = kruskal(*(asarray(ll[i]) for i in
xrange(len(age_groups)-1)))[1]
print "Kruskal-Wallis p-value: %.2e" % p

Btw, that part where I use eval to convert my lists into function arguments could hardly be any uglier. I’m sure there must be a better way of doing that? (Thanks Klaas!)

Last.fm's API, Python, and tagging behaviour

Update: (2008/08/25) I fixed the bug pointed out by thisfred. And I noticed that what I thought was the percentage of non-taggers was actually the ratio of non-taggers vs taggers... I changed that now.

My colleagues completely redesigned the Last.fm API. Inspired by their efforts and all the amazing things the Last.fm community has already built with the old API I decided that I wanted to try doing something with the API as well. The first thing that came to my mind was to use the public API to show that younger people have a smaller tagging vocabulary than older people. I couldn't figure out how to get a user's age from the new API so I used the old one. Anyway, here are the results and I also included the Python script I used. (Btw, any feedback on my Python coding is very welcome, I'm still very much a Python newbie.)

I crawled about 2000 users starting with RJ as seed. The first column is the age group, the second column is the ratio of users who haven't used any tags vs number of users who have used tags, the last number is the median number of unique tags which users who have applied tags have used.

14-19: zeros: 0.44 (120/155), median tags: 6
19-22: zeros: 0.42 (153/215), median tags: 6
22-25: zeros: 0.38 (114/184), median tags: 9
25-30: zeros: 0.43 (141/188), median tags: 10
30-60: zeros: 0.31 (79/179), median tags: 11

The numbers show that older users tag more and apply more unique tags.

from xml.dom import minidom
from urllib import quote, urlopen
from time import sleep
from numpy import median
from collections import defaultdict

seed = 'RJ' # start with Last.fm's CTO
MAX_RETRIES_URL_OPEN = 5

def get_xml(url):
for i in xrange(MAX_RETRIES_URL_OPEN):
try:
sleep(1) # be nice!
return minidom.parse(urlopen(url))
except IOError:
print "(%d/%d) Failed trying to get: %s." % \
(i, MAX_RETRIES_URL_OPEN, url)

def get_friends(user, friends, ignore_friends):
url = u'http://ws.audioscrobbler.com/1.0/user/' \
+ quote(user) + u'/friends.xml'
xmldoc = get_xml(url)
xmlusers = xmldoc.getElementsByTagName("user")
for user in xmlusers:
u = user.getAttribute("username")
if u not in ignore_friends:
friends.add(u)
print "%d/%d" % (len(friends), len(ignore_friends))
return friends

def get_age(user):
''' returns zero if user has not set his or her age '''
url = u'http://ws.audioscrobbler.com/1.0/user/' \
+ quote(user) + u'/profile.xml'
xmlage = get_xml(url).getElementsByTagName("age")
if len(xmlage)==0: return 0
return int(xmlage[0].firstChild.nodeValue)

def get_tags(user):
url = u'http://ws.audioscrobbler.com/1.0/user/' \
+ quote(user) + u'/tags.xml'
return len(get_xml(url).getElementsByTagName("tag"))

def print_stats(age_tags):
age_groups = (14,19,22,25,30,60)
for i in xrange(0,len(age_groups)-1):
nonzeros = [];
zero_count = 0
for j in xrange(age_groups[i],age_groups[i+1]):
for item in age_tags[j]:
if item!=0:
nonzeros.append(item)
else:
zero_count += 1
print \
"%d-%d: zeros: %.2f (%d/%d), median tags: %d" % \
(age_groups[i],age_groups[i+1],
zero_count/max((1,float(len(nonzeros)+ \
zero_count))),
zero_count, len(nonzeros), median(nonzeros))


users_notvisited = set([seed])
users_visited = set()

while len(users_notvisited)>0 and \
len(users_notvisited) + len(users_visited)<2000:
user = users_notvisited.pop()
if user not in users_visited:
users_notvisited = \
get_friends(user, users_notvisited, \
users_visited)
users_visited.add(user)

users = users_notvisited.union(users_visited)

age_tags = defaultdict(list)
i = 0
for user in users:
i += 1
print "%d/%d" % (i, len(users))
age_tags[get_age(user)].append(get_tags(user))
if i % 5 == 0:
print_stats(age_tags)

Thursday, 26 June 2008

Matlab, Python, and a Video

I've been using Matlab extensively for probably almost 10 years. I have written more lines of code in Matlab than in any other language. I always have at least one Matlab application window open. I've probably generated at least a few million Matlab figures (one of my most favorite Matlab functions is close all). I've written three small toolboxes in Matlab (and all of them have actually been used by people other than me). I've told anyone who was willing to listen that I couldn't have gotten even a fraction of my work done without Matlab. In fact, 3 times in a row I convinced the places I've been working at that I needed a (non-academic) license for Matlab and several of its toolboxes. I even had a Matlab sticker on my old laptop for a long time. I frequently visited the Matlab news group and I'm subscribed to several Matlab related blogs. If I would have needed to take a single tool with me on a remote island it would have been Matlab. I guess it's fair to say I was in love with Matlab.

However, I always felt that it wasn't a perfect relationship. Matlab is expensive. Matlab is not pre-installed on the Linux machines I remotely connect to. In fact, installing Matlab on Linux is a pain (compared to how easy it is to install it on Windows). Furthermore, not everyone has access to Matlab making it harder to share code. Finally, Matlab can be rather useless when it comes to things that are not best described in terms of matrices that fit into memory, and I can't easily run Matlab code on Hadoop.

I had been playing with Python out of curiosity (wondering why everyone was liking it so much) but I guess I was too happy with Matlab to seriously consider alternatives. But then Klaas showed me how to use Python with Hadoop. Within a very short time I've started to use Python more and more for things I usually would have done in Matlab. Now I write more Python code a day than Matlab code. I still use Matlab on a daily basis, but if I had to choose between Matlab and Python, it would be a very easy choice. SciPy and related modules are wonderful. If I'd redo my PhD thesis, it wouldn't include a single line of Matlab code and instead lot's of Python code :-)

Btw, James pointed me to the following visualization showing activities and shared code of Python developers over time. This is by far the best information visualization I have seen in a very long time. I really like the idea and implementation. I wonder if something similar could be done for a piece of music where the coders are replaced with instruments, and the files are replaced with sounds.


code_swarm - Python from Michael Ogawa on Vimeo.