Recommended Book: Probability and Statistics

The last two weeks I was camping north of London. Because I didn't want to drag a whole library with me I decided to take along only one book on statistics. From the books next to my bed I selected the lightest one which is Schaum's Outline: Probability and Statistics. And it was one of the best packing decisions I made.

It all started in one of my favorite book stores in London where I almost ignored the book in the first place. The book stuck out in the shelf of statistics books because of it's height and because of it's ugly front cover design. The pages felt like those of a telephone book. My expectations were as low as it's price tag (which was £12) and I ignored it. However, the shop only offered a very limited selection of books on statistics. So eventually I turned back to it out of curiosity wondering how bad a book on statistics could be. And then I stumbled upon one of the many interesting problems in the book and tried to solve it, and then I found the next interesting problem, and decided to move to the attached cafe. By the time the store was about to close I didn't want to part with the book. (One example of the fun problems in the book is: given 6 randomly sampled observations from a continuous population what is the probability that the last 2 are higher than the first four? (One way to solve it is to use calculus another way is to use combinatorics.))

The book does not only feature the ugliest front cover of any of the books I ever owned, it also contains many typos. And every time I tried to use the index it seemed to point me to random pages. For example, one typo can be found in the introduction to the multinomial distribution where they forgot an important exclamation mark. I don't understand how they manged to include so many errors in this second edition. In fact sometimes I wondered if errors were included to keep the reader alert. However, none of the errors I found were hard to identify as such. (In the case of the multinomial distribution formula there is an example just a few lines below the typo which uses the correct formula.)

Overall the book is amazing. I had a hard time choosing between packing up my tent in the rain or waiting for the rain to stop while reading a few more pages and drinking some hot tea (btw, I also highly recommend Trangia stoves).

The book is definitely suitable for people (like me) who work with probabilities and statistics on a daily basis but feel like they lack a solid foundation. It helps if you've had a basic course on statistics a long time ago and just want to refresh your knowledge. However, I think it is also largely and easily accessible to anyone who has not had any courses on statistics (although some understanding of calculus will help a lot).

The best part of the book is that it features lots of practical problems that help understand the theoretical concepts. The book is also structured in a way that makes it very easy to spend 30 minutes or less at a time with it. The topics covered include nonparametric tests, curve fitting, regression, and hypothesis testing.

Btw, I can also recommend: Old Man, Loch Lomond (and the West Highland Way), walking around the beaches of Holy Island at low tide (and reading books on the beach in front of Bambourgh), walking along some mountain ridge anywhere in the Highlands, listening to the choir in Durham cathedral, sleeping next to Hadrian's Wall, extreme hill walking in the Lake District... and Gwen recommend's Bill Bryson: Notes from a small Island.

Last.fm's API, Python, and tagging behaviour (Part 2)

Update: (2008/08/25) I fixed the same kind of bug I had in the previous post on this topic. While fixing it I decided to rerun it with a sample of 5000 instead of 2000 users. The code is fixed and the data is updated.

Last night I was a bit tired and quickly concluded "the numbers show...". But do they really?

To find out I use Python to do some simple statistical analysis to add some weight to the claim that older Last.fm users have a larger vocabulary and tag items more frequently.

First, I compute 95% confidence intervals for the percentage of non-taggers in each age group. Seeing the large margins (in the table below) helps explain why the age group 25-30 has a higher percentage than the age group 19-22. However, it doesn’t help explain why the age group 22-25 has a lower percentage. (I’d blame that on the relatively small and skewed sample, and I’d argue that they are still reasonably similar, both in average age and deviations of the percentages). (With the larger sample size this is not the case any longer.)

Computing the confidence interval is very easy. A user is either a tagger or not. The probability within an age group can thus be modeled with a Bernoulli distribution. The 95% confidence intervals for a Bernoulli distribution can be computed with:

z = 1.96 # norminv(0.975) for a 95% confidence interval
def binom_confidence(p,n):
    if n*p*(1-p) >= 5:
        return z*(p*(1-p)/n)**0.5

Btw, I couldn’t find the equivalent to the Matlab norminv function in Python. Any pointers would be appreciated!

To test the hypothesis that the vocabulary size of a tagger depends on her or his age I test the following: Given my observations, are all age groups likely to have the same vocabulary size, i.e, are the differences I observed just random noise? Since the distributions within each age group are far from Gaussian I can’t use a standard ANOVA. Instead I use the non-parametric version of a one-way ANOVA which is the Kruskal-Wallis test. In particular, I use the test to compute a p-value. The p-value is the probability that I would have made the same observation if the hypothesis that there is no difference between age groups would be true. (Thus smaller p-values are better. Usually one would expect at least a value below 0.05 before accepting an alternative hypothesis.) In this case the resulting p-value is nice and low indicating that it's extremely unlikely that older users don't have larger vocabularies.

Here are the results, and below is the Python code.

age   || % non taggers || tagger's median 
                          vocabulary size
14-19 ||     41.3-48.1 ||  6
19-22 ||     37.5-43.6 ||  7
22-25 ||     40.6-47.3 ||  9
25-30 ||     34.8-41.4 ||  8
30-60 ||     28.4-36.0 || 13
Kruskal-Wallis p-value: 1.04e-008

from scipy.stats.stats import kruskal
from numpy import asarray

def print_stats(age_tags):
    age_groups = (14,19,22,25,30,60)
    ll = []
    print "age   || % non taggers || " + \
           "tagger's median \n" + \
           "                          vocabulary size"
    for i in xrange(0,len(age_groups)-1):
        nonzeros = [];
        zero_count = 0
        for j in xrange(age_groups[i],age_groups[i+1]):
            for item in age_tags[j]:
                if item!=0:
                    nonzeros.append(item)
                else:
                    zero_count += 1
        conf = binom_confidence(
               zero_count/float(zero_count+len(nonzeros)), 
               zero_count+len(nonzeros))
        ll.append(nonzeros);
        print \
    "%d-%d || %8.1f-%.1f || %2d" % \
            (age_groups[i],age_groups[i+1], 
            (zero_count/float(max((1,len(nonzeros)+\
                                   zero_count)))-conf)*100, 
            (zero_count/float(max((1,len(nonzeros)+\
                                   zero_count)))+conf)*100, 
            median(nonzeros))
    p = kruskal(*(asarray(ll[i]) for i in 
                 xrange(len(age_groups)-1)))[1]
    print "Kruskal-Wallis p-value: %.2e" % p

Btw, that part where I use eval to convert my lists into function arguments could hardly be any uglier. I’m sure there must be a better way of doing that? (Thanks Klaas!)