UK Hadoop User Group Meeting

Last Tuesday the first UK Hadoop user group meeting took place in London. Johan did a great job in organizing it and speakers included Doug Cutting from Yahoo! (who leads the Hadoop project).

Here's are some links:

• Yahoo! developer network blog mentioning the event. (Includes a video of interviews with some of the presenters.)
  
• Doug Cutting: Hadoop overview
  
• Tom White: Hadoop Web Services on Amazon S3/EC2
  
• Steve Loughran: Deploying Apache Hadoop with Smartfrog
  
• Mark Butler: Distributed Lucene for Hadoop
  
• Last.fm related talks:
  • Martin Dittus: Hadoop usage at Last.fm
    
  • Elias Pampalk: Radio Log Analysis for A/B Tests
    
  • Johan Oskarsson: Hadoop: Lessons learned at Last.fm
    
  • Klaas Bosteels: Dumbo: Hadoop streaming made elegant and easy
    
  • Tim Sell: PostgreSQL to HBase Replication
    
  
  

Seems like the interesting talk by Miles Osborne on "Using Nutch and Hadoop for Natural Language Processing" is still missing on skills matter's website. I'll update this blog post when the talk is added.

Hadoop in London

August 19th there will be a Hadoop workshop somewhere in East London which some of my colleagues are helping to organize. Details can be found here. Further announcements will follow. The speakers include Doug Cutting (works at Yahoo and leads the Hadoop project). Johan and Martin will be talking about how we use Hadoop at Last.fm. Applications at Last.fm range from counting how many users listened to an artist to learning from skipping behaviour to improve our radio streams.

Simple Stuff

Jeff Hammerbacher from Facebook’s team on data and analytics gave an interesting talk and it can be viewed here (which might be broken, here is a direct link). I’d highly recommend it to anyone interested in what happens behind the scenes on websites dealing with a lot of interesting data.

Some of the stuff he talks about is related to what happens at Last.fm. For example, at Last.fm we also use Hadoop.

I found the part around minutes 33-38 most interesting. He talks about the skills needed to work with their data. He mentions that being able to write code is a must. He mentions that most of the people on his team write more code than they did in their previous positions. And he mentions that very simple statistical tools such as curve fitting and understanding statistical significance can be used to solve most of their learning from data challenges. He talks about how visualizing data is very important (e.g. to identify and understand outliers).

Obviously music recommendation is a much more complex problem than any of the challenges Facebook is facing. Scrobbles, tags, skipping behaviour, etc require very different treatment than the data Facebook gathers. Or maybe not?

To some of the most interesting things I’ve had the pleasure to work on at Last.fm I’ve only applied very basic statistical techniques: non-linear curve fitting and measuring the significance of improvements. However, while the “machine learning” parts could hardly be any simpler, the complexity of dealing with terabytes is something completely different.

Btw, at Last.fm we are hiring someone to work on data and analytics and we also got a position related to data warehousing. Both positions would be facing challenges very closely related to the stuff Jeff talks about. Except that the data we have is a lot more interesting! It made me a bit sad to hear that one of the things they were actually looking at is communication streams between universities… you’d think they’d have a lot more interesting insights to gain? ;-)