pskomoroch

more projects here: http://stackoverflow.com/questions/495698/what-resources-are...

and here: http://delicious.com/pskomoroch/ab

I did an analysis on this a while back that told the same story, there is a blog post that includes a nice visualization of the share of travel search categories across the companies mentioned in the article:

http://www.juiceanalytics.com/writing/search-competition-tra...

Direct link to the chart http://media.juiceanalytics.com/images/pattern_comparison.pn...

Distributed, Large-Scale Latent Semantic Analysis by Index Interpolation. Sebastiano Vigna

http://tinyurl.com/d99779 [PDF]

So by this logic, they could raid Amazon's datacenters and seize everything because a few people were using EC2 to send spam?

reserved instances: http://aws.amazon.com/about-aws/whats-new/2009/03/12/amazon-...

I'm planning on going into more detail in my next post, walking through some of the Python streaming mapreduce code. Is there anything in particular people want to see?

Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic

Here is a direct link to the full code example from the article:

http://elasticmapreduce.s3.amazonaws.com/samples/similarity/...

You will get much better performance out of Hadoop if you do this in Java, but I thought python would be more readable for a wider audience. Hadoop 18.3 doesn't have numerical sort in streaming (see https://issues.apache.org/jira/browse/HADOOP-2302 ), so the code is slightly more complicated than it needs to be. Dumbo will also probably be supported in the next release, then the code example would be even simpler.

This is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines.

Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/input/", "-output", "hdfs:///home/hadoop/output2/",

Step 2:

"-input", "hdfs:///home/hadoop/output2/", "-output", "hdfs:///home/hadoop/output3/",

etc..

There is no explicit support for mounting data from EBS yet, but in theory I think you can hack something together in the initial mapper to mount an EBS volume to be used in the second map task...

The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers:

http://elasticmapreduce.s3.amazonaws.com/samples/similarity/...

AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming.

The article is here:

http://developer.amazonwebservices.com/connect/entry.jspa?ex...

Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.

pete pete@datawrangling.com python, machine learning, statistics

I'd take problem solving ability and creativity over typing speed any day, but one area where typing speed seems to help is for live coding or instructing others...

There is a start at it here:

http://www.scipy.org/scipy/scikits/wiki/MachineLearning

Still some active development, but needs more people:

http://projects.scipy.org/scipy/scikits/browser/trunk/learn/...

I was at the meetup and there were few bits about how Facebook uses R, Hadoop, and python for different levels of data analysis. The guys sitting behind me were talking during the whole presentation which was really annoying... please don't do that.

http://twitter.com/peteskomoroch mostly tweets on python, machine learning, general data crunching topics.

I've added static tags to each link which should help a bit, If I can find a way to quickly cluster them together into logical groups, I'll so that as well.

Yes, I'll take a pass at sorting these into groups this afternoon or at least add more descriptive information. Some of the original link titles are a bit mysterious, like this one: "Voter registration data; or, HERE IS YOUR HOPE, YOU FOOLS! « The Edge of the American West"

I'm going to toss the ball into infochimps court for any deeper organization of the data, since they are already set up to handle it.

Thanks to whoever submitted this. It is an old post, but I just updated it this week with 230 new dataset links I collected in the last year. The page has about 400 datasets listed now. It also includes a tagged json version for groups like infochimps who are working to categorize open data or scrape it from the web.

I was thinking of doing a basic Django + Selenium + Hudson continuous integration how-to blog post, but this blows me out of the water :)

Google open sourced one: http://google-opensource.blogspot.com/2008/09/cmockery-easie...

I have a really tough time with this, and often bounce back and forth between side projects.

Erdos was supposedly really good at helping people pick research problems to work on which would have just the right difficulty given their capabilities. I can't find the exact quote, but it was in the book "The Man Who Loved Only Numbers"

pg had some advice in an essay on high school: http://paulgraham.com/hs.html

Excerpt:

"Put in time how and on what? Just pick a project that seems interesting: to master some chunk of material, or to make something, or to answer some question. Choose a project that will take less than a month, and make it something you have the means to finish. Do something hard enough to stretch you, but only just, especially at first. If you're deciding between two projects, choose whichever seems most fun. If one blows up in your face, start another. Repeat till, like an internal combustion engine, the process becomes self-sustaining, and each project generates the next one. (This could take years.)"

My friend Tom is a python hacker and works there, I pointed out the thread... maybe he'll chime in.

I used to have a lot more time for non CS reading and actually made an Amazon list on this topic back in 2005:

http://www.amazon.com/gp/richpub/listmania/fullview/2INJSM38...

1. On War (Everyman's Library (Cloth)) by Carl Von Clausewitz

2. Leadership: The Warrior's Art by Barry R. McCaffrey

3. Small Unit Leadership: A Commonsense Approach by Dandridge M. Malone

4. The Defense of Hill 781: An Allegory of Modern Mechanized Combat by James R. McDonough

5. The Art of Maneuver: Maneuver-Warfare Theory and AirLand Battle by Robert Leonhard

6. Strategy: Second Revised Edition (Meridian) by B. H. Liddell Hart

7. The Art of War by Niccolò Machiavelli

8. Hagakure: The Book of the Samauri by Tsunetomo Yamamoto

9. The Emperor's Handbook: A New Translation of The Meditations by Marcus Aurelius

10. The Art of War (Shambhala classics) by Sun Tzu

11. The Prince (Bantam Classics) by Niccolo Machiavelli

12. Evolutionary Game Theory by Jörgen W. Weibull

13. On Guerrilla Warfare by Mao Tse-tung

14. The History of the Peloponnesian War: Revised Edition (Penguin Classics) by Thucydides

15. The Histories (Penguin Classics) by Herodotus

16. The Persian Expedition (Penguin Classics) by Xenophon

17. Plutarch: Lives of Noble Grecians and Romans (Modern Library Series, Vol. 1) by Plutarch

18. Plutarch's Lives, Volume 2 (Modern Library Classics) by Plutarch

19. Livy: The Early History of Rome, Books I-V (Penguin Classics) (Bks. 1-5) by Titus Livy

20. The History of Rome from Its Foundation, Books XXI-XXX: The War with Hannibal (Penguin Classics) (Bks. 21-30) by Titus Livius Livy

We launched our new Django webapp this week at http://www.concentrateme.com (with a free version if your site has google analytics). One of the features involves pattern mining within search query data which allows you to perform some interesting competitive analysis. The post goes through a basic case study for a few Travel websites which HN people trying to find a niche might find interesting.

I talked to jeff barr @amazon a while back about these kind of issues (if you have large datasets to shuffle on/off EC2, is there an efficient & cost effective way to do it?) - shoot him an email or comment on the aws blog so they know more people have this need. Amazon public datasets can help with this for some use cases, for example certain bioinformatics or other academic use cases.

Numpy/Scipy syntax is very close to matlab, but python is a lot more powerful. I port matlab code over to python pretty frequently.

A toy example is here along links to other comparisons: http://www.datawrangling.com/python-montage-code-for-display...

http://code.google.com/p/canopy-clustering/

"This repository contains a set of example map-reduces using Hadoop.

Specifically, there are examples for:

    * Pagerank
    * Canopy-clustering "

more examples like k-means here: http://delicious.com/pskomoroch/hadoop+code

warning... visiting the versionista site crashed my iphone

I posted a bunch of machine learning video courses here:

http://www.datawrangling.com/hidden-video-courses-in-math-sc...

videolectures.net has a ton of material as well

Hastie and Bishop are good books to start with, assuming you have a reasonable mathematics background.

Hopefully they fix the usability issues they introduced quickly. I'm a long time user and I might bail. More feedback here:

"Suggestions and ideas: What do you dislike about the new design?" http://support.delicious.com/forum/comments.php?DiscussionID...

Maybe contact Bernard as well:

Bernard Kerr, lead designer user experience designer @ delicious http://infosthetics.com/archives/2008/04/bernard_kerr_yahoo_...