voberoi

This is insane. I took his class on Automata and Complexity Theory (cs154.stanford.edu) two years ago, and I have friends who came out of the final exam for the class only a few hours ago.

I remember really enjoying this class, despite not having much of a knack for deeply theoretical CS. I was introduced to the concepts behind automata, turing machines, deterministic and nondeterministic computing, decidability theory, NP-completeness, and so on. I enjoyed this class entirely because of the way he taught it -- he took great care in the precision, concision, and clarity of his lectures.

What a deeply talented, smart, and clearly influential individual. My condolences go out to his family and friends.

While a speed boost is great and all, don't you have the added concern of making sure your blocks don't wear out on an SSD? Because log structured file systems commit metadata and data in large sequential writes to the hard drive, they have background processes that constantly move data around to prevent massive fragmentation from files that are later deleted. It seems like there would just be too much writing going on if what you want is a long-lasting SSD.

They do mention in the article that it would be a bad idea to use NILFS on your root partition because of heavy traffic, so lets say I decide to use it on a volume where I store all my media (where I pretty much write once, read a lot, and never delete). Now, a) why would I use an SSD for this given that they're so much more expensive and b) wouldn't I get comparable performance using an HDD here since my files will be laid out pretty much sequentially given my workload? Doesn't the same apply to companies storing mounds of data? Where is the middle ground where I would want to use NILFS on an SSD? One where I a) won't be sending enough write traffic to my drive to wear out my SSD too quickly and b) will be reading a lot of random blocks?

What am I missing? I see the merits in using NILFS, but I just don't see the purpose in comparing its performance against other filesystems on SSD's.

I didn't, and I'm graduating with a BS in CS from Stanford in a few weeks. We have an introductory stats requirement, but I found that the stats class I took my sophomore year for my (dropped) economics major was far more enlightening than my CS stats requirement.

I think the fundamental issue is that there's very little focus on statistical practice in most of these courses. Social scientists have it good: they're always taught how to deal with and interpret statistics in the same way they'll have to in their line of work. It's totally useless to throw a bunch of theory at students. I think that teaching statistics in the context of real problems is the only way (most) students will actually learn and come to appreciate how useful it is.

The department revamped the major this year, and they've introduced a mandatory stats class tailored to CS students: cs109.stanford.edu. I haven't looked through it at all, but I think it's a step in the right direction.

Finally, I have to give a shout-out to The Little Handbook of Statistical Practice (http://www.tufts.edu/~gdallal/LHSP.HTM) in this thread. It's an amazing resource for anyone who works with statistics. I've referenced it while doing performance testing, building an A/B testing system, and working on problem sets. From the website:

"My aim is to describe, for better or worse, what I do rather than simply present theory and methods as they appear in standard textbooks. This is about statistical practice--what happens when a statistician (me) deals with data on a daily basis."

Read it now!

I think this article is great once you get past the rant-iness of it. He makes a bunch of valid points and it's true: a lot people actually don't take these pretty crucial details into account when they're dealing with statistics.

It's kind of a shame that his message is diluted by its delivery. Case in point: about half of the comments here aren't even about statistics!

Their report (linked to from the post) goes into greater detail: http://developer.yahoo.com/blogs/hadoop/Yahoo2009.pdf

I'd love to know why the 500 GB and 100 TB sorts ran at about half the speed of the other two (~0.5 TB/min as opposed to ~1 TB/min).

If you've got the time, I highly recommend taking CS107 at Stanford (for free): http://see.stanford.edu/see/courseinfo.aspx?coll=2d712634-2b....

I became incredibly comfortable with C -- memory management, handling pointers, having a very clear picture of how types are represented in memory -- after taking that course. The lecturer (Jerry Cain) is fantastic as well.

The course changed this year so I don't have any thoughts on the current CS107 offering if you plan on taking it through their online course offerings for pay. The lectures, syllabus, and materials on Stanford Engineering Everywhere (OpenCourseWare at Stanford) are all from the course I took two years ago.

I use last.fm recommendations + http://seeqpod.com

I suggest checking out the first video here: http://www.cloudera.com/hadoop-training-basic. It's a little more than 20 minutes long, and the lecturer talks about the problems that MapReduce and HDFS (Hadoop Distributed File System) are designed to solve.

If you're super curious about MapReduce and are down to spend a few hours learning about and playing around with Hadoop, most certainly check out the rest of the videos (you can leave out the ones about Hive) and work through the first two exercises -- the virtual machine they provide makes it very easy to implement and run your first MapReduce job. Doing this will answer your question better than anyone explaining it to you can. If you'd like to go even further and learn about how Hadoop works under the hood, buy the rough cuts version of Hadoop: The Definitive Guide here: http://oreilly.com/catalog/9780596521998/.

The Little Handbook of Statistical Practice: http://www.tufts.edu/~gdallal/LHSP.HTM

It's the best stats resource I've come across, particularly for the stats behind split testing and multivariate testing. I've used it as a reference while building an A/B testing system in the past.

For considerations in building a system to perform split testing and using such a system, the folks working on Microsoft's Experimentation Platform published an incredibly helpful paper: http://exp-platform.com/hippo.aspx

mattobrien's explanation was awesome. A lot about the current crisis was demystified for me in an episode of This American Life: http://www.thisamericanlife.org/Radio_Episode.aspx?episode=3...

I'd recommend this to anyone who's looking for a well-explained overview. Set aside an hour or so to give it a listen.