christopheraden

I'm worried your explanation may be confusing to people in some places.

The bell curve and Gaussian curve (and normal curve) are almost always synonymous. Bell Curve describes its shape, while the Gaussian curve describes the guy who derived that nasty-looking pdf. While some other distributions may look bell-shaped (especially ones that converge to a Normal in some asymptotic sense, like the t-distribution), people will look at you funny if you refer to anything other than the normal (or in rare cases, the t) as "The Bell Curve".

The point about "the data is laid out so the sum of the data adds up to 1" doesn't make much sense since the Normal takes values on the entire real line. Could you clarify?

The procedure you describe is not what most people would call "normalizing", so much as breaking the data up into quantiles/percentiles. Normalizing the data would be for the instructor will subtract the average from everyone's raw score, then divide by the standard deviation. If the raw scores themselves were distributed normally, then these new "normalized" scores are now standard normal, so letter grades are given based on how far away from 0 a student's scaled score is. This procedure is pretty flawed since scores are usually not super close to normality (many exams are not symmetric in scores, the distribution may have several modes, etc). The procedure you describe makes no assumptions about the underlying data, and does not require normality, so I like it more.

Consider that the source is High Scalability, a blog that focuses on how orthodox methods are inadequate for 1-10M concurrent connections and how different methods are being employed to reach these lofty goals, I think the advice is pretty spot-on.

It's important to consider that if your objective is scalability and performance, then the article's advice is appropriate. The article's title ought to read "Stop Using Linked Lists if you care about High Scalability", but I attach the dependent clause on many titles I read from that site.

I like that you ask for mindfck movies. They definitely hold a special place in my heart. A lot of the entries here have been very sci-fi. I tend to like more historical or present-day dramas with great acting that expose some sort of undesirable human behavior, but I snuck a sci-fi movie on the list ;). You can't take the nerd out of the cinephile!

There Will Be Blood - No "great movie" list is complete without a movie that stars Daniel Day Lewis. Go watch all his movies if you want to see what it's like to take acting to the extreme.

(M) The Spanish Prisoner - Probably my favorite suspense movie, along with a Hitchcock or two like Vertigo or Rear Window. This is one of David Mamet's best, in my opinion. All these random things happen in the beginning of the movie that make you question why they bothered mentioning them, and then the catalytic action ties it all together expertly.

Pirates of Silicon Valley - This is the lightest contribution on the list, but probably the most relevant to HN. For what it's worth, both Woz and Bill Gates have described their portrayals as fairly accurate.

Alien - Definitely plays out more like a space horror film than an action movie like the sequels, the acting and suspense is fantastic, not to mention the costume work that went into the alien on the part of HR Giger (if you want a true mindfck, skip the movies and go read his website--terrifying).

Grave of the Fireflies - I wanted to include an animated film on my list, but I really had trouble deciding between Interstella 5555 (Daft Punk's Discovery set to an anime with no talking whatsoever. Great concept) and a Pixar movie (personal favorite: Up). I went with this because I figured you've probably seen Pixar movies. Grave of the Fireflies is a super depressing Ghibli film about the aftermath of the Kobe fire bombings in WW2. The way the movie shows the horrible living conditions and bleak outcomes of the two main characters (the boy is in his early teens and his sister must be around 4 or 5--I can't recall). The movie manipulates your emotions more than any other film I've included on this list. I don't think I've ever had as strong an emotional reaction to a movie as when I saw this one.

While I might generally agree that the interview process is asinine and a rat race, I actually use some of those formal methods and algorithms in my day-to-day work--and I'm not even a computer scientist, nor do I have a CS degree.

I'll bet things are different in the startup world, but I've encountered several instances where having a very crude understanding of time complexity served me well. Hashing has been an especially powerful tool in my work, where the size of the data I work with really highlights the difference between an O(n) operation and an O(1) operation.

I would consider the interviewer asking questions about algorithms to be a sort of Fizz Buzz for the more technical positions. Perhaps it is misapplied to be asking technicalities of algorithm questions to people that will be doing more design-oriented work, just as it may be inappropriate to ask your sales force applicants to write programs during their interviews.

That said, most of the interviews I've conducted would not have been able to discriminate well the qualities that I believe make a great programmer. A focus on code longevity, discipline in conducting code review, dedication in writing unit tests, and how they search for information they don't know the answer to are important qualities that a technical interview will largely be unable to see.

From perusing the main page, this looks exactly like what I want. Thanks for the awesome suggestion.

You will find the intro books don't talk much about parallel computing. Most of the general data sets in intro books will be no more than 30 observations. They are trying to teach classical methods moreso than useful computational techniques. As for parallel statistics, I don't have a good book recommendation. Most of my knowledge on the topic comes from papers and vignettes from the R community and not books. Maybe check out one of those O'Reilly books about big data techniques?

I haven't seen this OpenIntro statistics before. I'll check it out!

Yankoff, you might want to be more specific. Intro statistics in general, or for computer scientists, or scientists, or looking to learn R at the same time? I liked Freedman, Pisani, and Purves [1], and have TA'ed using McClave, Sincich, and Mendenhall [2]. You may want something a little more advanced than these, but they are pretty good for intro level.

[1]: http://www.amazon.com/Statistics-4th-David-Freedman/dp/03939... [2]: http://www.amazon.com/Statistics-11th-Edition-Book-CD/dp/013... [3]: http://stats.stackexchange.com/questions/421/what-book-would...

Seems more like a brain fart than an error in understanding, especially because he didn't highlight why it was important to have unbiasedness in the first place. Having to explain the unbiased estimator for standard deviation would probably not have contributed much. It's more of a technicality than anything of philosophical importance--the square root of the unbiased estimator of sample variance is pretty accurate.

The usage of n-1 instead of n in the denominator was a question I was always asked when I TA'ed for intro statistics classes in grad school. An explanation of unbiasedness might be warranted if this is to be an introductory primer.

Hey Evan, from one statistics guy to another, thanks for fighting the good fight :). The formulas might benefit from examples, especially with some of the more complicated cases (KS test and onwards). The important part of statistics comes from knowing _when_ to apply something, rather than _how_ to (that part is just math/numerical analysis). A mention of the assumptions of each of these intervals would be good, too. Too often I see conclusions invalidated by using a probability model that doesn't make sense. This is a common failure I see with using a Wald interval for the slope of the regression line.

On #7: ACID is a concept that matters just as much now as it used to, but I think a lot of people that thought ACID was important realize that it might not be so. I wouldn't say a RDBMS is uniformly better than text files, but in some applications, I wouldn't have it any other way.

When dealing with crucial data where robustness and correctness are paramount, an RDBMS may not be a bad fit. I cringe at the thought of an insurance claims system being done in NoSQL or JSON instead of a more robust system. Things are probably different in the startup world, but I'd rather take a huge hit to performance than a multi-million fine and audit for losing records from my DB.

Where's openbox? #! users ought to be outraged!

I work in corporate where there is no admin access to my work machine and we're expected to use Notepad for small tasks and Word for big ones. The windows version of Gvim gives me all the familiar keybindings I have on my Linux box at home, with features my coworkers can't get in their editors (I use sed several times a day--what a lifesaver).

The unity in keybindings between my home coding environment and my work coding environment means there's very little cognitive dissonance--my muscle memory does all the work. People who don't spend their time equally split on vastly platforms probably don't understand just how huge of a godsend a cross-platform editor is (Emacs/Sublime have this too--not meaning to suggest there aren't other editors that have this).

Power analysis and CI's should be elementary, but I would assert that they are actually not commonplace. Most people have a very surface-level understanding of the latter, and little understanding of the former. In my opinion, A/B Testing has actually done a great service to power analysis. I have seen many experiments in the academic world (social sciences are somewhat notorious for this) forgoing the power analysis for various reasons (fear: they would not be able to get the sample size needed for 80% power, inability to control sample size: you take whatever you can get with a convenience sample). As a statistician, I breathe a sigh of relief with the amount of emphasis power analysis receives in the A/B world. It's a step in the right direction (if you're an acolyte to the dark world of Neyman-Pearson).

As for bringing better stats to high tech, I've thought of this as a wonderful challenge. I'd especially like to see more focus on not violating modeling assumptions (more non and semi-parametrics), and using some more modern techniques from the ML and Bayes literature.

Hypothesis testing is so last century :). Would love to discuss it further with some similarly-inclined HN folks.

Sorry for all the parentheticals. You'd think I was a lisp programmer with the amount of parenthesis I used.

"Use a dedicated statistical package from the '80s" Is your Wizard app not also a dedicated statistical package? Also, I'm being pedantic here, but how many "dedicated statistical packages" are actually from the 80s? The only ones that come to mind are Stata and Statistica.

Is there a way to view the source code or formulas you use on your pages? There's been a strong push in the academic statistics world for reproducible research, which means public data, open source statistical code.

I ask because I'm curious about your two-sample t-test. Does it pool the variances for all values of the two standard deviations? Pooling doesn't make sense when one sd is 50 and the other is 2...

Like all of A/B testing, it's applying a _very_ old statistical method (Chi-Square was one of the first modern statistical techniques--by that I mean it's 113 years old) to an area where statistics has not commonly been used. This makes it seem wonderful and novel as countless people suddenly realize that statistics can be applied to fields that were previously untouched by quantitative analyses.

The statistics being used in the A/B testing world is stuff you would've learned in your very first statistics class, it seems. From the success of Optimizely and VWO that the focus is definitely more on the viz and presentation than it is on using any cutting-edge techniques.

I meant checking assumptions not just to see whether the use of the big data moved a business metric, but also that the model makes sense from a statistical perspective.

A lot of statistics in business does not bother to check modeling assumptions. Models are chosen based on whether they've been used in the past and what the team is familiar with.

I don't doubt that big data (as we call it now) will one day rule. Ronald Fisher would keel over if he saw the size of datasets we work with nonchalantly on a daily basis. 50 data points (the size of the Iris data) is laughable these days.

My reservation with big data is that the technologies are often unnecessary for the size of the tasks being done. Other than a few data scientists working on truly large projects, most of the big data talk I hear comes from people who aren't fighting in the trenches (execs, marketing, journalists).

Statistics involves checking modeling assumptions. A lot of what I've seen with the big data people is the repetition of algorithms to the exclusion of understanding and checking modeling assumptions.

While it's nice that the big data craze is making statistics more popular in the mainstream press, it is important that statistics does not become just an application of numerical methods without consideration of underlying assumptions. I stress this because this has been largely underappreciated in my experience.

The fact that so many people are calling things "big data" when the data is not high volume (the most popular definition I've seen is the 5 V's definition--big seems to be a misnomer in this case, as only volume could really be called a measure of "big") lends credence to your statement.

I am grateful to finally see this in an article. The "big data" craze is being pushed in areas where it really doesn't make sense. We've been bit by the Big Data bug where I'm at, but it's not coming from the statisticians. It's usually the executives proposing a shift to big data.

People underestimate how much work it would be to shift an old server onto modern technologies and tell the statisticians to use MapReduce and NoSQL instead of SAS and SQL. If the Fortune 500 world has taken this long to catch on to R, imagine how long it'll take to completely change the DBMS and analysis software!

I met Chad Whitacre at the Gittip booth at PyCon (the heart-punching coin machine was pretty cool!). I found him to be very earnest and interested in the perception of his company.

I'm glad he felt strongly enough about the openness to refuse to conduct the interview. Hopefully the community can rally behind him in solidarity.

The correspondence reflects very poorly on the TC reporter.

Disregarding the morality and ethics that are often center-stage, they are faced with really interesting technical hurdles. It's an old article, but I was fascinated to hear about some of the challenges they face as well: http://highscalability.com/blog/2012/4/2/youporn-targeting-2...

Most of the work I do involves recent data--cycles are six months at most and 3 months on average. 3 months of data, sifting by a pretty strict filter, it's not unsurprising that hundreds of terabytes of claims over years and years gets filtered into a few GB.

Start to finish on jobs is a few hours (though it can be a few days if the filter is less strict or the task is more complicated), including pulling the data from the warehouse.

Without a doubt, the bottleneck of the process is the data warehouse query. I'm sure having a more distributed database (it's DB2--I've been more pleased with Teradata's speed) could make the queries faster, but things are slow to change. Second to the query, the latency from working with a remote server (I'm in CA--the server's in Minnesota) adds latency if there's something I need to pull to the local machine. The actual SAS code (Is it still "high level" if the syntax models Fortran? I kid, I kid.) takes a negligible amount of time compared to the query.

The types of queries we run against it don't require real-time results, and we do a pretty heavy amount of subsetting. By the time it reaches the point where we do numerical summaries and statistics, the largest set I've worked with here was around 30GB. Most times it's around 5-10GB.

I've had a similar question before. I have heard of as few as 60,000 observations was considered "big data" [1], yet at my company, we generate about 60 million pharmacy claims every 3 months, and no one here calls it big data. In terms of storage, it's on the order of a couple hundred terabytes for all our data. This is considered small enough that we can query it with traditional SQL.

Big Data, the experts say, is more about the novel way you analyze data, relative to the difficulty of the problem. A speaker at PyCon who was talking about algorithms and data structures for handling genetic data had a term that I like quite a bit better: "Data of Unusual Size" (C. Titus Brown at MSU was the speaker).

"Big Data" is a really big buzzword right now, but the term is overused and often does not convey the meaning it's supposed to. "Big" is a relative term. The novelty of how much data is being used as opposed to how much used to be used (in the sumo case, they had never handled so much data before) is what makes it big.

As for Hadoop, you'd want to use it when it's no longer feasible to keep your data stored in an RDBMS, or when speed becomes an issue, or when you want your schema to be more flexible than an RDBMS. If you are not concerned with the reliability of your data (RDBMS make the safety of the data a paramount priority--read the wikipedia page on ACID to see these guarantees), there's plenty of reasons for choosing Hadoop.

[1]: http://www.wired.com/wiredenterprise/2013/03/big-data/ [2]: http://en.wikipedia.org/wiki/ACID [3]: http://hortonworks.com/blog/4-reasons-to-use-hadoop-for-data...

Thanks for pointing out the distinction. You're right that this is what I was going for, but the articles I've seen thus far about the inaccuracies in Excel deal mainly with the poor capabilities of the optimization routines, RNG, and distribution functions. If you're balancing a budget with numbers that involve 2 decimals or taking measures of central tendency on similarly-scaled values, Excel isn't woefully inadequate.

We got around the inaccuracy problems where I'm at by doing all the sensitive calculations in a more capable program (SAS where I'm at now--R when I consult), then exporting the results to Excel. It avoids most of the problems faced by the poor statistical tools in Excel. It's frustrating to think how difficult it is to cut Excel from the pipeline, but such is the dilemma with Excel being as virulent as it is.

They are just now moving people off MSO 2003 at ours as well, following complaints from several people that they can't open the newer xlsx, docx, and accdb file extensions. IT and Clinical Analytics are the first wave to move to "brand new" MS Office 2010, as of Jan'13.

Sure hope you aren't doing any life-threatening numerical computations in your version of Excel!

I think that in jobs where statistics is the main function of the job, you will often not find them using Excel. The business intelligence folks at my company use it for cursory analyses, but I often question the rigor involved in their practices, and using Excel to do statistics is icing on the cake. I would trust the results of SAS or R or, hell, even Minitab, over the results I get out of Excel. Microsoft seems more content making their product shinier than improving the underlying numerical accuracy of their methods.

The sooner people realize it's not designed for real statistics, the sooner people can stop hearing statisticians rant when they receive "Report_Analysis_Feb1-2012_Update_Final-v3.xlsx".

They do one of these every iteration of Excel, and the results are usually similar. One day, Microsoft will make a good version of Excel, but I'm still waiting. http://homepages.ulb.ac.be/~gmelard/rech/gmelard_csda23.pdf

I currently use Dwolla for transferring funds over to Mtgox. Given this information, is there a better service for getting funds from a bank account over to Mtgox without doing a wire transfer?