tansey

The main problem I was having is that you are assuming our observation variable is the latent skill or potential value variable (which you're calling x here). However, the article by PG was talking solely about the average of returns (let's call it y).

So the reason I was confused is that, assuming that the outcome of a startup is dependent only on x, we are really observing y ~ f(x) = \int_0^1 g(x)h(x)dx, where h is your cut-off criteria for x, g(x) is some unknown payoff distribution for a given skill level, and I'm assuming our x is in [0,1] without loss of generality. So in essence, the real problem here, even if you could see all of the individual returns for a given portfolio, is that you have to perform a very, very difficult deconvolution problem. And I'm pretty sure it's non-identifiable without some other information or additional parametric assumptions.

Thinking out loud a bit, let's assume that y is actually log(return), where a return of 1 is breaking even and 0 is losing everything. Since log(0) is undefined, most startups return 0, and very few exit for less than 1, I would think we could model this as a point-inflated normal distribution: p(y) = c * \delta_0 + (1-c) * N(\mu, \sigma^2). Given this, we could then model our latent parameters (c, \mu, \sigma) as being functions of x. Since the model is separable, we can even just look at the zeros and non-zeros in isolation. Then we can come up with a test from there, but I'm not really sure what that test would be at this point. Anyway, that's a completely different line of thinking, but it seems much more tractable in practice.

PG's articles are generally filled with good intuitive insight. Unfortunately, statistics can be very tricky to turn into folksy wisdom. Rules of thumb like "you need 30 samples before you can say anything" that are derived from the CLT are a good example of ones that work well enough in practice, even if they obscure some underlying subtleties. This article is an example of a rule that sounds simple, but actually has so many asterisks that one would expect it to be mostly useless in practice.

If women are performing better on average, it doesn't mean that you should invest in more women necessarily. What if all the remaining candidates would have a negative mean return? If they included Uber and all of a sudden the women now underperform men, does that mean they're biased against men and they need to invest in less women?

There's just so many statistical fallacies at play here that it's a shame that Jessica, Sam, or Geoff didn't point out that maybe someone with a stats background should read the article first before publishing it.

They're so fast to run though, that just doing warm-starts and a huge solution path (or grid in the case of additional penalties) with a BIC selection criteria is a pretty decent way to auto-tune the parameters.

The way to use it in image processing is to basically think of each pixel as a point on a graph, with edges between its adjacent pixels. Then you apply a least squares penalty with an additional regularization penalty on the absolute value of the difference between neighboring pixels. The result is regions of constant pixel color.

The idea (from 5 minutes of skimming plus watching the little 3min tutorial) seems to be that any multi-dimensional time series implicitly contains all the information necessary to reconstruct it entirely within a single dimension. The authors show an example of a 3-dimensional time series with a butterfly-type pattern that you can reconstruct just by looking at the values of X(t) and then using X(t-a) and X(t-2a) as the other dimensions. Not sure how that leads to prediction-- can someone explain that part?

IMO, the "data science" label is too broad to properly differentiate statistical engineers. A fine definition for a data scientist is someone who runs experiments on user/company data and can assess the results. It's important work, but you don't need a PhD in stats or ML to do basic hypothesis testing.

You could simply call them "machine learning" experts, but that could be a bit too academic. People who are focused narrowly on theory or niche areas may be experts in ML, but they may also never do anything outside of running matlab simulations. It's unlikely that those people will make very good statistical engineers since they may never have had to think about the challenges involved in scaling algorithms.

Right. We're talking about the people more likely to be in the 15%.

>...and (as Sam Altman later says[1]) libraries are available for the less fortunate. Compare that to biology. Or chemistry. Or mechanical engineering. Or becoming a plumber.

That seems a little disingenuous or at least dismissive. You really can't compare the experience of learning to program on your own laptop, where you can install your own software and work in the comfort of your own home, to a public library that you have to walk to 30 minutes each way in a cold and dangerous neighborhood, to use a machine that is probably 10 years old and requires you to use internet explorer and prevents you from installing anything at all.

In fact, if it were really that easy to learn programming on the web, then it should be just as easy to learn biology or chemistry or mechanical engineering. The information is all on wikipedia-- there's even web courses on Coursera that people can take that are taught by some of the best lecturers in the world. You can even fill in gaps in your K-12 knowledge via Khan Academy.

> Mentors are helpful, but they're certainly not required to learn programming.

I'm going to say this is just false. I do not know anyone who is a competent programmer who has never interacted with someone else. I do know plenty of people who have tried to learn how to program and given up because they got stuck and couldn't figure out how to get to the next level. Having that mentor there is crucial even for the most determined individual.

> The only really necessary things are (again) a computer, internet access, and time.

This is exactly OP's point. The kid in this story has none of these, except the time component. And as he gets older and has to work his way through college, that free time is going to shrink even more while he has to compete with people who are increasingly better prepared for the courses he's in.

I think overall that coding is simply easier than the other disciplines you're comparing it to. People who learn "how to program" are not learning computer science, in the same way that people teaching themselves how to cook meth aren't learning chemical engineering. They're both just people who are motivated to learn a specific skill that is in high demand and lets them see the product of their work directly and immediately. The quality of that work though will suffer if they don't receive a more well-rounded education in the general field, which is much harder to receive without mentoring, time, money, and all the other conveniences of middle class life.

That's basically what is happening with the tool, I think. It is asking for how many users per day you get in order to approximate the sample size for x days, then it's asking how much power you want. Power is the likelihood of detecting a difference if there is one. It also asks what confidence level you want. All of those together give you an approximate answer to the amount of time, assuming # of users/day is roughly constant.

There is even some recent work on a "Big Data" (distributed) version of the bootstrap from Michael Jordan's group [1]. It's also pretty easy to implement and can be really useful in practice.

[1] http://onlinelibrary.wiley.com/doi/10.1111/rssb.12050/full

I have a particularly relevant horror story. For one of my graduate classes, I built a game where two people (a liar and a truth teller) would try to convince a third person (the judge) they had experienced something. The goal was to see if people used different language when lying than they did when telling the truth. It was a fun project and we were focused more on the NLP/ML side of things. Turns out to be reasonably possible to separate our ~500 examples via a linear SVM, with some interesting separating words. That's where our claims stopped.

Then I took the results to a psychology prof. They loaded the data into SAS or something and it proceeded to perform HUNDREDS of independent t-tests. The results came out in a few seconds and the professor exclaimed "Oh look! Pronouns are statistically significant! Oh and possessive nouns too!" -- I cringed.

The flip side of this is that as statisticians, we do know how to handle different kinds of testing. If you're going to be looking at all these different outcomes, that's fine, but you just need to correct for it. We've had Bonferroni correction since the 1960s; Benjamini-Hochberg and related false discovery rate methods have been around for almost 20 years now. In fact, there are even situations where the data can help define a prior for your hypothesis testing [1].

Lastly, there is this quote from the article:

> There is no statistical quality board that could enforce such larger analyses—nor would we believe such coercion to be appropriate

I'm not sure if that's such a bad idea. In theory, that should be the job of the journal reviewers and editor. In practice, it's often the blind leading the blind, with confirmation bias thrown in to boot. Maybe we need an SRB as a companion to IRB.

[1] http://arxiv.org/abs/1411.6144

[0] http://radimrehurek.com/gensim/

[1] http://www.cs.princeton.edu/~blei/topicmodeling.html

So... who wants a NIPS paper? Noel and I are happy to just be 2nd authors, since you know... it was our idea and everything. ;)

From a machine learning perspective, frequentist methods are great because they scale. All of these fancy nonparametric Bayes methods coming out these days take weeks to run and include so many approximations that you've got no idea whether what you're actually inferring is truly similar to the original model.

The biggest problem with frequentist methods is that they are conceptually odd in their approach. Bayesians take the view that you should fix the data and integrate over the possible parameters. Frequentists take the view that you should fix the parameters and integrate over the data. That is kind of weird to me.

So you either start out conceptually pure as a Bayesian and compromise to the point of meaninglessness or you abandon your morals at the start as a frequentist and things go smoothly from there.

Edit: As noted below, it's actually Exp3 rather than UCB-1. I glossed over the part about potentially adversarial bandits.

The only real "deep" learning here is that they used a GPU library and stochastic gradient descent to perform Q-learning updates on a network with 3 large hidden layers. It was an interesting application paper, but I suspect that the Google acquisition is for something more novel than this work.

They are trying to make Silicon Valley and startups look like Hollywood and scriptwriting. The first episode has the main characters (startup co-founders) hanging out at a trendy bar and programming there. Of course one is a stereo-typical aspie guy with no social skills but is a genius with computers. Then they go to a house party thrown by a big name VC that I think is supposed to be Ron Conway; the house is filled with professional models glomming all over the VC and everyone is generally gorgeous. The guys sneak in to pitch him on their new social networking app that somehow has data "hacked" from some government database or something. They just know that if only they can get 5 minutes of their time with this VC, they will be able to land funding-- which will... make their business successful somehow?

You know, exactly like real startup life.

I think you need to check your privilege here. You grew up in a household with successful parents that wanted their children to succeed. If you didn't get into Berkeley, you'd have your parents be disappointed in you. The main economic driver in your area wasn't something you were interested in, so you'd likely have to move when you grew up.

I would go so far as to say that most people in the US (not even the rest of the world) have childhoods that better resemble a living hell. I suppose most tragedy is relative, and I'm not trying to delve into Candide-style one-upping. I'm just saying that maybe it's not Silicon Valley that sucks, maybe it's just growing up.