cmansley

Aren’t many ports completely autonomous?

https://youtu.be/zm_rlLyelQo

nyt is wingnut? https://www.nytimes.com/2018/07/25/style/sperm-count.html

Can you detail some of the rock solid solutions in 2008 for the problems addressed? For example, building multiple C++ packages from source. Google's Bazel wasn't open sourced until 2015, so solutions were still appearing long after 2008.

Why does no one get mad at the governments who have the large tax loop holes? Shouldn't it be "Ireland close your loopholes!"

I feel that many of these legal arguments have not caught up with modern technology. If we had had the capabilities we have today 100 years ago, I am sure the lawyers would have pushed for maximum recording of all things. But face to face discussions are some how grandfathered in.

How is this any different than in-person conversations that are not recorded? Should companies be required to record all conversations that happen face to face?

I am thinking about the end application. All the more reason to ignore all of the benefits of a web solution and spin the highest performance version. Because performance is critical when viewing tons of cameras and lidar with near-realtime playback.

I have a serious question. Should this be a web visualization? Doesn't all the abstraction lead to performance hits? Tons of data needs to be pulled off the hard drive, network, RAM and shoved into the graphics card. Is putting a web browser in the middle of that pipeline the most efficient use of resources? I feel like it is a hammer situation.

Social science isn't broken. Math and physics are like what's p-hacking.

That's not how that works and there are whole websites for figuring out if the plumber is useful. Angie's List, Yelp, etc... Plus plumbers normally have to be licensed by the state.

Instead this process is left to the individual corporations. So, you get mixed results in judging quality, because there is no standardization.

"Vogt couldn’t change lanes because of California state law around autonomous vehicles"

This statement is complete bullshit. The California DMV doesn't have any clear details defining operations of vehicles at this point in time. They only have rules about what is required to test on public roads. This includes 5 million dollar bond, applying for permit, etc...[1]

This statement alone (if correct from the reporter) makes me nervous about the amount of spin going on.

[1] : http://www.dmv.ca.gov/portal/wcm/connect/d48f347b-8815-458e-...

Random comment: Could similar attacks be used to extract the private key for Bitcoin accounts?

Non-competes are illegal in California. And no other place matters right?

Tesla HQ is near Stanford, which has some Audi cars which have sensors on the roof.

I was encouraging you to write down your mathematical evidence of better performing algorithms, because I know from my experience that when I try to write down proofs, my assumptions and reasoning become clearer. And often a step in the proof or logic that seems trivial becomes less trivial once you actually try to lay out the proof.

I believe that one of two things will arise when you go to write up a mathematical proof. One, you will be dependent on a statistical test that makes a normal distribution assumption implicitly or explicitly. Two, you will dependent on a bound that only applies in the limit. There is a third possibility, which is the most interesting to me, which is that you are dependent on a period of stationarity in your data, in other words, the distribution you are measuring must be stationary.

The proofs in the MAB papers are not intentionally obtuse or ignoring mainstream statistical ideas. They are written that way to say very explicit things under well defined assumptions. Locking down assumptions and being very precise is what writing down a mathematical proof is all about.

Ok, you are making a point about sampling. In periods of high traffic you have more samples that will bias the calculation. But, since the number of samples will be consistent between A and B, everything is fine. Fine.

I believe further discussion will not be productive. But, I suggest if you have a proof about A/B testing as it relates to the MAB problem or even if you have a proof about A/B testing in general that drops the normally distributed assumption or tells you exactly when to switch from exploring to exploiting, I suggest you write it up and publish it on arXv.

Thanks for your time.

First, I was using Student t-test as a stand-in for whatever test or statistical measure you would like to use. I believe the popular one is Hoeffding's inequality in the bandit literature, hence the log term in the MAB algorithms. I agree this was a poor choice of example.

Second, I believe I am getting hung up on the fact that different arrival times are "weighted" differently. I think you are claiming that the Poisson assumption gives us equal numbers of A and B trials, so we can combined the statistics (counts) and avoid Simpson's paradox. This is fine, but why would you say "different times of arrival are not weighted the same". Does this mean you are somehow weighting periods of heavy traffic down and weighting low traffic up?

So, what happens when trial A becomes less favorable over time or is less favorable for brief periods? This means that the underlying random variable's mean is changing over time. Most statistical bounds cannot handle this situation.

I am not saying that A/B testing is not something we should do in general. I am saying that it is a good heuristic with very few provable properties compared with logarithmic regret MAB algorithms.

Also, I agree that MAB algorithms do not consider the case of adding bandit "arms" during the trial. A dynamic number of bandit arms problem may not be well studied, but is very interesting. This might be another reason for requiring a reweighing of the samples.

I don't know how we got onto arrival rates of our visitors. I would just like to state that Simpson's paradox is exactly why we shouldn't compare percent conversions. They are meaningless. However, many of the statistical tests like Student's t-test compensate for this paradox by including the number of samples in the tests. See : http://en.wikipedia.org/wiki/Student%27s_t-test#Unequal_samp...

I think you said one thing that are at the heart of the issue. We assume that the E[conversion of B] > E[conversion of A] for the entire period sampled.

I think all of the details about Poisson processes are not required if you just assume that each person is drawn IID from the population.

I just don't think you are answering the right questions here.

Let's assume that each person arrives IID from the infinite population. Then, we have a Bernoulli process for each A or B query. A "conversion" results in a 1, a failure results in a 0. Since, these people are arriving IID, we can select a sub-sample which is also IID. We would now like to estimate the parameters for each process and/or compare the two processes. We can do this using t-test. This will give us the statistical significance that the one group had a higher "conversion rate" than the other group. Note: rate does not factor into this problem at all because we assume the participants were IID, so the t-test (used correctly accounting for different number of samples) will tell us which test is larger.

My question is now what happens when the parameters of your queries for A or B change over time. Still under the assumption that E[B] > E[A], it now matters greatly in which order you use your samples.

I think the only reason you brought the Poisson model into the discussion is to weight the more recent samples higher and down weight the earlier samples in your basket of samples. This is a heuristic for considering a fixed interval in which the samples are stationary. It effectively considers a window that slides with the time of arrival.

Wait.

I am slightly confused and this may demonstrate my ignorance, but I was under the impression that A/B testing worked by allocating two different approach to the users and then scoring the response. This provides a sampling from the population of users as to how effective A or B is. You can then run some statistical test on the averages of the scores for each test to determine which one is the winner.

If what I said is true, these statistical tests almost always assume that the distribution that is being drawn from is stationary. So, the only way things work out is if you have an underlying stationary distribution. Otherwise, your statistical test might indicate the wrong thing.

I freely admit that many of your points are still valid, but I don't see how A/B is a more powerful algorithm that has less assumptions.

I think the real distinction here is stationary verses non-stationary distributions. Many of the arguments made in this article hinge on the fact that the responses to the same input change over time (nights are different then daytime, which is different than the weekends). By continuously running the A/B testing, you are looking at a small window in time, which you assume is stationary so you can do your t-tests or whatever statistical test.

But, to be clear, this is a heuristic in A/B testing. If you have a window of time over which you know your distributions are stationary, you should always use a logarithmic regret MAB algorithm because it is theoretically better.

I think the best way to frame this argument is that because the domain does not match the assumptions of MAB, A/B testing has been shown to be robust and easy to modify for non-stationary domains, while logarithmic regret algorithms are somewhat more fragile.

Does anyone know if the table of contents is somewhere online?

You mean spin-off company? Suitable is developing the Texai robot into a product.

May I ask how this is Bayesian in anyway? I understand that using the term Bayesian is good for directing clicks to a site, but this seems like good old fashion frequentist math. None of the hallmarks of a Bayesian approach to the problem are here: having a distribution over hypothesis, having an explicit prior, computing the posterior of the distributions.

I have some experience with the medical trial literature and specifically bandit algorithms and using cumulative regret verses other statistical measures like PAC frameworks. And regret is most certainly not a Bayesian idea. Instead you are explicitly modeling the cost of each action (providing an A or B test to a user) instead of assuming all costs are equal.

Yes, this is a better approach because it explicitly models the costs associated with the exploration/exploitation dilemma. But, it is not Bayesian.

Just a note. The distribution being created here is the Binomial distribution. It just so happens that because of the central limit theorem, you can approximate it with the Gaussian distribution. For a low number of samples (n<20), this approximation will degrade. http://en.wikipedia.org/wiki/Binomial_distribution#Normal_ap...

Ghrist is a brilliant guy. He does his presentations at conferences in exactly the same style and it is brilliant. He has a framework of notes that he scrolls through, but as he is speaking he will fill in parts of overall picture with sketches. He is one of the few speakers that can make technical topics both interesting and intuitive.

This was exactly my experience. Everything from the two technical interviews with the addition of a third to the technical questions with queries about bugs in the code. I think 2->3 conversion is exactly what you said. The first two interviewers split on their opinion of you, so you get scheduled a tie-breaker interview. Unfortunately for me, the tie-breaker interview was my worst, partly due to me and partly due to non-existant feedback for 45 minutes on the part of the interviewer.

But, I believe there is a method to the madness. They have so many qualified applicants that they would rather err on the side of caution. They would rather reduce the false-positives even if the number of false-negatives grows. They can afford this luxury. They want the best and they can afford rejecting good candidates.

My favorite way of thinking about this was from Steve Yegge's blog post about the interview process : "Because of the inherently flawed nature of the interviewing process, it's highly likely that someone on the loop will be unimpressed with you, even if you are Alan Turing. Especially if you're Alan Turing, in fact, since it means you obviously don't know C++"