johnmyleswhite

Can you link to a proof of the equation starting at "The standard error of the sample Median approximates to"?

Mypy types don't usually have any runtime enforcement.

Is this an Ask HN thread? If so, why is the title a statement rather than a question?

Not sure if Spain is the best example since AC is already very popular there and growing in popularity: http://www.ine.es/jaxi/Datos.htm?path=/t25/p500/2008/p01/l0/...

But the Financial District was much emptier at night historically than it is now, so I'm not sure this is closely linked to the article's claims.

Small nit: if the curve is continuous, should the y-axis be labeled density rather than probability? Closely linked: is anything being done to ensure the curve integrates to 1? I imagine the answer is "the scale changes as you draw", but that seems a bit confusing since drawing doesn't usually change the structure of the surface being drawn on.

Perhaps I've misunderstood your desire to be sarcastic, but I think your timeline is off. To give two concrete examples, Geman and Geman's "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images" paper is from 1984 and Holland's "Statistics and Causal Inference" paper is from 1986.

All of Statistics by Larry Wasserman is the book I recommend for anyone who knows calculus and linear algebra and wants to learn statistics.

Prop 13 passed when California was a red state: http://www.latimes.com/projects/la-pol-ca-california-voting-...

As a person who knows both of the people involved in that debacle, I think it's fair to say that there's a lasting interpersonal conflict between two specific people (one of whom is the author of the post you've linked to) that's been framed as a community-wide issue.

The easiest way to think about Julia's performance is closely related to the observations that inspire tracing JIT's for many languages -- most code in dynamic languages doesn't make use of the features that make efficient compilation impossible. Julia's response to that observation was to build a dynamic language that lacked some of the most extreme features in Python or R that act as barriers to efficient compilation.

Christopher Chedeau's original post clearly doesn't represent Facebook's official stance on functional programming (nor does it ever purport to do so from what I can see), but I'm not sure that describing him as "a junior front end engineer" is appropriate either.

I'm not a big fan of the percentile bootstrap, but that reference is a little too cavalier with material that deserves to be treated more rigorously. Chapter 8 of Wasserman's "All of Statistics" is much more careful about outlining the conditions under which the percentile bootstrap will work. Moreover, he works through a specific example that demonstrates that the percentile bootstrap does not generate results that are profoundly different from other methods.

Thanks! I really should have finished and written the post about the SVD as well. One of these days...

Glad you enjoyed that post so much. It really is a shame that we do such a bad job of teaching students about the inherent subjectivity of descriptive statistics and let students leave their courses with dangerous ideas about the existence of a Holy Grail statistic that will solve all of their problems.

Note that the first sentence of the linked page says that:

> This is a draft textbook on data analysis methods, intended for a one-semester course for advance undergraduate students who have already taken classes in probability, mathematical statistics, and linear regression.

This argument would seem to assume that a frequentist is required to use maximum likelihood estimation for all problems.

If you remove that assumption, the distinction between Bayesian and frequentist methods becomes murkier. If you ultimately want a point estimate, you don't particularly care whether a method constructs a posterior distribution as an intermediate step.

This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Julia, like any language, has intrinsic limitations, but this is not a good example: it's an example of how good API design for Julia differs from good API design for other systems.

I think most people who study causal inference would say that Pearl's writing is seldom as clear as a person unfamiliar with his ideas would like it to be. These days, I suspect most people would benefit from starting with Morgan and Winship instead: https://www.amazon.com/Counterfactuals-Causal-Inference-Prin...

In addition to presenting Pearl's ideas more clearly than Pearl himself tends to do, Morgan and Winship also describe Rubin's work more fairly than Pearl does. (On the other hand, Rubin also tends to disparage Pearl's work more than is necessary.)

I'd like to see the title changed to something like "Futures for R" since it's so tongue-in-cheek that it's effectively clickbait.

Really glad to see this, but I think we really need to assess how much money is available from each of these channels. My experience with some of these channels (e.g. bounties) is that the effective hourly wage is radically lower than offered by any industry position.

From the article:

""" Democrats always politic on the basis of emotion and their quixotic Green Energy is one of their favorite feel-good causes. There is no way to prove, or disprove that it exists or that it is caused by humans. Scientists are paid to find evidence, so they find evidence. If they do not, their services would no longer be necessary. If there was incontrovertible proof, an evidentiary talking-point would have been constructed, which would silence all non-believers. What goes unmentioned is the fact that Global Warming has morphed into Climate Change and the media took no notice. The inconvenient truth is that the globe has not become warmer for the last 18-years. There was a slight warming trend for 20-years, but that ended in 1998. The left will claim that these two trends are proof positive that Climate Change exists. Thinking people call it weather. """

They're both great books, but they make heavier use of potential outcome notation and focus less on graph-theoretic formulations than Pearl's work.

Yes.

I think it would be really interesting to understand exactly why Julia underperforms LuaJIT on some benchmarks. I'm pretty skeptical that Julia loses to LuaJIT uniformly; I suspect it depends on the benchmark taking advantage of LuaJIT's superior discovery of run-time type information.

I understand your concerns better now. Here's my input for what it's worth:

(1) List comprehensions are a mess and a recognized problem with the existing compiler. There's a lot of work that's been done to fix the erratic typing of comprehensions. As such, I am sure that this concern will be resolved before 1.0. I would bet that it will be resolved much sooner than that.

(2) I don't really believe that scientists can't learn that Array(Int, 1) and Array(Float64, 1) do different things. This seems like a place where education can be very effective.

(3) Julia's error messages are indeed generally poor. I'm hopeful that we'll see a push towards Elm-quality error messages in the future: http://elm-lang.org/blog/compilers-as-assistants

(4) The core problem with your fixed size array example is that you're building it on top of the wrong primitives. Julia's Array isn't a suitable data structure for building up fixed-length arrays: Julia's Array is more like C++ std::vector than like C's arrays. To build a real fixed length array, you need to build it on top of lower-level primitives. Many people are already working on that problem.

Putting it all together, I think your concerns all reflect real problems with the language. You may be better off waiting until Julia 1.0 comes out.

I would use SGD when I can tolerate an inexact estimator. For problems when I need an exact MLE (because, for example, I need to invoke asympotic normality to get CI's), I can usually do exact computations using less memory than IRLS would require.

In particular, what I had in mind is that most black-box optimization tools (e.g. L-BFGS) only need an implementation of a function and its gradient -- both of which can be accumulated iteratively using O(P) memory and O(N * P) time per iteration for any data store that can stream all N observations. Given code that can accumulate function values and gradients, an optimization method like L-BFGS can estimate parameters for many kinds of models. In my experience, there's often no need to materialize the kinds of dense matrices that are typically used in R's default approach to GLM model fitting.

That really depends on how you fit models. Maximum likelihood estimation (or MAP estimation) for models of IID database rows is often trivial to implement using an iterative algorithm that applies to both in-memory and out-of-core data stores.

I don't see how this could be done without putting information about the internal representation of DataFrames into the compiler. Such an approach would seem to require breaking important abstraction barriers and would couple the language to a specific back-end for data representation that might need to be replaced in the future to handle things like out-of-core data frames.

It would be very helpful if you provided some specific examples of code in both Julia and Fortran that is under-performant in Julia. As it stands, your concerns about things like "fancy abstractions that no engineer or scientist will ever understand" are rather too vague to be effectively resolved.