Hacker News Reader

Comment by WhompingWindows | Hacker News Reader

Where is the discussion of R? You talk of scientific computation and don't speak of R? That's an oversight, given the majority of the scientists I know have used R. There's also STATA, which economists love, which can do somethings in my workflow much quicker than R. There is also a huge contingent of analysts that uses SAS, especially in healthcare and finance.

The car analogies in the blogpost are not particularly useful... Why do people feel the need to dumb down a topic with off-the-wall analogies? Talking about Julia like it's Tesla is laughable. Tesla is a huge innovator, Julia is another tool that does similar things to the other tools. The apt analogy for Julia would be a new ICE company, not a new EV company.

View on HN

"Julia is a huge innovation; a Tesla is another car that does similar things to the other cars."

Yes, but the majority of computational scientists do not use statistics - something that may be hard to believe.

As an example, most physics programs don't even have an introductory statistics class in their curriculum.

Some engineering disciplines use it more, and it is often relied a lot in industry. But most engineering research makes little to no use of it.

"If your experiment needs statistics, you ought to have done a better experiment." Lord Rutherford (maybe)

Except there's a whole field of statistics called Design of Experiments, which describes the mathematical basis for how to create a good experiment :-)

In physics you often have the luxury, not always available in other disciplines, of having a good theory :-)

Oh, I know (I have a PhD in physics)--but I'd say the main reason statistics isn't used in most fields of physics isn't because it wouldn't help, but rather because most physicists don't know much about statistics.

And publication-wise, it's a bit of a chicken and the egg problem: since most reviewers are also not well-versed in statistics, they are less likely to accept a paper that uses statistical methods to confirm or reject a hypothesis. Thus, there's no institutional pressure for a physicist to learn statistics.

(And a graduate course in statistical mechanics does not count, saying that as someone who used to use that excuse)

>And publication-wise, it's a bit of a chicken and the egg problem: since most reviewers are also not well-versed in statistics, they are less likely to accept a paper that uses statistical methods to confirm or reject a hypothesis. Thus, there's no institutional pressure for a physicist to learn statistics.

Not to mention that if experimentalists actually used statistics, they would have to describe their experiment in more detail when publishing for their statistical analysis to make sense. And in my experience, they really prefer to list as little about their experimental setup as they can get away with.

>And a graduate course in statistical mechanics does not count, saying that as someone who used to use that excuse

Heh heh. Most physicists I know would invoke "But we use probability in quantum mechanics all the time!"

> Most physicists I know would invoke "But we use probability in quantum mechanics all the time!"

I have absolutely heard that one before :-)

I agree with you but IMO, this could be a good thing sometimes. Avoiding statistics to some extent requires more deterministic experiments, where statistics are limited to quantum uncertainty or gaussian distribution. Otherwise there is probably some unknown mechanism to be revealed, and that’s exactly what the community expects physicists to do.

There's no such thing as a deterministic experiment. There are so many sources of uncertainty in all fields of experimental physics, and statistics can help quantify and even design better experiments. It's just a shame statistics has never become a part of the standard physicist's education.

I agree that statistics can help a lot in experiments and should be a part of standard education. My point here is, when physicists encounter uncertainty, they are encouraged more by the current culture to design a better experiment with deterministic theory to reduce the error, compared to analysing the results statistically. It would be better to have both, but it's not as bad as you might think for focusing more on the former one, at least for some fields I'm familiar with.

I think a variant from A Random Walk in Science was "The experiment failed, so we had to use statistics", which I appreciate up to a point. However, I'd hope that physicists calculate error estimates in some sense, and goodness of fit. You hear the statistical significance of results from CERN, for instance.

There is a lot more to scientific computation than statistics, and the only thing R is good at is statistics.

It's quite popular in bioinformatics as well: https://www.bioconductor.org/

Unfortunately the quality of the Bioconductor packages varies wildly, from pretty good to downright abysmal.

Some can't even work: take a look at this one[1], which is part of the current release of BioC: it will never work because the hardcoded host there is no longer functioning and the author wrote that for their master's thesis and have since moved on.

In addition, there's no centralized bug reporting, and some packages use GH issues while for others you may need to email the author and hope for the best (sometimes works, sometimes doesn't).

I use BioC regularly every day because I have no other alternatives, but its main advantage is the sheer numbers of packages available for almost every bioinformatic niche, rather than the quality of the packages themselves.

[1] https://github.com/AllenTiTaiWang/anamiR/blob/7a7a133c553f8c...

That's not a true statement. R has packages for a wide variety of fields--statistics is just the most well developed.

And an important aspect of scientific computation is data visualization, an area in which R is decidedly more advanced than other languages.

One thing I can think of is that R is not well suited for computationally expensive work --- by "computationally expensive" I mean something that takes months to run on a cluster. Not only is R 100--1000x slower than Fortran, but it also does not have strong support for parallelism or GPU computing. I have yet to see a computational fluid dynamicist using R. Though Python is also not much better in that aspect, it can be used as a glue language to interact with native C/C++/Fortran code under the hood.

R has interfaces for Fortran, C, and C++ as well. Quite good ones, in fact.

R is actually the basis of one of the CORAL2 benchmark sets, and there are published scaling results to 5000 cores as far as I remember. I've not done it, but presumably you can substitute its BLAS with a GPU version, the same as with an OpenMP-parallelized one to get performance for anything linear algebra-ish.

[HPC clusters on which you'd run CFD typically don't let you run for more than days, let alone months. There's probably no reason you couldn't use R the same way as Python for such things.]

You are soo wrong. R for earth scientists is a god send thing. Not even Python has as many packages/libraries for earth science as R

Reply on news.ycombinator.com