Hacker News Reader

Comment by bregma | Hacker News Reader

> why aren't the "more expressive" (which in this post seems to be a synonym for "functional") languages used in these fields?

90% of all programmers are cargo-culters whose knowledge of programming is gained from the answers they cut-and-paste from Stack Exchange or Quora. This is as true of "data scientisis" as it is of "web developers." The result is that there is a network effect making certain answers determinant of the language or monthly flavour of xxxScript that gets pasted most often, and reduces the domain in which further answers are examined.

If you want it in simpler terms: the new dudes looked around and saw all the cool kids were using Python and only a few boffins (UK) or nerds (USA) were using anything else, so they knew who to crib the answers from if they also wanted to look valley cool.

View on HN

If anything, Python in science is the opposite of cargo cult - it's more like pragmatic to a fault.

Its ubiquituous in academia/science, has a low barrier to entry, and a large amount and arguably often unmatched quality of specialized libraries/frameworks/tools for all kinds of scientific fields. So the first go to language/ecosystem will of course almost always be python.

I'm definitely not the biggest fan of Python (the language). And there certainly are more elegant/fitting languages for many problem classes. But the thing is, from my experience many people in the field are quite aware of this. But it simply doesn't matter if it's the best language from a developer POV. It just has to be good enough to get the job done, apart from this the points above are simply more imporant.

And it's hard to understate the importance of ubiquity in this case:

Python is the code equivalent of a lingua franca in science.

I think you're leaving out the most salient fact.

Scientists are smart within their fields, but they're usually terrible at programming. I once saw a hundred-page source file (one file for everything) that defined no functions. And of course it was in python.

So if you're arguing that python is good if you want to exchange source code with people... I guess you might do that? Personally, nobody has asked me for my code in quite a while, so I do all my work in Calysto scheme (which is written in python and can accesss the whole library ecosystem).

If, on the other hand, you're saying scientists' opinions on what constitutes a good programming language ought to be taken seriously, then you're wrong. Scientists are crappy programmers for the same reason they're crappy writers. Programming and writing are tools they use to get their real work done. Most of them master the minimum they need and spend most of their time in the lab

"large numbers of libraries" is almost a subgenre of cargo cult. You're not programming, you're using code you assume does a thing because either it says so or you saw someone else do; but not doing it yourself. Python is paying my mortgage right now and I enjoy using it, but the community around it is no more enlightened with respect to cargo culting than any other I've seen.

Yes. So many developers think, I string together some APIs into something that looks like an app, I've done something. Forgetting those libraries are designed/tested to work only on whatever subset of the problem space they were used for previously. They probably don't handle your case; they don't have explicit capacity promises; they don't work together (don't respect each others backpressure signals, don't understand each others error spaces etc).

I read open source, then extract the bits of IP I think was useful. Often rewriting it to fit my case. Its useful source, but as an example to help me write my app.

{ Please don't use Linux as a strawman (You rewrote Linux?! I don't believe it!) That's the single largest, most successful (along with maybe 5 others) open source project. But absolutely not representative of the 1M other projects out there. }

Seems to me like if you want to look cool nowadays you would use Rust not Python.

Different communities have different needs. Data scientists, or physicists, or biologists don't care about memory safety, the programming language is a tool to run a computation for a limited time (not a server). The tool should not get in the way of fast iteration and rapid prototyping which many languages do in varying amount: C with memory management, Haskell with category theory, Rust with borrow checking, C++ with being C++, ...

Basically you are criticizing data scientists for choosing to not invent a new thing every time and reusing an established ecosystem.

So why aren't they using Clojure?

I agree that people typically want to get from start to solving their problem as quickly as possible, and don't care about things that don't obviously contribute to that. But that doesn't mean they are perfect rational agents, or immune to cargo-culting, or that a little time invested in learning something else wouldn't pay off.

What does Clojure bring?

Here is my data science stack https://github.com/mratsim/arch-Data-Science

Let's go over it:

- Is there a mature tensor/multidimensional array library in Clojure? Is there a Dataframe library?

- Are they compatible with common formats: .npy, .csv, .hdf5 and sqlite databases

- Is there a fast PCA and DBSCAN algorithm?

- Can you pass those arrays to XGBoost or LightGBM for Gradient Boosting based classification or regression.

- Do they support out-of-core processing (i.e. memory-mapped arrays) for processing that takes more than your RAM (happens often with geolocalisation)

- Can you use PyTorch through Clojure? Can you use Tensorflow? What's the GPU computing story

- Does your ML library support things like One-Hot-Encoding, cross-validation? Or more advanced like Target Encoding?

- NLP: stanfordNLP wrapper? spacy? Huggingfaces? Rasa-nlu? sentencepieces?

- Deep learning: Do you support image augmentation like Nvidia DALi or Albumentations? Mixed precision training like Nvidia APEx

- Vision: How is the OpenCV support? Does it interface with Clojure multidimensional arrays? Do you have morphological operations like erosion or dilation?

- Visualization: is there a dataviz library like plotly, ggplot2 or Altair/Vega?

And more importantly

- Are there tutorials and courses in Clojure from beginner to advanced concept? Even Julia is struggling with that.

In other words, you'd consider switching to Java if it was exactly like the system you already use.

And note the question you didn't ask: does Java do anything better than python? Didn't even cross your mind.

He wouldn't because Java gets in your way much more than Python does. A better language needs to offer these things AND be a better language.

Agreed that Java is an obnoxious language. I think originally the grandparent and I were talking about Clojure, which uses the Java library ecosystem, and I started talking about Java libs.

Clojure is certainly the context I'd use Java's machine learning libraries in.

Well then, instead of putting words in his mouth like "exactly the same", go down his list.

> - Is there a mature tensor/multidimensional array library in Clojure? Is there a Dataframe library?

Is there one in Java? If so, what?

And so on with his other items. It doesn't have to be exactly the same (offering him the benefit of the doubt), but it has to offer substantially the same functionality with substantially the same performance. So, you want to use Java libraries from Clojure? Go for it. Where are the things he needs?

Not sure what they cover, but there are surely libraries available,

https://deeplearning4j.org/

https://github.com/originrose/cortex

http://www.onyxplatform.org/

https://neanderthal.uncomplicate.org/

https://clojurecuda.uncomplicate.org/

https://clojurecl.uncomplicate.org/

https://github.com/uncomplicate/bayadera

https://devblogs.nvidia.com/grcuda-a-polyglot-language-bindi...

Also since Pytorch 1.4 there is initial support for Java, built on top of the new pure C++ API.

The ceremony to start using Java is significantly worse than Python or Clojure.

Science is about functions, not objects, a language that requires all functions to be associated with an object is a non-starter.

Given the choice between python with an almost free ffi to c and Fortran libraries for numeric processing with a syntax that's so easy we teach it to children under the age of ten and clojure which is sits in the centre of the universe feeding on all memory it comes across, you wonder why people don't choose clojure?

To backproject the causation of popularity of a given ecosystem and language towards cultural relativism and sociological in-groups rather than pragmatics just seems like a post-hoc rationalization. But, to take your comment in good faith (I don't want to dismiss it) I wonder if there really are generational dynamics at play here.

Earlier on, software engineering wasn't as glamorous as it is today, from either a material or cultural standpoint. I even recall a staggering difference now versus twenty years ago, when I first got into it, which even then was right before the bust of the first dot com bubble. While ostensibly by that point, the field of software engineering as a creative pursuit and an intellectual space had already thoroughly lost its pioneer spirit to the metaphorical urban and suburban "gentrifiers", there's no denying that the field itself continued to grow, and even at that point is a small fraction of what it is today. If software has truly eaten the world, isn't it possible that the languages and ecosystem that survived and reached dominance outcompeted the other ones?

That is to say, what makes you believe that another language is more fit to be in the position where Python is? While I will use whatever language is necessitated by the job (and have never had issues doing that) and even prefer the aesthetics and "handling" of Ruby and modern JS, Python is still the tool I reach for most often when I need to get some general purpose scripting or systems design task done. The ease with which I can go from zero to prototype and prototype to production and production to pain-free maintenance is unparalleled. The only two ecosystems that are remotely close are JavaScript and Java, and ostensibly Rails for web dev. Nothing else is competitive.

What would make you dispute this?

PrincipalSkinner.png:

Is Haskell a bad language?

No, it's the people getting actual work done that are wrong.

Reply on news.ycombinator.com