Why I Still Use Python for High Performance Scientific Computing | Hacker News Reader

Why I Still Use Python for High Performance Scientific Computing

nbviewer.jupyter.org

273 pointssubnaught151 comments

Summary in the conclusion:

"The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]

All the points the author makes through the post are interesting, and Python is definitely great for protoyping, but I think the initial premise is false: "people don't tend to think of [Python] as a high performance language; for that you would want a compiled language -- ideally C or C++ but Java would do."

Java is compiled to bytecode, but it isn't a "compiled language" since that bytecode has to be interpreted by the JVM. All the good libraries the author mention are probably implemented in C or FORTRAN, and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code.

Java(well, Sun's Hotspot) is most definitely compiled, but mostly by the JIT at runtime. The interpreter only runs for the the first warmup, usually 10,000 method calls and the JIT kicks in and translates the method to machine code. The tricky part is it's not simple to control exactly how and when it does its job, although it usually works very well.

The major implementations of the JVM are jited, not interpreted. Yes, you get great performance with jitting, tho not necessarily without unpredictable GC pauses and without a loss of energy efficiency from having to run a jit alongside your actual code.

GC pauses have little to nothing to do JIT process itself. You can get JVMs w/ read barriers and soft-realtime guarantees.[0][1] Please note, gc and jit are quite a different thing, even if the gc needs help from the jit.

[0]: https://www-01.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/... [1]: https://www.azul.com/products/zing/

"...and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code."

True. But this assumes that time is not a constraint.

I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X hours, you stop both and see how far they've gotten.

I think the argument then is that you might find that the C programmer hasn't yet solved the problem, whereas the Python programmer might have a solution + a deeper understanding of where the slow parts are, and started optimizing those.

So in the end the C programmer may win, but perhaps it's not about winning in this sense but more about how fast you can get to a point where you can move on to the next thing.

Absolutely. For this kind of one-shot scientific computing, the only way the C programmer can win is if the Python programmer is sitting on their hands for weeks waiting for their program to run.

Unless (1) the relevant library happens to be written in C and there are no python bindings and the problem is simple OR (2) there is an existing solution which is 95% complete in C and one needs to write from scratch in python. I've never come across situation (1) with a new project. Situation (2) is quite common.

But in those cases, the Python programmer could just wrap the C library in Python, which I suspect is often a lot easier than starting from scratch.

I've tried to do a test like this and the results surprised me. While the testing conditions weren't perfect, I tracked how much time it took to port Fortran code to C++, and how much time it took to write the C++/CUDA optimized version. I would have expected writing all the optimized CUDA kernels would add significant time to the project, but in reality it was something like 1.3x for a solution 15x faster.

I think as developers we often underestimate the amount of work we need to put into the non coding part of development for larger projects, like testing, debugging, optimization, and design. When these components become important then despite the local productivity gains of using rapid development languages like python or Julia, it would overall be a wash if you're writing the system from scratch in a language like C.

On the contrary, I pick Python to make the "testing, debugging, optimization, and design" faster.

Actually, even the subsection headings in bold give a very succinct summary:

- Python has easy development (https://xkcd.com/353/)

- Great libraries (ie, free matlab)

- Cython for efficiency via C

- The algorithms themselves determine speediness (ie numerical methods)

The algorithms themselves determine speediness

This is so important I wish people would focus more on it. I recently rewrote some Javascript code in (pure) python and got a good 2 orders of magnitude speed up on large inputs just by picking the right data structures and replacing an O(n^3) nested loop with an O(n log n) approach.

Which data structures did you use? In Python, I tend to rely on dict, list, and set for 90% or more of my code. I wouldn't want to rely on structures written in pure Python.

Nothing exotic. One of the changes for example was replacing a list of list with a set of tuples, which greatly sped up checking if an object was in the collection. Another change was using a generator comprehension and an included itertool function rather than hand rolled nested for loops.

Once you've exhausted all the low-hanging fruit, like people calling .keys() on dicts, or doing unnecessary linear searches, Cython really starts to shine. I've seen it perform ~40 times better than pure Python in time-consuming loops.

We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.

Can you explain the context in which .keys() is called often and is not appropriate and the alternative?

.keys() returns a list (in python2) so if you write

  for k in dict.keys():
    ...

then python first builds a list of all the keys, loops through them and then throws away the list. If the dict is large, this can be quite expensive. The correct way is to either use .iterkeys() which returns an iterator which generates the keys one at a time, or simply iterate directly over the dict, saving you need to first copy all the keys into a list you'll just throw away.

This has been 'fixed' in python3 and .keys() now returns an iterable view of the keys, and if you actually want a list of the keys you have to explicit and write list(dict.keys())

The easiest is to never use .keys() or .iterkeys(), and always iterate over the bare dict:

    for k in dict:
        ...

    if k in dict:
        ...

If you do need a list of keys, list(dict) has the advantage of working in both Python 2 and 3.

Just to clarify your python 2 note: In python 3 keys() returns an iterator, so there is no penalty (i.e. iterkeys was dropped, keys assumed iterkeys interface).

The equivalent python2 behaviour can be obtained using list(somedict.keys())

> Numpy does 90% of the work

Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is

"Could this be solved with matrix multiplications and summing?"

Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy.

I recently swapped out something that was running at about 100 rows calculated/second to about half a million in about 0.2s.

In fairness, I should point out that the really slow version was also written by me :)

http://www.vetta.org/2008/05/scipy-the-embarrassing-way-to-c... ;)

Fantastic!

Have you written a blog post about this? If not, care to? Trying to speed up a codebase, it's of interest to me (and no doubt others too).

Yes, this is actually one of the rare moments where I find my past background in competitive programming helpful. There's a habit I've picked up of constantly running through techniques to speed up whenever I'm coding. Not necessarily the most efficient way to code, but it's a habit difficult to change.

Just typed "import antigravity" in python repl. It took me to the same link. :-)

Let me just add Numba to the list. It compiles numpy-using code to native via llvm and can remove temporary arrays in the process.

I tried a very simple toy program the other day and while I had to write some things slightly un-pythonically (it can't deal with syntax like a[:] = b+c yet), it performed practically as good as hand-written C code.

If your code falls within the subset it supports :) I've not yet got it to run our code in nopython mode - I think the latest problem is a function expecting a function as an argument, but the error messages aren't helpful or enlightening, even with DEBUG turned on.

The dev team is pretty responsive and prioritize adding features based on bug reports like this. Can you open an issue?

https://github.com/numba/numba

I agree with his conclusion that Python has great tools to help performance, but the GIL makes it impossible to effectively make use of separate cores, so I have to wonder just how badly the Java version was written. Properly written high-performance Java (store all data in contiguous arrays, do away with all the OO crap, reuse objects like crazy) should perform as fast, or faster than optimized Python, and you can scale it to as many threads as you like.

Python has MPI4PY and Multiprocessing. Not sure why you need threads in the first place. Also, many matrix operations autoscale to all cores when compiled against Atlas.

Finally, you can release the GIL with Cython: http://docs.cython.org/src/userguide/parallelism.html

But you can have both. In Scala I can write prototypes just as rapidly as Python, but I can run them with close-to-native performance. I can even explore interactively in a REPL but backed by the power of my company's big computer cluster, using spark-shell. The profiling capabilities are excellent, but when I spot a bottleneck I can solve it in the language directly, without needing the awkwardness of cython or of converting to/from numpy formats.

(And while I personally love Scala, there's nothing magic about it in this regard. There's no reason a language can't offer Python-like expressiveness and Java-like performance, and many modern languages do)

Except for the fact that JNI is such a piece of utter... garbage ... as far as performance is concerned. One has to think twice, thrice ...countless times before jumping back and forth over the runtime bridge of JVM and native. Not that Python is that good at it either, but better than JNI, almost everything is. The best I have seen is Lua's FFI.

I consider it an accident of history that Numpy, Scipy, Pandas, Scikits got written for Python and not Lua. Thanks to Luajit I think Lua would have been a better choice. Now that cause has been taken up by Torch and Julia.

JVM by itself is terrible for reaching close to the FLOPS that the CPU is capable of. Try sparse matrix multiply with it and see it for yourself.

I think Swift's seamless bridging is pretty damn good as well.

A bridge is always easier to build if there is no river in between.

Could you elaborate a bit on how JVM hinders reaching maximum FLOPS when multiplying sparse matrices?

I could think some examples where lacking SSE/AVX support would hinder it, but I don't see the connection with sparse matrices.

What make sparse matrices harder to JIT is proving that the loop bounds will not be exceeded, so all accesses re bound checked. In any case in my experience array bounds checking and escape analysis never gave the boost that theory and JVM fans promise. So even normal matrix multiply will trail behind. That said Hotspot JVM is possibly one of the most optimized VMs we have got.

A structural problem of JVM is that its runtime semantics is over-specified, there is very little room for the JIT to do its stuff. For example function arguments are evaluated right to left, there goes an opportunity for parallelism.

For normal dense matrix laid out as a double[] and accessed directly as i* N_ROW + j probably won't get its loop check elided. For double[][] I would _think_ that happens more easily.

But how are sparse matrices then generally laid out? A naive approach would be some hash map, perhaps with some locality, in which I don't see JIT problems.

There are some defacto standard formats such as CRS, CSC, list of tuples etc. Layout of the third should be obvious and it is not used much for cases where speed matters because one loses locality in this layout. For the other two they are laid out column after column (or row by row), row ids, and offsets to indicate the start and end of columns (rows).

Thanks, this helped. Using CSC and CRS is probably quite problematic with JVM bounds check elimination (or the lack of it). So if they would need to be used on JVM, I think it would be wise to drop the safety and use _sun.misc.Unsafe_ for unchecked array access.

Correct me if I am wrong but Spark does not use JNI for Python. It uses py4j to allow Python programs to access Java Objects. All the core functionality is implemented in Scala/Java. py4j uses sockets to communicate. See https://www.py4j.org/about.html

That's my understanding as well. This is the reason why you are caught between the rock and a hard place on the Hadoop stack. Java code wont be efficient in terms of FLOPs and a popular and often effective escape hatch: code number crunching parts in C, C++ or Fortran, is also not very effective / convenient. Particularly so if it requires going back and forth over the bridge frequently because crossing the bridge has significant overhead. So typically you have to move majority of the core into the high performant language of choice and what remains is glue. If gluing is what I want to do, there are other languages that can give Java stiff competition. The core capability thing going for Java in this domain is HDFS, and its not that great a file system for big data.

When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+ you lose by implementing numeric code in pure Python. So Scala gives you a 10x faster cycle during early development. Maybe it's fast enough that you don't need that last factor of 2, or maybe just JNIing a few core operations is enough (you can also do some tricks with the advanced type system; with NumPy if it looks like Python iteration then it is, whereas with something like Breeze you can potentially have something that looks like Scala iteration but won't actually copy the data back and forth). But in the cases where you do need to push things right down into Fortran, Scala is just as good a glue language as Python is.

> When we say JVM code is not efficient in terms of FLOPs we're talking about a factor of 2 or so though, not really the same as the factor of 20+

Not so in my experience. Write a matrix multiply in pure Java, and pure C, C++, Fortran. The difference would easily be north of 5X, typically more.

Regarding JNI, please see the root of the comment tree. Rarely have I seen more stinky garbage. Its hopeless if you have to go back and forth across the bridge. If that be so I might as well be on the other side of the bridge.

JVM is fantastic, but if you are doing number crunching and performance matters, then JVM is a wrong choice.

> But you can have both.

It seems you misread the article as saying Python created a trade-off. Perhaps the author shouldn't have tried to create a surprise ending. Regardless, the conclusion was that the Python implementation was both easier to code and more performant than the Java implementation.

From the article: "Easy development comes at a cost of course. That initial experimental implementation was terribly slow, taking thirty seconds or more to cluster only a few thousand points." Even if the final implementation in mostly-Python is fast, having a slow feedback cycle during early development can be a huge cost.

Large-scale data processing jobs normally arrange themselves into data acquistion/cleaning, grunt numerical work and result formatting/display. These tasks have very different requirements so a combination of a tool that can do all the data handling easily (ie Python) + a tool that can throw the CPU at a numerical problem (ie C) will work as a great combination.

In contrast, if you work in Java, you are trying to use the same tool for both jobs, and you may well fall between 2 stools. And I say that as a typical Java-head.

My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?

As for the number crunching, I think Rust would be a better choice here. Good memory management is its USP and that can have significant performance benefits.

> My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?

Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fast and break things" mentality. The Node ecosystem is built on that mentality, and while outlier developers at the front line of the JS community are able to be extraordinarily productive in JS/Node, developers building user-facing programs just get bogged down in breaking changes in dependencies, buggy and poorly-designed 0.x libraries, untraceable framework code that prioritizes configuration over convention, and intractable errors caused by a broken type system (this final issue is significantly improved in ES6).

Source: Worked in Python 2.x for a few years, worked in Node for a few years, currently work in Python 3.x and Node (general design is to limit Node to compiling and testing the browser-based part of our product). JS/Node is maybe 20% of our code, 20% of our features, 50% of our dev time, and 80% of our bugs.

Yes you're right, Node wouldn't be good.

Then I remembered Perl and realised that data pre-/post-processing is precisely why it was invented in the first place.

Perl + Rust would be an interesting combo IMHO.

Agreed. It is often true that different parts of a problem are best solved in different languages. That's one reason we built Beaker and released it as open source: http://BeakerNotebook.com. Its polyglot architecture and autotranslation makes working with multiple languages easy.

Julia could be a contender once a few bugs are worked out.

This may be a liiiitle bit off-topic, but I really need to get it off my chest: Python for high-performance scientific computer works beautifully... it's a dream. Scipy/numpy, matplotlib, pandas, ipython. They're all unbelievably awesome. It all just works.

Except, when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible.

So, use Python, but use it only on Linux.

(Okay, if you absolutely must do it in Windows: Use Anaconda).

I was going to mention anaconda but you already said that.

I guess, if you don't know what to do with your money, you can get the similar results on any real operating system (e.g., Mac OS X).

Interestingly though, Windows version of Numpy on Anaconda is quite a lot faster than what you'll get with Ubuntu, like 2x faster easily. I think it has to do with the fact that AVX instructions are used on the windows linalg routines that it binds to. If you buy MKL of course you get much more speed again.

Christian Gohlke distributes free NumPy binaries built against MKL.

FUD. I've been using Python on windows for years and between Anaconda and Christoph Gohlke's python packages and I've yet to run into something that didn't just work.

Yeah with Anaconda it's not as bad. But no-one told me that's what I needed if I wanted all the science packages to work on Windows... I'd never even heard of Anaconda before this. It took all of my blood and tears for weeks before I got everything fixed. So I guess what's wrong here is the lack of documentation.

NumPy/SciPy's install page tells you to use Anaconda (or equivalent) for Windows, and build from repositories for Linux.

http://www.scipy.org/install.html

Granted, it does not say "and by the way, when you do that you get tons of other great things like pandas, matplotlib, etc". But they are very clear about not trying to do this yourself.

This is a problem for many languages that have external libs requiring local compilation (C, C++, etc.), given many of them are principally developed on Linux, with Windows compatibility being an afterthought. Its a problem not unique to Python.

It's similar on Linux. Every time I'm trying to get some outside-written Python software to work, there are many, many troubles.

Python works fine, in your own virtualenv. Outside of it... not so much.

> once I had a decent algorithm, I could turn to Cython to tighten up the bottlenecks and make it fast.

What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io, it's felt a step backwards.

I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree and KCachegrind. I've found the cProfile output confusing when it crosses the Python-C barrier for numpy, sklearn etc.

I've optimized quite a bit of code with Cython. I do something like:

python -m cProfile module.py | gprof2dot -f pstats | python -m xdot -

cProfile takes a while to learn how to use well.

What didn't you like about line_profiler?

Here's a good guide on how to write fast(ish) code in Python: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run into issues is when implementing algorithms in which case I usually isolate the slow function and turn it into a Cython module. Recently have been playing around with https://github.com/jboy/nim-pymod which seems like a much better solution.

> cProfile takes a while to learn how to use well.

I bet :) I have only scratched the surface, that's for sure.

> What didn't you like about line_profiler?

In this case I was trying to profile the overall simulation codebase to find the slow spots (rather than guess) - a simulation that takes 60 minutes to run. line_profiler wasn't great at giving digestible results from that - and I haven't worked out how to write all output (across multiple modules) to file, without specifying the file in each decorator.

I've started to break the codebase down into 'tests' to measure each algorithm separately, and will give line_profiler another go then.

Re memory_profiler, while the mprof command showed all peaks, the line by line output only showed the result after executing each line - so when 4GB of RAM disappeared in a skimage call, only to be released at the end - it wasn't reflected in the output. Which is tricy when trying to reduce overall memory usage.

After using the profiler to look at the right piece of code, an important part of the process when optimising with Cython is:

cython -a module.pyx && open module.html

the -a command produces an annotated html file, which lets you see how Cython is compiling your Python into C. There's a colour coding to give you an indication of which bits are fully typed and being translated directly, and which parts are hitting the CPython API at run-time (and hence will be slow).

Why isn't Haskell, or any other functional language, popular for this sort of thing? Turning A into B is what FP excels at, and you shouldn't have to reason about side effects, besides writing the graph images somewhere.

From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable type system would be a huge improvement.

The only functional langauges to gain much popularity here are OCaml and F#.

Haskell tends not to be a go-to choice here because it... well... there are a lot of reasons. It's a tough language to learn compared to its competitors because it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them.

> it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them.

Err... can you qualify this? Based on my knowledge of Haskell this isn't a fair assessment at all.

Based on my knowledge, it's very fair. How many language extensions are available vs placed in common use? Where is the guidance on which to use? You WILL use at least 3 I can think of off hand very commonly.

And how many failed lens libraries preceeded the current (quite good) dominant library that still show up on google and even get included in? How many people still run versions of Yesod using now-unfavored abstractions?

The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs. But these developments do not magically erase people who have navigated the prior 5 years of Haskell where a lot of these technologies got established.

I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.

Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging." You will know these people when you find them. Compared to the Clojure community or the F# community or the Erlang community which are also forward thinking, interesting and skilled... it looks positively hostile. You may not care about that, but I refuse to endorse or use technologies who's community leadership is so toxic. Why, I've stopped using Linux in anything but legacy apps because I completely despair of that project from ever rising above is puerile and cliquish roots.

Alright, I'll give you that there are a ton of GHC extensions after getting a list of them, all 106 as of GHC 7.10[0].

This does make me wonder why having so many extensions was never an issue for me.

> The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs

Agreed.

> I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.

Fair enough.

> Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging."

I've seen this personality type in other languages, but not Haskell yet. I too abhor the "negging" type, as well as the subculture that term arose from.

0: all ghc extensions

    OverlappingInstances	
    UndecidableInstances	
    IncoherentInstances	
    DoRec	
    RecursiveDo	
    ParallelListComp	
    MultiParamTypeClasses	
    MonomorphismRestriction	
    FunctionalDependencies	
    Rank2Types	
    RankNTypes	
    PolymorphicComponents	
    ExistentialQuantification	
    ScopedTypeVariables	
    PatternSignatures	
    ImplicitParams	
    FlexibleContexts	
    FlexibleInstances	
    EmptyDataDecls	
    CPP	
    KindSignatures	
    BangPatterns	
    TypeSynonymInstances	
    TemplateHaskell	
    ForeignFunctionInterface	
    Arrows	
    Generics	
    ImplicitPrelude	
    NamedFieldPuns	
    PatternGuards	
    GeneralizedNewtypeDeriving	
    ExtensibleRecords	
    RestrictedTypeSynonyms	
    HereDocuments	
    MagicHash	
    TypeFamilies	
    StandaloneDeriving	
    UnicodeSyntax	
    UnliftedFFITypes	
    InterruptibleFFI	
    CApiFFI	
    LiberalTypeSynonyms	
    TypeOperators	
    RecordWildCards	
    RecordPuns	 (Deprecated, use NamedFieldPuns instead)
    DisambiguateRecordFields	
    TraditionalRecordSyntax	
    OverloadedStrings	
    GADTs	
    GADTSyntax	
    MonoPatBinds	
    RelaxedPolyRec	
    ExtendedDefaultRules	
    UnboxedTuples	
    DeriveDataTypeable	
    DeriveGeneric	
    DefaultSignatures	
    InstanceSigs	
    ConstrainedClassMethods	
    PackageImports	
    ImpredicativeTypes	(deprecated)
    NewQualifiedOperators (deprecated)
    PostfixOperators	
    QuasiQuotes	
    TransformListComp	
    MonadComprehensions	
    ViewPatterns	
    XmlSyntax	
    RegularPatterns	
    TupleSections	
    GHCForeignImportPrim	
    NPlusKPatterns	
    DoAndIfThenElse	
    MultiWayIf	
    LambdaCase	
    RebindableSyntax	
    ExplicitForAll	
    DatatypeContexts	
    MonoLocalBinds	
    DeriveFunctor	
    DeriveTraversable	
    DeriveFoldable	
    NondecreasingIndentation	
    SafeImports	
    Safe	
    Trustworthy	
    Unsafe	
    ConstraintKinds	
    PolyKinds	
    DataKinds	
    ParallelArrays	
    RoleAnnotations	
    OverloadedLists	
    EmptyCase	
    AutoDeriveTypeable	
    NegativeLiterals	
    BinaryLiterals	
    NumDecimals	
    NullaryTypeClasses	
    ExplicitNamespaces	
    AllowAmbiguousTypes	
    JavaScriptFFI	
    PatternSynonyms	
    PartialTypeSignatures	
    NamedWildCards	
    DeriveAnyClass

Probably because functional languages don't actually at excel scientific computing relative to Python, C#, Java, and JavaScript. :D

Edit: I kid, but most of the languages I just mentioned have very fast native compilers, easy ways of invoking low-level interop, functional-style libraries if you want them, and (except for Python) C-like syntax making it easy to cut-and-paste.

No need to kid. Self-described "FP" languages are a pain in the ass for the type of programming scientists do. I don't even know what kind of programming those languages aren't a pain in the ass for.

"I don't even know what kind of programming those languages aren't a pain in the ass for."

If we limit the discussion to ML languages, they are generally very nice e.g. for datastructure transforms (including interpreters and compilers). Static typing with type inference makes implementations short, and, if written in good style, are correct when they compile.

There are various other reasons why one would not want to use them. For me it's platform stability and availability - while I enjoy functional languages in principle - I yet seem to choose C++ more often than not for my hobby projects.

I would say learning self-described fp-languages have made me a better programmer who values static typing more than before.

"I would say learning self-described fp-languages have made me a better programmer who values static typing more than before."

FP paradigm has indeed didactic value and the experience gained in a functional language may translate into a plus of productivity ...in other languages. When you have to get things done, the functional language's embedded arm-twisting (well, mind-twisting actually) doesn't help! That's when you'll either abandon the (idea of) "pure" functional language for a multi-paradigm language (that will allow you to keep a mainly functional style with limited deviations when the situation warrants it) or you'll keep pushing yourself and throw the blame on you for not being good enough with a language that wasn't designed from very beginning for much anything but the FP standpoint (for the sake of it).

I could make the same assertion about "self-described OOP languages" or "self-described procedural languages" with just as much evidence. Once you've done it enough, all programming feels like a pain in the ass.

I see FP languages being able to excel at the kind of projects with large, maintainable codebases that evolve over a decade and are shared between dozens of developers. Service oriented architectures serve the same goal -- isolating behavior and state locally, and being explicit about mutation and state changes over time.

Using Scala, I've found the ability to pass around first class functions, and the default immutability of variables, enables me to make profound algorithmic optimizations I wouldn't undertake in C++ et al. Then, if the computation generates too much garbage or goes slowly in a loop, I can then take shortcuts, like using mutable local variables, to make that section go faster.

Actually, if you squint a lot, R (or S) is a functional language. And it shows, crazily powerful for something, absolutely horrible for something else.

No squinting needed :)

"R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functions, and even return them as the result of a function."

http://adv-r.had.co.nz/Functional-programming.html

Yeah, lots of squinting needed, or a terribly shallow understanding of FP to say that. Merely providing first-class functions and map/reduce idioms does not a functional language make. Python has those, so is it also FP really? R is as much a FP language as Haskell or OCaml are imperative "at heart."

Sorry to STRONGLY disagree with you 100% but R clearly is a functional language and it might just be that you are unfamiliar with R or have not spent time with R.

Weak example is Wikipedia list of Functional Programming Languages list R as one. https://en.wikipedia.org/wiki/Functional_programming#R

R describes itself as a Functional programming Language:

The language syntax has a superficial similarity with C, but the semantics are of the FPL (functional programming language) variety with stronger affinities with Lisp and APL. In particular, it allows “computing on the language”, which in turn makes it possible to write functions that take expressions as input, something that is often useful for statistical modeling and graphics. https://cran.r-project.org/doc/manuals/r-release/R-lang.html

Academic Evidence: "The Journal Trends in Functional Programming" also list R as a functional programming language.

R is an environment and functional programming language for statistical data analysis and visualization. Largely unknown to the functional programming community, it is popular and influential in many empirical sciences.

http://link.springer.com/chapter/10.1007%2F978-3-642-40447-4...

The issue might be it isn't Haskel so it is not 100% functional but there are more Functional Programming Languages then Haskel.

PS This is why so many programs in R are poorly written because people try to force OO into R when they could do things Functional. That is also why my eyes bleed when I see people looping through everything in their R scripts.

Like many modern languages, R supports multiple programming paradigms, including sequential, object-oriented, and functional, but it should take more than a couple of functional features to declare something a full-blown FPL. A duck-billed platypus is not a duck.

Yes, you can write R code in a functional style in places when convenient, but one floor down it's still running that loop you didn't want to see. It's a language based on mutable values and sequential execution and looping, not pure function calls and recursion. The syntax itself is very much sequential, unlike stereotypical FPLs. Even the lexical scoping design relies fundamentally on mutable environments. So yes, people do seem to enjoy proudly declaring R a FPL citing a few features, but as I said, those features make R as much a FPL as Python.

> "terribly shallow understanding of FP to say that"

That was very condescending and I am sure don't know who Hadley Wickham is? I wouldn't call his understanding "Shallow."

No, I meant that he was squinting. What is condescending is you making assumptions about whom I don't know and what I'm unfamiliar with.

The large number of high quality scientific and computational libraries that are easily available for Python (most things are a "conda install" or "pip install" away) plus the Jupyter (literate programming) notebook (although that now supports other languages) currently make a very compelling case for using Python instead of Haskell and most other functional languages.

Because the Python ecosystem is huge, with real scientists writing real libraries to get stuff done.

The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).

And Lenses!

C'mon zippers are not that hard, and really useful.

Lets say you want to do processing of some xml file. Normally you'd walk the tree and do manipulations in place. With zippers however, you can inspect every intermediary tree result, you can rethink you problem so that you walk the tree once to extract interesting information, then compute a changeset for the tree, maybe merge it with a differently computed changeset, and then apply the union of them.

I build a ocr system in clojure on zippers and it was a lot of fun. You could for example extract a list of all the words, with line wraps removed, then do the correction on that view/lense of your data, and reapply the changes without having to worry to reintroduce the pesky linewraps, because they were never removed from the original document.

Haskell tends to attract people who like programming languages as programming languages. It encourages a style that's completely different from what you're taught in your basic Fortran/C/Java/Matlab course. If, like a lot of scientist, you've done a little imperative programming already, Python isn't much of a step; Haskell, at least as it is presented usually, is another bag entirely.

This means you have a much larger audience, and therefore it's more likely that someone will already have written a tool of your domain in Python.

> The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).

There are "Real World" Haskell resources, in fact there is even a book named Real World Haskell[0], though I recommend Haskell Book[1] these days since it takes an approach of teaching from first principles.

0: http://book.realworldhaskell.org/

1: http://haskellbook.com/

The diff between Haskell and Python is not functional programming, I program functionally in Python, it is the type system.

This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just executing subexpressions and seeing results and the shape of the tensors interactively is more useful than thinking really hard how the operations transform the shape.

If you're using persistent datastructures like pyrsistent I could agree, but vanilla python is very unsuited for functional programming. Even JS has better lambdas.

> This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just executing subexpressions and seeing results and the shape of the tensors interactively is more useful than thinking really hard how the operations transform the shape.

If it is true that the type system, perhaps static type systems in general, make you think hard about things that ultimately don't matter much... why not go for the most dynamic and interactive language possible? Clojure perhaps.

I've been toying with the idea that either fully dynamic and as interactive as possible or full dependent typing is the generally right answer.

By right, I mean what you should reach for with most programming tasks if you aren't going to put effort into selecting one tailored to your task.

Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures, which is what you need to efficiently implement many numerical algorithms. But I'm a fan of Haskell's [REPA](https://hackage.haskell.org/package/repa) which provides multidimensional arrays with complete control over which operations are computed immediately and which are deferred and can be combined later with other operations. This allows many computations to be expressed in a high-level array-oriented way without incurring the cost of creating huge temporaries. C++ template libraries like Eigen can do this, but give much less control to the user on what is deferred.

> Regarding Haskell specifically, Haskell makes it cumbersome to write code that mutates data structures

Can you provide some Haskell examples that are cumbersome, then provide an example that isn't in another language of your choice?

Also, have you tried using lens for this?

Compare the implementation of a simple numerical program in Haskell

https://github.com/jmoy/avb-econ-hs/blob/f6d65af480c99045573...

to that in Julia

https://github.com/jesusfv/Comparison-Programming-Languages-...

Despite the Haskell program using the elegant facilities provided by REPA, the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.

I agree that the Julia looks nicer, despite the fact I usually dislike so much nesting.

> the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.

It has to be written differently more because it's using the state monad. You can write monadic actions that resemble pure code, for instance:

    > replicateM_ (readFile "/etc/issue")
    ["Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n"]

> Why isn't Haskell, or any other functional language, popular for this sort of thing?

Because some of the scientists doing this are bellow average programmers. Look at the author singing praises to Python while benefiting from C, Fortran and Cython. He doesn't understand that what actually matters is what's under the hood, and you want him to learn functional programming and rewrite his algorithms?

That seems very unfair. I have no reason to think the author doesn't understand that Python libraries are built on carefully written C and Fortran, and based on the article they seem to be a perfectly competent programmer.

Have you missed the title? He gives undue credit to one of the most unfit languages for HPC ever. Python is not made to be efficiently implemented, yet the author cheerfully writes "I choose to use Python for performance critical scientific computing code".

That's the unfair part right there: lying to people about the proper uses of Python.

Not enough developer uptake for Haskell early on; quite frankly because it requires a thoughtshift that many developers find too difficult to do.

I have in my hands a pretty interesting BI project for a big company. So far, the proposal on the table has been .NET and SQL Server, but I am wondering if I should at least try to give python a chance. Pandas is a great library, with great people working on it. Django the same. On the other hand, .NET has lots of professional (aka: with paid licenses) libraries that seem more fit for an enterprise project. Looking from a company perspective, the drawback python has is, strangely, the lack of paid for alternatives. It's not that people in companies don't trust open source (hadoop is becoming big here too), but one wonders if the developers will be able to find the support they need in case any issue arise from a free library.

Python is fine but it can be more work...

I have first-hand experience with a BI-ish system, squarely targeted at the enterprise and doing quite well there, that we wrote using Django and a whole list of open source components.

We did run into some resistance initially, because our stack is almost the opposite in every way of what our enterprise colleagues are used to. However, our development velocity, especially around analytical features and just in general, has made a most gratifying impact.

(I have to add that the 5-man dev team we have working on this is stellar. It's hard to determine scientifically what the interaction is between team quality and the choice of a Python-oriented software stack. See Paul Graham's essays for more discussion on that point.)

In terms of support: There are many highly professional often boutique software agencies that can support Django systems, if you're not around. To my mind this is even better than the normal commercial support you get from a different vendor for each different component in your commercial enterprise system.

It would be interesting to hear about your experience, did you do a write up somewhere or could I email you with few questions?

My project would be to put different data sources together + to allow users to upload their own structured data via Excel (think financial estimates). The current system has about 450 users, the next might have much more depending if it gets extended to other divisions.

You could send an email to the work address on my personal website (see HN profile). I can't promise that I'll be able to answer everything. :)

one wonders if the developers will be able to find the support they need in case any issue arise from a free library.

Continuum Analytics is founded by the creator of Numpy and employs many leading python developers, and they offer support contracts for basically the entire Python/Numpy data analysis stack.

How about F#? It feels like Python and has a Pandas equivalent in Deedle (https://bluemountaincapital.github.io/Deedle/), but fits well in the .NET ecosystem, including using SqlServer effortlessly. This report is a good starting point: http://fslab.org/report/

I would like to second the suggestion for F#. It has a nice balance in that it straddles the line between being an OSS ecosystem and an enterprise ecosystem with paid libraries. FsLab and Deedle got mentioned already but the F# website has a lot of great resources too: http://fsharp.org/guides/data-science/index.html

maybe Play framework, which supports both Java and Scala, is also a good fit here!

A very beginner Java programmer here. It's a nicely organized notebook, great demo, but: seems like a lot of effort was put into optimizing the python efforts, and none for Java. Isn't that an unfair comparison?

My real question is, is it so much easier to do this excercise in Python than Java, assuming equal proficiency in either case?

Java has no operator overloading. Many Java developers vehemently oppose addition of operator overloading into the language, as if it were the root of all evil. The lack of that feature results in convoluted function calls where a clear and concise math expression would suffice. Consequently, not many people choose Java for math tasks, and not many people write math libraries for Java.

That is just my guess.

> it became necessary to see how our python version compared to the high performance reference implementation in Java.

It sounds like the Java version was also optimized - though it's hard to say to what degree. I think in general if you're doing a lot of numerical computing and matrix operations, the basic java builtins are not going to cut it anyway, and you're going to end up using something like ND4j or JBLAS to get comparable performance to something like numpy, in which case I have hard time imagining it could possibly be /easier/ than numpy.

Granted, the cython stuff is more "advanced", and you probably would struggle a bit (or just would not care for the task) if you were a researcher with a surface understanding of what's really going on, but it's no big deal for a programmer. The other thing is that the defaults usually work well enough that you don't really have to pull out the big guns most of the time. You can also decide to do your optimization progressively. "Let me rewrite this part in cython" is a much quicker win than "let me rewrite all of this to run on top of platform X" or something.

Also it definitely does depend on the kind of problem. For example, the JVM does have awesome tools for distributed computing and streams like http://akka.io/, vs python is more lacking.

In conclusion though, I've been always very pleasantly surprised by how far I can get away with by just dumping larger and larger datasets (current record is in the hundreds of GB for me) into the python / pandas / scipy / numpy stack, and how much of a pleasure it is to use compared to anything else. To toss that away, I'd want to have tried and failed at a problem with the python stack first.

Agreed. Python is an excellent tool in that respect. Batteries included helps. Being able to access fast C routines help. Compile to C projects like Numba and Cython also help. And of course, ipython (Jupyter) notebooks for exploration.

Jake Vanderplas previously wrote an excellent blog post about Python performance and scientific computing: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow...

The interesting insight from the article is that python might be a good language for learning algorithms. The fast development time allows you to write a complete program (albeit clunky) without the pre-optimizing you might be tempted to do in other languages.

My current setup for "scientific computing" is RStudio and CSV files for quickly running typical stats-tests (a couple of t-tests and a tost + krippendorff's alpha the last couple of month) and python+libraries for anything that resembles "building stuff" (mostly scikit-learn to build some classifiers). I mostly use R "as a consumer" i.e. I basically use RStudio whenever my colleagues fire up SPSS. That combination works fairly well. I'd recommend it to anyone who enters academia in any field that involves statistics who doesn't want to use the typical proprietary tools (I've also tried PSPP and it works ok for basic tasks but lacks a lot of functionality. If all you want to do is run a quick t-test or ANOVA it's a decent tool).

Good article. Couldn't find who wrote it since it doesn't have a byline. I'm guessing it was Leland McInnes?

I think you're correct, the source .ipynb has only one contributor:

https://github.com/lmcinnes/hdbscan/commits/master/notebooks...

Couple of questions:

- Could this be/Was this developed in Python 3.x

- what is this "notebook" he keeps on referring to?

The "notebook" he refers to is the web page itself. It's a format that's becoming popular among Python / data science folks: a web-based, often interactive layout with code and results embedded in it. See "Jupyter Notebook", the viewer for which is what you're viewing as you read this.

Absolutely and it is even easier if you use Ceemple for your IDE

I use Spark. Will using Python help a lot ?

If Numpy, Pandas, etc. were wrappable from JavaScript this could have easily been titled "Why I use Node.js for High Performance Scientific Computing".

The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!".

I agree some of the tooling and niceties are beyond a doubt best in breed with Python, but it's disingenuous to equate this to "writing HPC code in Python". If you had written a RPython to Verilog translator that produced an FPGA of your algorithm would you call that "using Python"?

Numpy can basically be considered to be part of the standard library for Python so I'd say using it still qualifies as "using Python". Pandas is newer but is graduating to that level of ubiquity too.

Completely bizarre attitude (creator of pandas here).

Not disagreeing with you, but would you mind elaborating on that?

Claiming that using wrapped libraries written in another programming language (LAPACK, anyone?) is an inauthentic usage of the language (here, Python) is pedantic and unhelpful. C is just a wrapper for assembly, then, right?

Sometimes it's not - sometimes LAPACK is written in C. Sometimes your C is just a wrapper around FORTRAN and people should take this fact seriously: modern FORTRAN is often a better option than C for numerical workloads.

I never stated what language LAPACK was written in.

Then what was "C is just a wrapper for assembly, then, right?" supposed to mean?

Drawing a parallel!

What's the parallel? Python wrapping a C library is very different from C compiling down to assembly (which is often not even how C is compiled).

Why is it bizarre?

His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.

By bizarre I mean impractical and unhelpful. What's the point of programming at all if we cannot leverage abstractions to make ourselves more productive?

I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.

Well, the word "bizarre" has a commonly understood meaning, but you somehow decided to use it to mean something completely different. That's a bit bizarre :P

But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC.

In reality, the reason why he "still" uses Python is that the libraries are fast enough for HPSC. But that's not what people see when reading posts like this.

The message they see is that "Python is fast", not that some of its 3rd-party libraries are fast.

But I should probably stop repeating myself here.

First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter.

Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easier and safer than doing it by hand (e.g., generating correct reference counting code). All of this is a direct benefit of Python and its scientific ecosystem.

> First, there's no reason to only associate "Python" with "the language", it is an environment, ecosystem, etc. It's not interesting to narrowly focus on the efficiency of the interpreter.

Sure, but that's how people construe the post, which I think the author knows too.

The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC", but that wouldn't have generated nearly as much interest as "Why I still use Python for HPSC".

People want to see others say good things about their favourite language, which in this case is Python.

I knew where the article was going (with the libraries), but still wanted to read it because even I wanted to see someone compliment Python itself, because it used to be my "primary language".

Python is a good language, but to the extent you're not using those highly optimized "C-libraries", it just doesn't perform well, and it's not suited for concurrency. That's a part of why I switched to Clojure.

Obivously clojure, being written in Java (at least the intersting parts - collections and multithreading) - isn't fast (if Python isn't fast).

Right, but I wouldn't claim it is. I'd say something like "The JVM is fast" :)

Just because people construe the post that way doesn't mean they're right. Truth is not democratic.

You can make any title more accurate by making it longer and more specific, but picking a good title is a trade off against other things as well. A charitable reader would not necessarily insist on the narrow interpretation of Python the language you insist on.

And your title is not more accurate, because it's not just the existence of certain libraries, it's also the way they can be easily glued together and incrementally improved. Compare this to say Java, where you'd have to use JNI to use those same libraries, which is much more cumbersome to the point that most people prefer to write everything in Java.

> People want to see others say good things about their favourite language

To be honest I only think this only describes people that enjoy internet debates on programming languages. If what you are interested in is getting work done, you don't care about the merits of a particular tool in isolation but about end results. Hypothetically, you could find the "perfect" language but if it doesn't have users or libraries, it's of no use, and therefore the insistence of discussing only this or that language is pointless.

The question is: what language should I use for HPC.

An answer is Python.

If you use C for HPC (the way you mean it) it will be slow as a dog. Because HPC is done with BLAS, LAPACK, etc. If you do HPC in C you are calling into BLAS and LAPACK. They are Fortran libraries. Anything else is dog slow in comparison (a slight stretch, but not by much).

So, the "real" answer is Fortran. But not a lot of people are sitting at their desk writing Fortran for scientific computing these days. They are writing Python, C, or C++ for the most part. All of which use libraries that call into the Fortran libraries.

That's all this article is saying, and we all understand it. No one is wringing their hands about what language the libraries are written in unless you are trying to write one of those libraries.

Working professionals use Python for scientific software because it excels at data munging, making reports, and very high speed computation. Everyone gets that you need to add some libraries beyond the libraries provided from the base install.

The point I think a lot of people are missing is that we've always used optimized libraries no matter what language we've used. C devs would use hand optimized assembly when speed was critical. There's no perfect language that just does everything.

To be fair, the article doesn't say "Python is fast." It says "Python is slow," and then goes on to say something like "But if most of your heavy-duty calculation involves calling prewritten C routines, you can do your development work in Python without getting killed on performance."

Language discussions always seem to go down the same dismal tube in the end: "Is C > Python, or is Python > C?"

Thank you for pandas.

> If Numpy, Pandas, etc. were wrappable from JavaScript

But they aren't, of course.

For science, the fact that so many great libraries are available and they all use Numpy arrays as base datatype (from image processing to GIS to machine learning etc etc) is a real strength of Python and I doubt another lanuage is going to achieve that any time soon.

Python in science is more than mere wrapping: it gives you new level(s) of abstractions, unrivalled in any other generic programming language AFAIK. If it were mere wrapping, other languages would have been able to catch up fairly quickly after all.

Big parts of numpy (and to a lesser degree pandas) are not in python, but lots of libraries on top of it are mostly python (scikit learn, scikits image, etc.). It is no more wrapping than lisp is wrapping the C VM.

I'm currently working on bringing BLAS [1] and LAPACK [2] to JavaScript. And here is a matrix/vector library with some of the BLAS functions implemented [3].

[1] https://github.com/mateogianolio/nblas

[2] https://github.com/mateogianolio/nlapack

[3] https://github.com/mateogianolio/vectorious/tree/opt-cblas

You clearly do not understand software.

Shameless plug...

If you want to import, export, tabulate, edit, plot and analyze CSV data in browser-side Javascript, here's something I wrote to paste numericJS, jqPlot, lzstring, and various html5 storage and file handling features together.

https://github.com/DrPaulBrewer/html5csv

Reply on news.ycombinator.com