Comment by jakobnissen | Hacker News Reader

Comment by jakobnissen | Hacker News Reader

Parent Full thread

I don't think that's so convincing, actually. When I learned programming, Python was _just_ replacing Perl as the "common language" in bioinformatics. Until then, there were lots of people who claimed Perl would essentially never be dethroned because it had become the standard.

The issue is that while _some_ users only experience the surface level of packages like PyTorch, Numpy, SciPy, Pandas, and so on, people who actually have to develop these packages must work with the C++ and C underneath. To the developers, the problem of Python's performance is front and center, and a real issue they face every day.

So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Over time, this will mean the development power switches to Julia - and as soon as these developers have build tools as good or better than existing tools, the majority of users will slowly migrate, too.

> So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C.

Not so fast. You should also consider that writing the core in C/C++ allows some of those packages to offer bindings for several languages other than python (for example, both tensorflow and torch have bindings for Go, .NET, Rust, Haskell, R, Java, JavaScript).

This significantly expands the use-cases that these frameworks can cover without requiring a complete rewrite for every platform. For example, it is relatively easy to have data scientists train models with Python and deploy such models to mobile/edge devices and web apps.

That's true, but for most packages, that's not really an issue. I.e. there might be a tensorflow library, but there is no libnumpy or libpandas, as far as I know.

As I also mention in the blog post, if you set out to create fundamental library software like FFTW, you should indeed use a static language, and Julia is no good.

> there is no libnumpy or libpandas

I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on:

https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...

Arrow is just an intermediary

What is your point? (I honestly do not understand.)

Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments.

Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts.

My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".

In response to "rise of," I too was excited about Arrow until I played with it and realized it didn't even provide a shape attribute. Anyways, people shouldn't be dependent on a low level lang like C to write fast code.

Fair. I agree Arrow is still more of a vision than anything else.

> it didn't even provide a shape attribute

I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame libraries, which are generally 2d. I think they (correctly) are ceding the "n-dimensional tensor computation" space to the current incumbents.

Arrow is getting support for N-d arrays, so if anything they're expanding in that area (which is exciting). I don't think they're interested in creating a universal libarrow though, the point of the data format and C data interface is to have languages define their own implementations.

I may be wrong. It happens a lot! But I think Arrow's vision encompasses compute, not just a data format and data interface.

https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...

Slide 43: The "Arrow C++ Platform" encompasses a "Multi-core Work Scheduler" and a "Query Engine"

Slide 38: "It would be more productive (long-term) to have a reusable computational foundation for data frames"

Again, I agree that, today, it's more data format, and the shared compute stuff is more a vision.

EDIT: See also https://ursalabs.org/tech/

For sure, I didn't mean to imply they weren't looking at compute too! https://github.com/apache/arrow-datafusion is another example of the shared compute vision. What I was trying to point out is that (at least for Arrow core) they seem to eschew FFI and generating shared libraries in favour of from scratch implementations in other compiled languages and direct bindings in interpreted ones.

This is doable in Julia too. DifferentialEquations.jl has python and R bindings. I expect more to follow as Julia starts getting more best in class packages.

Writing Python packages in Rust with PyO3 is pretty awesome, though.

What are your favorite open source Python packages written in Rust?

tokenizers

tokenizers is a great example. Some other, perhaps simpler ones, are orjson and tantivy.

I mostly live in the Rust world, not the Python world, but I worked on two Python packages wrapping over my own Rust libraries and it's been pretty great.

https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-segment

Reply on news.ycombinator.com