Hacker News Reader

Comment by temp8964 | Hacker News Reader

Is there also something about timing?

Given that R and Python are so mature and active, this really leaves not much space for Julia. Yes, Julia is faster than R and Python (just the language, not packages), but R and Python can easily integrate with C/C++ and many R and Python packages are in fact internally C/C++. This really leaves not much advantage for Julia against R and Python.

If Julia had appeared earlier, it may have had better position in the data analytics programing ecosystem, and gain more attention and opportunity to grow. Now, the whole data industry is booming, but people would rather contribute to the matured R and Python for real work, instead of waiting for Julia to grow.

View on HN

I don't think that's so convincing, actually. When I learned programming, Python was _just_ replacing Perl as the "common language" in bioinformatics. Until then, there were lots of people who claimed Perl would essentially never be dethroned because it had become the standard.

The issue is that while _some_ users only experience the surface level of packages like PyTorch, Numpy, SciPy, Pandas, and so on, people who actually have to develop these packages must work with the C++ and C underneath. To the developers, the problem of Python's performance is front and center, and a real issue they face every day.

So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C. Over time, this will mean the development power switches to Julia - and as soon as these developers have build tools as good or better than existing tools, the majority of users will slowly migrate, too.

> So, the developer folks will make the switch first, as they realize it's much nicer to develop Julia packages in Julia than it is to develop Python packages in C.

Not so fast. You should also consider that writing the core in C/C++ allows some of those packages to offer bindings for several languages other than python (for example, both tensorflow and torch have bindings for Go, .NET, Rust, Haskell, R, Java, JavaScript).

This significantly expands the use-cases that these frameworks can cover without requiring a complete rewrite for every platform. For example, it is relatively easy to have data scientists train models with Python and deploy such models to mobile/edge devices and web apps.

That's true, but for most packages, that's not really an issue. I.e. there might be a tensorflow library, but there is no libnumpy or libpandas, as far as I know.

As I also mention in the blog post, if you set out to create fundamental library software like FFTW, you should indeed use a static language, and Julia is no good.

> there is no libnumpy or libpandas

I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on:

https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...

Arrow is just an intermediary

What is your point? (I honestly do not understand.)

Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments.

Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts.

My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".

In response to "rise of," I too was excited about Arrow until I played with it and realized it didn't even provide a shape attribute. Anyways, people shouldn't be dependent on a low level lang like C to write fast code.

Fair. I agree Arrow is still more of a vision than anything else.

> it didn't even provide a shape attribute

I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame libraries, which are generally 2d. I think they (correctly) are ceding the "n-dimensional tensor computation" space to the current incumbents.

Arrow is getting support for N-d arrays, so if anything they're expanding in that area (which is exciting). I don't think they're interested in creating a universal libarrow though, the point of the data format and C data interface is to have languages define their own implementations.

I may be wrong. It happens a lot! But I think Arrow's vision encompasses compute, not just a data format and data interface.

https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...

Slide 43: The "Arrow C++ Platform" encompasses a "Multi-core Work Scheduler" and a "Query Engine"

Slide 38: "It would be more productive (long-term) to have a reusable computational foundation for data frames"

Again, I agree that, today, it's more data format, and the shared compute stuff is more a vision.

EDIT: See also https://ursalabs.org/tech/

For sure, I didn't mean to imply they weren't looking at compute too! https://github.com/apache/arrow-datafusion is another example of the shared compute vision. What I was trying to point out is that (at least for Arrow core) they seem to eschew FFI and generating shared libraries in favour of from scratch implementations in other compiled languages and direct bindings in interpreted ones.

This is doable in Julia too. DifferentialEquations.jl has python and R bindings. I expect more to follow as Julia starts getting more best in class packages.

Writing Python packages in Rust with PyO3 is pretty awesome, though.

What are your favorite open source Python packages written in Rust?

tokenizers

tokenizers is a great example. Some other, perhaps simpler ones, are orjson and tantivy.

I mostly live in the Rust world, not the Python world, but I worked on two Python packages wrapping over my own Rust libraries and it's been pretty great.

https://github.com/InstantDomain/instant-distance https://github.com/InstantDomain/instant-segment

> This really leaves not much advantage for Julia against R and Python.

It's a bit of a chicken and egg problem. Really the argument is the problems with dual-language model will never go away, so Python & R face an ongoing cost disadvantage in the long run. However, they are also much more mature.

So assuming previous assertion is correct, you need to get enough people interested in Julia to grow to the tipping point.

I'm not really sure it will work, but can see the argument. After all technically superior platforms for this existed when python got started (heck, early python was next to useless for data work beyond file parsing) but that didn't dissuade it's use. The non-technical arguments made it a good-enough language and once that had momentum it's compelling.

On the third hand, what julia really needs to attract right now is mostly library/package writers - that's a tiny fraction of the users of python, but basically steers "what's next". However, for library writers, how many people might use it can be a deciding factor on investing the time...

> for library writers, how many people might use it can be a deciding factor on investing the time

Yes. Especially given that a high percentage of library development these days is being done by industry, not just academia. (At least in machine learning.) They're not writing packages for fun or for academic citations, they want users.

Julia does appear to have a higher than usual proportion of package authors to total users, FWIW. I think it's both a blessing and a curse, because feedback around gaps in the ecosystem is sometimes brushed away with "write your own package". That is, because a large part of the community writes packages to scratch their own itch (I've yet to see one written for citations), they expect others to do the same (which may not be fair).

I think Julia has a pretty good path forward since it is often faster than C++ (especially once you consider language interop overhead), and it is much more pleasant to write than c++. Since Julia and most of the packages are written in Julia (for the most part), Julia makes it really easy for users to become developers. Python and R have much bigger barriers to entry since users will usually hit a binary blob, which makes it much harder to interact with library implimentations.

I dunno. I have written FFIs in many languages and by far julia was the easiest to call a c abi in.

I would say if anything what holds people back in julia is a drive to RIIJ instead of calling the c abi (despite that julia's linalg basically does exactly that with BLAS).

Lua is pretty smooth in this respect.

Also the slowness of Python is overstated - for many of the niches Julia is targeting, Python is plenty fast if you use NumPy / CuPy / SciPi.

Yes, it's easier to write the equivalent for Julia using native Julia code, but it's not clear to me that the average user will care.

> Python is plenty fast if you use NumPy / CuPy / SciPi.

I think this misses the point. You are basically saying that python is fast enough in cases that someone has written already written a fast package in not-python (plus bindings), whereas the value proposition for julia is basically: writing such packages in the future will be lots better.

Obviously the average user doesn't care, but if enough whiz-bang stuff starts to come out in Julia but not python, they'll move.

I think it will be interesting if enough clever new whiz-bang stuff comes out particularly based on Julia's metaprogramming powers.

However, when talking about the "two language problem", the main issue traditionally was prototyping computational code in Python / Matlab / whatever and then having to drop into Fortran / C / Cuda for optimizing it. That's pretty much a solved problem in Python now - thanks to the well developed ecosystem, you don't really have to leave Python to write fast numerical code.

The first problems I came to Julia to solve were ODE simulations in which it's just as critical that the stuff I write and rewrite over and over again (the actual differential equations of my model and associated simulated controller callbacks) runs fast as it is that the pre-written stuff of the library (the solvers or integrators) runs fast. To the best of my knowledge I can't do that in python.

This example (DifferentialEquations.jl in particular and SciML.ai in general) is one of the places where Julia leads the pack.

When I first used Julia it felt rather strange in a lot of ways (I had used python for about 20 years), but after 2 years when I go back to python (and of course there are places where it makes more sense to use python) it feels primitive as there are a lot of very nice features of Julia that I suddenly miss.

> That's pretty much a solved problem in Python now - thanks to the well developed ecosystem,

I don't understand this argument. The same C and Fortran libraries that make numeric Python fast can be called just as easily from any other programming language, just like in Python.

> you don't really have to leave Python to write fast numerical code.

Well, you need to, for example, if you want to write a new numeric algorithm to call from python. As in, not use a numerical solver, but write a numerical solver from first principles. For that, Python is mostly useless and you'd need to use C or Fortran (or Julia).

It should go without saying that Julia also has had a C FFI for years. And a Python FFI. And an R FFI.

The C++ one was dodgy last I looked (which was several years ago; I'm sure it's better now). In practice I typically wrote explicit C wrappers, restoring genericity on either side of the wrapper. Faintly tedious, but nothing a couple macros couldn't solve.

Yes, Julia has them, but if you use them, it basically renders one of the promised advantages of Julia ("it does away with the two-language problem") moot, as you now have two languages anyway. If you have to languages anyway, why not use one of the more established languages?

----

I personally don't subscribe to the idea that the "two-language problem" is a problem that needs fixing. In the end your stack will have multiple languages anyway, unless you rewrite everything is Julia, which is utopian, so having one more isn't really a problem (and it might also be one that you are already using).

> Yes, Julia is faster than R and Python (just the language, not packages), but R and Python can easily integrate with C/C++ and many R and Python packages are in fact internally C/C++.

This is what I was responding to. The grandparent argued that Julia had less library support than R and Python because, so they implied, those languages could interop with C/C++ in a way Julia can't.

The two language problem refers to code in user space, anyway. It's not like Rust, where there are so many wheel-reinventing projects in progress. If you peruse the main Julia package registry, and you'll see plenty of library-wrapping modules.

Reply on news.ycombinator.com