Hacker News Reader

Comment by beforeolives | Hacker News Reader

So instead of dropping down into C, it has Python as a depedency and wraps sklearn? Way to solve the two-language problem.

View on HN

Absa-bloody-lutely it's a way of solving the two language problem, because it provides a pathway from the current two-language setup (Python with C for the computationally intense portions) to a single language (Julia everywhere, because it can reach C speeds). In the short term, library writers can provide Julia wrappers around legacy code from other languages, but that needn't affect the library users.

I think you can only make the snarky remark because you don't truly get the 2-language problem. The problem has zero to do with the complexity of dealing with dependencies, as if the hard problem of scientific computation languages is like a Linux distribution with too much bloat. No, no, no. It's the cognitive load to solve scientific problems, when at the end of the day performance really matters. We need a language that is as terse as mathematics where we can customize all the way down to a single array entry, without performance loss. Python itself will never have the necessary performance. To do truly new numerical stuff you cannot avoid the computational kernel details (e.g., PDEs or image processing). So you're forced, as a researcher, to become good enough to be effective at two languages, and you often have a complex data exchange interface between the two, and it sits right near the critical conceptual problems that you're trying to solve.

That's a genuinely mean comment towards a new library which is using existing code to bootstrap itself. Julia does solve the two language issue, but not in all situations. Like I'll never write a driver in julia. But this mean spiritedness towards anything that's not perfect is disheartening.

>Julia does solve the two language issue, but not in all situations

And not instantaneously. Developer time being finite, using what works from existing libraries while waiting for a pure-julia implementation seems like an excellent solution.

Julia has always had the ability to call Python code. For example you can use PyPlot/matplotlib as a plotting backend for Julia.

There is no way to bootstrap a scientific language without leveraging existing ecosystems. But this is a starting point. Whenever someone find a sklearn model to slow, or misses the automatic parallelism that Julia brings, they can just implement the model in Julia, and bit by bit the non-Julia bits get swapped out where it brings a real benefit.

You hardly need python for models at this point. There are native julia implementations of linear regression, naive bayes, gradient boosting, random forest, PCA, T-SNE and more.

The python interop is just a stopgap

The ability to call other languages is a strength. The need to do so is a weakness.

I disagree; consistency is the most important part of this. With Python, you know where you stand - performance comes from elsewhere. With Julia it might be internal, a C/Fortran library, or apparently other things now too.

I beg your pardon? Knowing that Python is slow is a strength of Python? Then, if Python suddenly became fast, it would be a negative?

Did I misunderstand something?

Presumably, all fast languages are at a disadvantage, then.

I never called it a strength; I simply don't care how fast Python is because I don't need to. Reasoning about what is going on under the hood with Python is just easier than Julia - if it needs to be fast, it's a fast external library being used.

This is really a minor issue stemming from the relative maturity of the languages - if Julia becomes more established I would hope usage of external libraries which don't offer a performance advantage (ie everything besides C and Fortran) eventually gets replaced with native packages to preserve the sanity of the users.

So then that 'weakness' in python is not an issue to you personally, because you fluently drop down to a fast language anytime you need to, with no particular loss in productivity?

Keep in mind though, that this could be a hurdle to those who are less multilingual. So even though it's not a weakness to everyone, it is to many.

One problem relying on fast libraries causes is it makes doing compiler tricks like automatic differentiation (AD) basically impossible. Also, it restricts the types of APIs that make sense. A simple example of this is to compare Scikit learn to Julia. In Scikit learn, most clustering methods don't allow the user to specify a distance function because doing so would require running python inside a tight C loop, tanking performance. In MLJ, on the other hand, basically anything that requires a distance function will allow you to pass one in rather than assuming euclidean distance. This is possible, because a distance function written in Julia can still be fast, so it can be used without slowing down the whole program for people who only want euclidean distances.

On the other side, we observe more and more inter-languages's ecosystem crossing from/to Python to leverage its gigantic set of libraries. Julia interface with a lot of other languages but it is a big trend for the last years.

I think the two-language problem begins to be as we move forward a network of multi-languages calling each other instead of either having to rewrite everything from scratch either relying on a C/C++/Fortran/Rust lib and the C ABI to patch them. We may finish with a set of "meta"-langurust in C code nameages permitting the interaction of a large number of libraries and programming languages outside of their silo.

Moreover Julia coupled with Pycall could "corrode" à la Rust for a more smooth migration to pure Julia.

Maybe this comment was too sarcastic but the core point stands. Julia’s main advantages for most ML and Data Science users are still theoretical. The package ecosystem is still way to immature. It’s not just MLJ. I was excited about Julia and made a good faith effort to port a work project to Flux and Zygkte and it was a disaster. There’s nothing wrong with passionate open source tinkerers working to improve the ecosystem, but don’t advertise it as ready for production when it’s not.

When was this and what were the issues?

The two-language problem is not concerned with calling out to or wrapping pre-existing libraries in a second language. It means having to write parts of your code in a second language, due to, for example, performance reasons.

I dont like Python, but the language have dominant libraries in many domains

Julia hides Python, Julia good

I understand that using Python feels wrong, because Python is not know for its performance, but still it seems like a Practical decision, rebuilding everything from scratch will be too much effort

Reply on news.ycombinator.com