Hacker News Reader

Comment by cercatrova | Hacker News Reader

Unfortunately, Julia has a number of correctness flaws [0]. Just based on this alone, I can't use Julia simply because I can never be sure whether my code is wrong or the compiler itself is wrong. In scientific computing and machine learning, these problems are very important, unlike in other types of programs where it's more tolerable, because they deal with vectors and tensors with potentially billions of parameters and computation/training time might take several days. If I get an incorrect result, my time has just been wasted, not to mention money via compute resources.

[0] https://news.ycombinator.com/item?id=31396861

View on HN

Can you point to a compiler without correctness flaws[1]? If you use Fortran, or C++ your compiler will also have correctness flaws. Is the claim that bugs are more prevalent or fixed more slowly than with other compilers? Yuri kind of implied that in the article you link, but we would have to take his word for it.

1: http://compcert.inria.fr/ is the only non-toy one I'm aware of to be formally proven.

Also cakeml.

The claim, as I understood it, is not that other compilers provably have no bugs, but that maintainers (and the whole community) react differently when one is found.

If you demonstrate a bug in gcc or rustc (or one of their base libraries), it is a major issue that will at least get documentation and a warning (and likely a quick fix). In Julia, even documenting it may be relegated to the middle of the todo list.

I think that's a reasonable take for a usecase like ML. In most code, >99% of your bugs will be the one you write yourself, and debugging the few bugs from Julia itself is really not much harder than debugging your own, in my experience.

Of course it's annoying that Julia itself is this buggy, but the problem is sometimes framed as if it causes some fundamental uncertainty and doubt about your program because the compiler can't possibly be debugged, whereas is my experience, when you see a Julia bug, it's just another bug, usually in some bog-standard stdlib function which you can easily inspect using code introspection.

And again, I've maybe seen 20 Julia bugs in 4 years of coding it daily, compared to probably 20,000 of my own bugs.

Besides your points, i've found Julia to have deceptive marketing about speed. I've found it to be super slow, mainly when writing expressive code. So then it boiled down to writing ugly code which turned out to be a little less slow. I don't buy "it's compiling at run time" argument, since other (interpreted) languages do not have this problem. Also there were a lot of inconsistencies mostly when trying to use (or abuse, who know?) broadcasting. Did it improve somehow since last year? I hope so.

That definitely sounds like cause for concern! You seem to have spent some time on this, can you share an example?

It is quite likely that you have fallen afoul of some of the standard performance 'gotchas', like non-const globals or inadvertently creating type instabilities. Have you consulted the Performance tips?: https://docs.julialang.org/en/v1/manual/performance-tips/

Efficient Julia code should normally look simple and elegant, not 'ugly' (unless you are going into deep optimizations, like manual simd, or some times heavy reliance on in-place operations.)

I think calling it 'deceptive' is problematic. Of course you can write slow code in Julia, like in any language, but did you write code that you think should be fast, but wasn't?

> I don't buy "it's compiling at run time" argument, since other (interpreted) languages do not have this problem.

I'm a bit confused by that statement. Compiled languages have to compile, interpreted don't. What did you mean here?

Almost all of those 'correctness flaws' are bugs in packages that weren't interfacing with each-other correctly. Everything has bugs and correctness problems, but it may be that Julia users run into them more often because they compose packages in novel ways very often.

But we also have very rigorous testing infrastructure and a community that cares deeply about these things and moves very fast to fix them. You'll notice that every legitimate issue in that post is now closed and fixed, and the remaining ones are about the users misusing in place functions in the presence of aliasing.

> Almost all of those 'correctness flaws' are bugs in packages that weren't interfacing with each-other correctly. Everything has bugs and correctness problems, but it may be that Julia users run into them more often because they compose packages in novel ways very often.

I agree with this. Yuri's post requires a bit of Julia experience to understand the context of it, that much of it is about drop-in combinations of packages and types that one wouldn't even attempt in many other languages. Julia allows you to do that and makes it programmatically easy, which has significant benefits. But it also requires work on the implementers' side to get right.

> But we also have very rigorous testing infrastructure and a community that cares deeply about these things and moves very fast to fix them.

Not sure how much I agree with this. If you're talking about the language itself, sure. But a lot of the libraries don't have extensive testing infrastructures, especially not the kind Julia obviously needs based on the above, with beyond-the-expected-usecase tests. Maintanence of packages varies a lot, with the median being on the slower side. There's definitely "a community that cares deeply about these things", we just need more of it.

I think it's fair to point out that Julia makes it really hard to write robust, correct code. There are a lot of different interacting reasons for this (it deserves a blogpost on its own), but here are a few:

* There are no interfaces

* Existing abstract types are mostly undocumented: It's unknowable and certainly untestable what constitutes an IO or a Number or even an AbstractArray (yes, even AbstractArray leaves important edgecases unspecified). I've written 100 methods that takes `::IO`, yet I don't actually know what it can take. Much of the issues with unsupported package interactions come down to this: One package doesn't know what it promises, and the other one doesn't know whether the promise is upheld. E.g. it's still unclear to me whether `OffsetArrays` actually fulfill the contract of an AbstractArray. If not, it's a bug that it's an AbstractArray. If so, Base is insufficiently tested, as it ought to test its AbstractArray code with an AbstractArray with offset axes.

* Base Julia have several functions that are simply not tested. CodeCov of Base is far from 100%

* Iterators are assumed by Base and Iterators to be immutable - a buggy assumption in many contexts

* It's not even clear what is public and private in a package. E.g. are fields of exported struct private? Where is that documented? And it's way too easy to rely on unexported symbols.

* Speaking of which, you can export stuff that does not exist.

* Projects does not have compat entries by default

* Generic functions are rarely tested generically - i.e. not with any minimal abstract type.

* Promotion rules of non-numbers are unclear and underspecified, and accidentally changed recently on master because it's not documented nor tested anywhere

* There is a lot of "Yeah, X isn't really semantically correct, but I can exploit its weird behaviour in my own code, so we shouldn't fix it, it's actually a feature" hacker attitude among Julians.

There are like, 100 more small things that make Julia more bug-prone. I think this is a serious issue about the language that we should take note of and try to work on. You'll notice several of these issues can be resolved. But we need to take it seriously.

I'm starting to think that multiple independent implementations of a language are important for catching implementation quirks before they become entrenched.

They're also really useful for differential testing.

In Julia's defense, its ecosystem is nascent and it's not designed with a focus on guarding undefined behavior. So far the community has been really good at cataloging & fixing interop issues between packages. As time goes on I'm sure it will get better.

There's nothing in Common Lisp that would've prevented similar errors if its package library had been as big as Julia's. CLOS has even fewer guardrails than Julia's object system.

I don't use CL either, I was just warning others about Julia's correctness bugs and maintainers' attitudes towards them.

Reply on news.ycombinator.com