Comment by one-more-minute | Hacker News Reader

Comment by one-more-minute | Hacker News Reader

one-more-minute

It might be useful to separate the issues that are "just" bugs from the problems that come with Julia's unusual level of composability. I have no idea if Julia has more bog-standard, local bugs – things like data structure problems or compiler faults – than other languages of comparable maturity and resources, but clearly the OP has bumped into several, which is frustrating.

The composition bugs – as in offsetarrays or AD – are a bit of a special case. In most languages package A will only work with package B if it's specifically designed to, and the combination will be explicitly developed and tested. That A and B can work together by default in Julia is really cool, but it also means that as you add new types and packages, you have a quadratically growing set of untested edges.

The canonical solution is strict interfaces. But Julia is laissez faire about those too (with some good reasons). Together this means that if A doesn't work with B as expected, it's not always easy even to assign fault, and both might be reluctant to effectively special-case the other. Program transformations (autodiff) compound this problem, because the default is that you promise to support the universe, and it's not easy to opt out of the weird cases.

I think it's absolutely right to celebrate Julia's approach to composition. I also hope new research (in Julia or elsewhere) will help us figure out how to tame it a bit.

> That A and B can work together by default in Julia is really cool, but it also means that as you add new types and packages, you have a quadratically growing set of untested edges.

But as the authors example showed, they clearly can't work together - they just fail at runtime instead of at compile time.

Other languages have generics and interfaces to make stuff like this dynamically exchangeable. Sure, your code needs to be designed to support this, but it also means that the author explicitly thought about what they expect from their data structures. If they don't, you might suddenly find yourself violating implicit assumptions like arrays starting at 1.

any tutorial/blog on what make julia compositionability special vs othe languages? Is there a relation with multiple dispatch or delegation?

Julia has a very nice type system, the nicest of any dynamically typed language I am familiar with. This is something to do with multiple dispatch, but it's more to do with trying to have a type system that allows all the JIT to unbox all the things that have to be unboxed for high performance without sacrificing the freedom of dynamic typing.

IIUC, Common Lisp is the giant on whose shoulders Julia built in this respect.

Yes. Its a side effect of multiple dispatch being the core paradigm of the language. See Stefan Karpinski's talk about it: https://www.youtube.com/watch?v=kc9HwsxE1OY

The title of Stefan's talk is great: The Unreasonable Effectiveness of Multiple Dispatch. He gives a nice example of composability: how you can throw a new type into an existing algorithm and it just works.

The "Unreasonable Effectiveness of Multiple Dispatch" talk is a good example of how multiple dispatch is special in a good way, in that everything (should) work together as new types and functions are added to the ecosystem. However, this also means the scope of potential integration bugs encompasses the entire ecosystem. The Julia manual has a small section about special composibility pitfalls arising from multiple dispatch: https://docs.julialang.org/en/v1/manual/methods/#man-method-...

As best as I can summarize it: Multiple dispatch is supposed to dispatch a function call to the implementation with the most "specific" call signature. This means that you must design your functions with an eye to what everyone else has implemented or might implement so whatever function gets called does the "right" thing, and also that your implementation doesn't block someone else from writing their own implementation specialized to other types. This requires some coordination across packages, as shown in one of the manual's examples.

The rules defining type specificity (subtyping) are complicated, and I think not in the manual. They have been inferred by observation: http://janvitek.org/pubs/oopsla18a.pdf. To quote from that paper, "In many systems answering the question whether t1 <: t2 is an easy part of the development. It was certainly not our expectation, approaching Julia, that reverse engineering and formalizing the subtype relation would prove to be the challenge on which we would spend our time and energy. As we kept uncovering layers of complexity, the question whether all of this was warranted kept us looking for ways to simplify the subtype relation. We did not find any major feature that could be dropped." Julia's multiple dispatch allows a high degree of composibility, but this does create new complexity and new problems.

Reply on news.ycombinator.com