Julia's Arraypocalypse Release | Hacker News Reader

Julia's Arraypocalypse Release

github.com

62 pointsViralBShah61 comments

Probably worth noting that the GitHub issue in question relates to work being done on Julia 0.5, a future release that is not yet ready to ship and won't ship for several months.

Within a few weeks, Julia 0.4 will be shipping, which is not focused on changes to arrays.

The "arraypocalypse" name is somewhat tongue in cheek, referring to the fact that this change will include some breaking changes. To provide some very brief motivation for the two most significant changes that are planned:

1. Array slices will create views rather than copies. This is just a matter of performance – although possible, it is currently way too annoying to write copy-free array code. This change will make it the default.

2. Slicing will behave more like APL: the rank of a slice of an array will be the sum of the ranks of the slices. This is a regular and powerful scheme, but sometimes at odds with how slicing works in Matlab.

These changes will make the language easier to write high-performance, general code, while, with some care not making linear algebra harder.

> Array slices will create views rather than copies. This is just a matter of performance

Except for breaking any existing code which mutates slices (unless slices are COW?), and implicit sharing can have disastrous effects on memory usage.

How can implicit sharing have "disasterous effects on memory usage"? It will certainly break some code, hence the large amount of warning and the "pocalypse" appellation. Note that this is how array slices work in NumPy and that ecosystem has not collapsed.

> How can implicit sharing have "disasterous effects on memory usage"?

A small long-lived slices keeping a original arrays alive implicitly. Oracle recently undid the same optimisation in Java's strings because of that exact issue (which obviously broke code the other way around, code relying on cheap substrings would now take O(n) instead of O(1)). Erlang developers are also warned early and often about that behaviour in binaries.

Very true. Less of a problem in scientific computing than web services, I think, since it's not common to keep slices around: you generally process a "frame" or an iteration of a simulation (Monte Carlo, etc.) and perhaps compare it to a previous frame, but the only history you keep beyond that are aggregates of the data, rather than individual slices that might go (say) in a global dictionary and stay around for a while. I don't think I've ever seen NumPy code where array slices were used as values in a long-living in-memory key-value store or cache... whereas you see that all the time with substrings in code for webservices. Thus it hasn't been as big a problem as in the Java community.

Still something that should be in bold text in Julia 0.5's readmes, though; slicing is cheap now, but it comes with this very important caveat!

The primary use case for Julia's coming array slices is to provide views into an array that you always intended to keep alive anyway.

Iterating over the columns of an in-memory matrix, for example, is the use case that this change is meant to speed up.

And so it was with Java's substring and is for Erlang's binaries sharing. I will acknowledge that array slices are less likely to cause problems than string slices, but it is still a possible issue for the unwary. Which was my point.

Totally different typical usage patterns. Taking substrings and throwing away the original string is the norm for text. Taking a column of a matrix and throwing away the original matrix is not. Different domains call for different design tradeoffs – this change stems from extensive practical experience with writing numerical algorithms.

would there still be an easy way to make copies?

Like advanced indexing (http://docs.scipy.org/doc/numpy/reference/arrays.indexing.ht...) in the Python world?

You could just use a list comprehension, e.g. `[ x for x in arrayveiw1 ]` or perhaps a copy function will be added to Base.

There's already a copy function that does this.

Great, Julia seems to have a good assortment of those "I need XYZ" (for scientific programming) and it's already in Base. I should check the apropos more.

Note that I am not saying it's a bad default (it still is the default of Erlang binaries).

I am saying this change is not quite "just a matter of performance", that the documentation and changelogs should be very very clear about the behavioural changes and the possible issues with the new behaviour, and that there should be an easy way to "unshare" slices and force a copy.

The post you're responding to says explicitly in the first sentence that these are breaking changes.

1. the comment I replied to stated — and this is the part I quoted and to which I was responding — that slicing creating views is just a matter of performance.

2. there are gradations in breaking changes, the breaking changes I outline are insidious because they are silent, they won't cause the program to fail loudly, but they will cause it to behave very differently than it used to and possibly to corrupt data.

This is great to hear. This should have been how arrays worked in the first place in my opinion. Julia is all round very good but currently, as someone looking to move from a Fortran codebase, the array implementation is lacking in many areas and the performance is bad due to implicit copies everywhere, allowing arbitrary number of indexes to get elements, temporary arrays created for array arithmetic, and temporary arrays inside functions having to be allocated and deallocated on every function call. All of which is a bit disappointing in a language which is trying to be the future of numerical computing.

As it stands if you translate Fortran into Julia without a deep understanding of what is happening below the surface, your code will be much, much slower.

Could someone maybe tl;dr what is going on here?

I've tried out Julia a few years back, but didn't find it ready for serious use (yet). Am I right to assume that this means the language is still in serious flux?

Compared to a few years back, Julia is in seriously good shape. The long and short of this is that while the language has largely been stable, the array library in Julia will get an upgrade to become a lot more modern and clean.

For most users, this should not change much. However, it gives us the ability to do some really amazing things with our array infrastructure going forward.

"However, it gives us the ability to do some really amazing things with our array infrastructure going forward."

What kind of things? Super curious.

The github issue linked actually has a detailed list of all the things that are planned - native bounds checking and removal, reshaped arrays, a lot more consistency in indexing and concatenation, slices as views, etc.

Also, any plans for the stats ecosystem? (dataframe, dashboards, imputation, data cleaning etc)

It is still a bit wanting.

I've been doing a lot of work behind the scenes there, but it's still going to take a while to offer a mature product. What we really need are more qualified engineers to speed up the process, but there's a vanishingly small group of people who are interested in doing eng work on statistics libraries.

Makes sense. Thanks for both your open source work and your reply!

A lot of interesting things are going on in the DataFrames area - especially with the recent work on NullableArrays by David Gold and John Myles White. A blog post will be out soon describing it all. In my opinion, Escher.jl makes it really easy to build amazing dashboards and is progressing well.

But, a lot more remains to be done, and contributions would be really welcome in these areas that take Julia's statistical computing to the next level.

I'd like to add a note about my experience with doing this; I ended up forking Distributions.jl and added a fitting function for the Weibull distribution. Still need to clean it up and do a pull request,.

The amazing thing about Julia I've come to enjoy is how clean the source code, how easy it is to find functionality, and it's all written IN Julia. That means I can modify libraries that are normally C or C++ for perf reasons. It's changed the prospect of needing to add functionality from one of hesitation to feeling that it's actually enjoyable.

I'm really glad to hear this! I'd like to think that the community and inspectability argument for Julia is actually one of the strongest ones out there, but it's not easy convincing people who haven't had that experience.

It really is pretty great.

I must say that while some of this work may not change much for users, like array slices as views, for many - the entire set of the arraypocalypse work is certainly a huge change.

The key is to get it all done sooner so that we can keep getting that much closer to 1.0.

There are many many tomes of discussion that have led to this point (I'm mbauman).

In short, there is an inherent disconnect between arrays-as-containers and matrices-as-linear-algebra-operators. There are many ways that you can define array operations such that the universe is closed and self-consistent, but there always seems to be some awkwardness. Right now, Julia uses some Matlab-ish semantics (sometimes ignoring trailing singleton dimensions), but there is an interaction with Julia's types that makes the status quo a little strange.

So there are "just" two major breaking changes proposed here: changing our array semantics universe to be more APL-like (instead of Matlabish), and making indexing return a view instead of a copy.

A little bit of history here. Back when we started Julia 6 years ago, I personally was in a much more of a Matlab/Octave frame of mind from my own experience. That is why the early array implementation followed that style.

We had no idea where Julia would go. In some sense, it has been a personal journey of maturity in my own thinking, and if I can take the liberty to say so, the same is true for the community as well.

Has your stance on one-based array indexing changed in those 6 years? How many off-by-one issues have you encountered?

I'm not from a Matlab/Octave background, so one-based indexing feels really foreign to me.

For what it's worth, I've ran into this a couple of times. Maybe it's just because I'm more familiar with 0-based indexing, but I think it's not just that. In every case the 0-based indexing had fewer (+1)'s and (-1)'s than the 1-based code. For instance k%n returns a number in the range [0,n), or in Julia notation [0:n-1]. When you want to do array indexing with that, you need (k%n)+1 in Julia.

Btw, Matlab also uses 1-based indexing, no? I thought that Julia chose 1-based indexing to get brain backwards compatibility with Matlab.

Not just Matlab, but many technical computing languages use 1-based indexing. Julia was never intended to have any compatibility with Matlab.

It's just bikeshedding IMO. When I write C# code, array indices are like memory operation in my mind. When I go to Octave indices are like subscripts on sequences to me. Anyone who think it really makes a difference is being ridiculous.

I think zero based is "prettier" because of half open intervals, but it really doesn't matter.

I don't think it is bikeshedding IMO from a pure mathematics point of view. In set theory you begin with the empty set as zero, and the next integer is N+1. I find it interesting that the Julia devs have gone with APLs style of slicing, and did not move to 0-based indexing. Ken Iverson, who created APL and later the J programming language, chose 0-based indexing for J. APL has a system variable QUAD-IO that allows you to choose your indexing value of 0 or 1, which leads to many issues. Iverson's books on math (Arithmetic, Calculus, and the Concrete Math Companion (Knuth's book done in J)) all to demonstrate mathematics in J, all begin at Chapter 0. I think the choice of index in APL is a bad idea. I have run other people's code, and realized it doesn't work because they set their indexing to 1 vs. 0 for me. People and programmers can work around most things, but I think if you are a functional programmer, and mathematics is your thing, you are more comfortable with 0-based indexing. This of course does not address scientists, for whom pure mathematics is not at the forefront of whatever they are working on. I do think it leads to problems down the road for 1-based indexing.

Julia looks very interesting but honestly, exactly the 1-based indexing makes it feel really awkward and turns me off every time I think about using it somewhere.

Wouldn't it be possible to create array classes in Julia that all indexed zero-based? I would welcome a 'Julia0' fork :-)

While I agree that it's an annoyance, it's a very small annoyance. Since Julia has for loops and Julia ranges use the same 1-based convention you don't run into it often.

I feel that 1-based indexing has been the right choice, and I am aware that many do not like it. I really haven't run into off-by-one issues much, but this is probably more about how one is used to writing code. I like that you can iterate over a vector of size n, by just going from 1 to n.

Coming from a regular programming background, as opposed to a technical computing background, can certainly make this look alien. However, I think it starts feeling natural quickly. There are many other things in Julia that can take a lot longer to get used to.

It sucks for slicing though. In python (zero based), you can use 0:n to slice the entire array with a half-open interval. With one based, you have to either have a closed interval, so 1:n is the whole array (which sucks because what does an empty interval starting at 1 look like? 1:0?) Or you can have a half-open interval and index 1:n+1. I don't see what you saved yourself other than making things uglier.

For those who don't like one-based indexing, your problem is solved: there is now a package which provides two-based indexing https://github.com/simonster/TwoBasedIndexing.jl ;-)

Edit: For a history of 0-based indexing see http://exple.tive.org/blarg/2013/10/22/citation-needed/ My understanding is that with modern compilers, there's no need for 0-based indexing; you can use 0-base, 1-base, or 69-base indexing as you prefer. 0-base is applicable to people creating chips or writing hardware drivers. Most scientific programmers are not so closely tied to hardware; 1-based indexing is one of the things that helps make Julia more accessible to high-school programmers or PhD students who don't want to learn binary arithmetic.

> My understanding is that with modern compilers, there's no need for 0-based indexing; you can use 0-base, 1-base, or 69-base indexing as you prefer.

It's never been needed, Pascal had arbitrary array bounds (including negative) 50 years ago.

> 1-based indexing is one of the things that helps make Julia more accessible to high-school programmers or PhD students who don't want to learn binary arithmetic.

What does binary arithmetic have to do with anything? And have there been any study pertaining to the accessibility of 0-indexed versus 1-indexed arrays to high-scool programmers which would warrant such an assertion?

Zero based indexing counts offsets. Literally in C, you are saying "add n times sizeof(T) to the base pointer and dereference it". It rather implies arrays are contiguous in storage and lack any "holes". It's a crude but close-to-the-metal approach. C arrays are not general, they are one specific structure. An array could be implemented as a tree, but C doesn't have built-in support for that.

One based indexing counts, or labels boxes. Counting implies no "holes", but abandons any implication about how they are laid out in storage. Labelling allows treating numerical arrays as sparse hash tables, with as many "holes" as needed. It's a high level view, suited to mathy ways of thinking about data.

You can "label" by just agreeing that zero is in the set of counting numbers and labelling with those. Just because it's zero based doesn't mean it has to be an offset.

One-based is just wrong. I've never been in a zero-based language and wanted one-based, but I go the other way all the time. Arrays beginning with zero are isomorphic to the ring formed by Z modulo N, and you can use a bit mask for modulo with a power-of-two array size. It's not an "implementation detail" that arrays start with zero it's a "zero is important to mathematics" thing.

If I want to build a hash table, a power of two size for the underlying array is a nice performance optimization since you can use a bitmask instead of a modulo operation, and it makes a big performance difference, what does one-based give you other than some nebulous "it just _seems_ simpler to think about" kind of nonsense? I love how we expect people to grok the type system but we're afraid zero-based arrays will scare mathematicians off.

See: https://en.wikipedia.org/wiki/Modular_arithmetic#Integers_mo...

My impression is that you aim to produce a perfectly clean and elegant – perhaps in order to avoid a Python-3-type scenario. However, if that is what you are doing, I am afraid that Julia could become the GNU Hurd or programming languages. Most languages, at least those that have practical relevance, have design flaws. Heck, R is one giant mess of design flaws, and yet it is actually amazingly useful. Another example would be Perl. I think design flaws can be a sign that the creators of these languages are reasonable people who are capable of making pragmatic decisions (instead of just academic decisions). I’m clearly not an expert and I’m not up-to-date with what is going on in Julia but as a potential user I have the impression that we’re not moving closer to a point where Julia is ready for production use, and I’m worried that Julia will lose momentum while R is growing stronger and deeper roots. Personally, I really like Julia and would prefer if it would replace R but over time this is becoming less and less likely. The world is full with examples showing that inferior technologies can become dominant just because they had a head start. I think this article is relevant here: http://paulgraham.com/startuplessons.html

On the other hand, Rust spent a long time undergoing tumultuous upheaval, considerably more radical than this, i think, and yet it did eventually stabilise and produce a 1.0.

Oh yeah. Rust was really four different programming languages over a roughly eight year period. Always the same focus, but radically different ways of accomplishing it.

Personally, I would much rather see Julia undergo serious upheavals in an attempt to make an elegant language that works really well for scientific/HPC computing. Too many languages get trapped trying to provide backwards compatibility for everything and you end up with a lot of unintuitive cruft and quirks. If you want a language for "getting stuff done", use Python. But I find that I get a lot done with Julia in spite of constant change (in fact, I don't use Python at all).

Whether Julia replaces R has more to do with how R evolves over the next 5 years than Julia IMO. As far as "production use" I think that depends on whether what you're doing is well served by the existing libraries or not.

R as a language is not evolving at all and that is not a problem because the strength of the R system is its incredibly rich ecosystem. Julia cannot evolve a similar ecosystem as long as it doesn't provide stable foundations and for this reason it is completely uninteresting for most of the audience despite its superior design.

Yes and no; base R is extremely stable, but if you need to install dplyr, maggrittr, ggplot2, Rcpp and knitr/rmarkdown to reproduce a large fraction of the work that's been done in the last few years that seems somewhat equivalent to language evolution. But with rough edges, because some of the nice workflows that this opens up (say piping) are not necessarily in line with the way R wants to deal with objects.

In my experience, Julia is uninteresting to most statisticians because of the lack of mature libraries and because R is already good enough for most of the stuff they want to do. I don't think that relatively small language changes like the one discussed in the original issue are even on their radar, and I doubt that these changes are driving away people that would otherwise be maintaining packages.

I love Julia for its expressiveness but I am somewhat confused about its dynamic typing and its powerful type system. Are we to write mostly Python-like function and rely on multiple dispatch and JIT code generation for speed? Duck typing is one reason I stopped using Python for technical computing. I tried Julia out a few months ago and found that type inference has some holes that make type annotation brittle for function parameters. I finally gave up when I found out functions have no return type in Julia.

Hopefully Julia can work out all the language features and mature into a good technical language. It would also help if there is a guide about the canonical programming style. I prefer static typing but thought using type annotation I could come close to static typing while still retain the freedom of quick-and-dirty code. I believe for the current Julia I was wrong.

If you'd like to know canonical programming style, you should read the code in Julia Base. Learning how to write good Julia is somewhat difficult because the language occupies an unfamiliar position between traditional static languages and traditional dynamic languages.

Can you provide some concrete examples of the functionality you're looking for? What exactly does "quick-and-dirty" code look like to you?

I don't use "quick-and-dirty" as a derogatory term, so mostly I mean no error checking for rare occurrences or not ensuring correct values are being passed through. An example would be in Python if I just write MatrixA * MatrixB without any checking. I should check that they are both matrices if that is what I want, and that is what I did for reusable code. For one-off data analysis or computation, no-checking is certainly quick.

This StackOverflow answer may help clarify things: http://stackoverflow.com/a/28096079/659248. Making return types work well with multiple dispatch is not so simple; we feel that it's better to wait to add that feature and get it right than to rush things and have to fix it later. Some background, context and tentative plans for return type annotations: https://github.com/JuliaLang/julia/issues/1090.

Is there any progress when it comes to the slow package loading times of Julia? That made it quite unusable as a data exploration language when I last tried it about 1 year ago, but perhaps there was progress in that regard.

That issue is largely resolved in 0.4. Some functions are still a little slow, but most packages will now load much more quickly.

Julia 0.4, already in the RC phase, has precompilation thanks to Jameson Nash's work. It has been enabled for several packages, and gives sub-second load times.

Thank you both, there are very good news. I'll give it a try as soon as 0.4 is launched.

I'll be glad once this is all implemented. I recently rewrote most of my functions that took abstract vectors and matrices to instead take concrete arrays along with a few index parameters. I did this because I realized SubArrays were making huge memory allocations inside the tightest parts of my loops for some unknown reason. Since I didn't want to spend days debugging that weirdness, I just took the simpler (but cruder) option of passing in the "kitchen sink" to each function.

Reply on news.ycombinator.com