Hacker News Reader

Comment by StefanKarpinski | Hacker News Reader

The "arraypocalypse" name is somewhat tongue in cheek, referring to the fact that this change will include some breaking changes. To provide some very brief motivation for the two most significant changes that are planned:

1. Array slices will create views rather than copies. This is just a matter of performance – although possible, it is currently way too annoying to write copy-free array code. This change will make it the default.

2. Slicing will behave more like APL: the rank of a slice of an array will be the sum of the ranks of the slices. This is a regular and powerful scheme, but sometimes at odds with how slicing works in Matlab.

These changes will make the language easier to write high-performance, general code, while, with some care not making linear algebra harder.

View on HN

This is great to hear. This should have been how arrays worked in the first place in my opinion. Julia is all round very good but currently, as someone looking to move from a Fortran codebase, the array implementation is lacking in many areas and the performance is bad due to implicit copies everywhere, allowing arbitrary number of indexes to get elements, temporary arrays created for array arithmetic, and temporary arrays inside functions having to be allocated and deallocated on every function call. All of which is a bit disappointing in a language which is trying to be the future of numerical computing.

As it stands if you translate Fortran into Julia without a deep understanding of what is happening below the surface, your code will be much, much slower.

> Array slices will create views rather than copies. This is just a matter of performance

Except for breaking any existing code which mutates slices (unless slices are COW?), and implicit sharing can have disastrous effects on memory usage.

How can implicit sharing have "disasterous effects on memory usage"? It will certainly break some code, hence the large amount of warning and the "pocalypse" appellation. Note that this is how array slices work in NumPy and that ecosystem has not collapsed.

> How can implicit sharing have "disasterous effects on memory usage"?

A small long-lived slices keeping a original arrays alive implicitly. Oracle recently undid the same optimisation in Java's strings because of that exact issue (which obviously broke code the other way around, code relying on cheap substrings would now take O(n) instead of O(1)). Erlang developers are also warned early and often about that behaviour in binaries.

Very true. Less of a problem in scientific computing than web services, I think, since it's not common to keep slices around: you generally process a "frame" or an iteration of a simulation (Monte Carlo, etc.) and perhaps compare it to a previous frame, but the only history you keep beyond that are aggregates of the data, rather than individual slices that might go (say) in a global dictionary and stay around for a while. I don't think I've ever seen NumPy code where array slices were used as values in a long-living in-memory key-value store or cache... whereas you see that all the time with substrings in code for webservices. Thus it hasn't been as big a problem as in the Java community.

Still something that should be in bold text in Julia 0.5's readmes, though; slicing is cheap now, but it comes with this very important caveat!

The primary use case for Julia's coming array slices is to provide views into an array that you always intended to keep alive anyway.

Iterating over the columns of an in-memory matrix, for example, is the use case that this change is meant to speed up.

And so it was with Java's substring and is for Erlang's binaries sharing. I will acknowledge that array slices are less likely to cause problems than string slices, but it is still a possible issue for the unwary. Which was my point.

Totally different typical usage patterns. Taking substrings and throwing away the original string is the norm for text. Taking a column of a matrix and throwing away the original matrix is not. Different domains call for different design tradeoffs – this change stems from extensive practical experience with writing numerical algorithms.

would there still be an easy way to make copies?

Like advanced indexing (http://docs.scipy.org/doc/numpy/reference/arrays.indexing.ht...) in the Python world?

You could just use a list comprehension, e.g. `[ x for x in arrayveiw1 ]` or perhaps a copy function will be added to Base.

There's already a copy function that does this.

Great, Julia seems to have a good assortment of those "I need XYZ" (for scientific programming) and it's already in Base. I should check the apropos more.

Note that I am not saying it's a bad default (it still is the default of Erlang binaries).

I am saying this change is not quite "just a matter of performance", that the documentation and changelogs should be very very clear about the behavioural changes and the possible issues with the new behaviour, and that there should be an easy way to "unshare" slices and force a copy.

The post you're responding to says explicitly in the first sentence that these are breaking changes.

1. the comment I replied to stated — and this is the part I quoted and to which I was responding — that slicing creating views is just a matter of performance.

2. there are gradations in breaking changes, the breaking changes I outline are insidious because they are silent, they won't cause the program to fail loudly, but they will cause it to behave very differently than it used to and possibly to corrupt data.

Reply on news.ycombinator.com