My Ideal Array Language | Hacker News Reader

My Ideal Array Language

ashermancinelli.com

148 pointsbobajeff71 comments

The author of this post was the guest on the most recent episode of the podcast The Array Cast

https://www.arraycast.com/episodes/episode111-ideal-array-la...

Strange to read this article and find no mention of Julia (but APL, Mojo, MLIR BQN etc.. which are not exactly widely used languages). It checks many of the boxes

User-Extensible Rank Polymorphism is just beautiful with the broadcast dot syntax. I don't think any other language has this clean and flexible implementation.

Others -- GPU programing, parallelism, etc. are pretty good with Julia. Real shame it hasn't taken off.

I love working in Julia, it makes clean numerical code so easy to write.

> User-Extensible Rank Polymorphism is just beautiful with the broadcast dot syntax ...

Just to be clear, I guess that Julia's broadcast (dot) syntax is an implementation of "User-Extensible Rank Polymorphism"; is that right? Or does Julia's dot syntax include more than UERP?

I'm in a position of supporting a Julia environment but not writing Julia myself. From my perspective, they need to fix time-to-first-plot before it can be adopted more broadly. It's horrendous. The "2-3 minutes" I see online is an aggressive estimate; it's more than that for our modest set of data science packages on very beefy workstations. I had to use PackageCompiler.jl to build a sysimage which shifted a ton of burden onto me (and onto GitHub Actions) to avoid a long precompile on every user machine. I had to do the same to get my Julia Docker image to stop precompiling on every new cloud machine even though it had already been precompiled during docker build. I would describe this process as a nightmare, and it was a serious problem--the thing was precompiling every time on every job run in the cloud using the docker image.

That' s solved more than 1 year ago https://news.ycombinator.com/item?id=38783886 https://modernjuliaworkflows.org/optimizing/#compilation .

Are you actually sure it's solved? I note that Julia proponents have a history of claiming that an issue has been solved when it is apparent that it has not been solved (see e.g. https://danluu.com/julialang/).

Yes. I don' t know why you' d rather trust an article from 10 years ago to understand current status.

The events of my post happened in the last two months on Julia version 1.11.5, and this poster simply linked to a primer on PackageCompiler.jl which is the thing I said I was using in my post. It's not solved; at best there are complicated mitigations that let you control when the long compilation happens without avoiding it. If you try out Julia, you will run into the issue, and as a first-time user you won't be prepared to implement the mitigations, none of which are suitable for ad hoc usage. The sysimage only works for us because we rarely change our packages so we can build it ahead of time in GitHub Actions.

The tradeoff between compilation time and programmer productivity and running speed is worth it compared with using any other language. You should consider most users don' t need to build an image . They just do things in the REPL with Revise.jl (which is fast). Or use a bit of PrecompileTools.jl to get good enough startup time (which isn' t complicated if you are the code writer). Package compiling works even large and slow is a bonus. https://pypl.github.io/PYPL.html Julia still grows despite all those controversial articles for a reason.

Julia may be cool, but it's not an array language in the tradition of APL.

Sure but neither is Mojo.

I've understood array language to mean "a language with focus on array processing" rather than "APL descendant". The array language comparison I found online lists Fortran, MATLAB, Julia etc. as array languages: https://github.com/codereport/array-language-comparisons

I think none of the languages in the linked article are, and that's not what it's about as sibling comments point out.

It has on some industries, keep it steady and slow, eventually more people will notice it.

https://juliahub.com/industries/case-studies

Many languages could dream to have even a third of such high profile case studies.

Shame that it had so many nice ideas, but everything about Julia is half-baked at best.

I like to play with it, but just it.

> User-Extensible Rank Polymorphism

> IMO this is what makes something an array language.

Great to hear. So what is it?

Not op, but I assume it means that there's rank polymorphism (i.e. data can be of arbitrary dimensions, and there's support for things like functions working on both N-dimensions, without having to specify n, or maybe setting constraints on n), and that the polymorphism can be used on the programmer side (so it's not limited to just a handful of language builtins) through the oop equivalent of subclasses and interfaces.

A question, would you interpret this as rank polymorphism?

  schema do
    input do
      array :regions do
        float :tax_rate
        array :offices do
          float :col_adjustment
          array :employees do
            float :salary
            float :rating
          end
        end
      end
    end

    trait :high_performer, input.regions.offices.employees.rating > 4.5
    
    value :bonus do
      on high_performer, input.regions.offices.employees.salary * 0.25
      base input.regions.offices.employees.salary * 0.10
    end
    
    value :final_pay,
      (input.regions.offices.employees.salary + bonus * input.regions.offices.col_adjustment) *
      (1 - input.regions.tax_rate)
  end

  result = schema.from(nested_data)[:final_pay]
  # => [[[91_000, 63_700], [58_500]], [[71_225]]]

I think i'm misunderstanding, rank is explicit throughout this example but i'm not familiar with this syntax (ruby maybe?) but whatever the case i don't see the rank polymorphism.

If i'm reading the syntax correctly, this would translate in kdb/q to a raze (flatten) on the 3 dimensions (regions, offices, employees). Probably more likely to be expressed as a converge but in either case, the calculations here are not possible in a rank polymorphic way.

The broadcasting handles the rank differences automatically. When bonus (at employee level) multiplies with col_adjustment (at office level), each employee's bonus gets their office's adjustment applied, no flattening or manual reshaping. The structure [[[91_000, 63_700], [58_500]], [[71_225]]] was preserved.

This is from a Ruby DSL I'm working on (Kumi). Probably the broadcasting semantics are very different from traditional rank operators in q/APL?

Edit: I realized that I missed the input structure:

  Region 0: Office 0 has 2 employees, Office 1 has 1 employee
  Region 1: Office 0 has 1 employee

Just had a quick browse of the kumi readme - very cool, i like it

That means a lot, thanks!

I didn't downvote but was utterly puzzled with your example. After your response below, it occurs to me that you are confusing Employee rank (a business concept) with Array rank a mathematical concept. Either that or it is very strange explanation for rank polymorphism.

The programmer can define functions that operate on matrices without having to be explicit about the number of dimensions and possibly (types of data, size of data, or length).

Example 1: A function that can take as input a 4x2x8 matrix or a 3x7 matrix.

Example 2: A function that can take as input a 4x2x8 matrix and a 3x7 matrix and output a third matrix.

> A function that can take as input a 4x2x8 matrix and a 3x7 matrix and output a third matrix.

which shows that this feature request is complete jibberish

Why gibberish ? It's a common feature in both array languages and Iverson ghosts, and many find it extremely useful.

You mean like a "winner" function able to check for both Tic-Tac-Toe, a Connect Four field and a Similar 3D+ tower game?

I'm not sure I follow? The "winning" condition is different in all of those examples?

X adjacent cell values in an N dimensional array? For tic tac toe, it's 3 in a row, for connect 4 it's 4 in a row.

Fair enough, I guess. The "available moves" would be dramatically different, such that I'm not clear I see much of a win on this reuse. But, yeah, fair enough.

It was just the simplest case I could think of that would be reasonable. I'm not a heavy enough pure math guy to really get how an Array centric language is even all that valuable myself TBF.

Rank polymorphism means that a function can be polymorphic in the additional dimensions of arrays. For example, if you write a function that takes a 2x3 and a 4x5 array, it can also work on 10x15x2x3 and 10x15x4x5 arrays by broadcasting.

If rank polymorphism results in accepting both 4x2x8 and 3x7, then that means the function was a function on elements to begin with. Which is possible, but not the most interesting application of rank polymorphism.

> Rank polymorphism means that a function can be polymorphic in the additional dimensions of arrays. For example, if you write a function that takes a 2x3 and a 4x5 array, it can also work on 10x15x2x3 and 10x15x4x5 arrays by broadcasting.

Thanks, this is what I was ineloquently attempting to describe with "A function that can take as input a 4x2x8 matrix or a 3x7 matrix."

game math libraries often have this (and glsl gpu shader language), like "2 * vec3(1,2,3)" results in "vec3(2,4,6)"

There are other cases like adding vectors to matrices and so on, but in the end this logic is defined in some custom add operator overload on a class or object in the language.

(I had no idea what it meant either until i searched for examples..)

I wondered the same. Similarly I wish the author had provided examples for statements like

> Numpy also needs to be paired with a JIT compiler to make python a real array language

My ideal array language is one in which array operations are function compositions, since arrays are functions. A functional view of array expressions naturally minimizes needless temporaries in most cases.

See https://github.com/llvm/llvm-project/blob/main/flang/docs/Ar....

Funny, on another totally unrelated domain (business logic/rules engines) I was building something very very related - array broadcasting with semantic preservation through arbitrary nesting levels

A previous relevant discussion since there is so little on Array Languages - https://news.ycombinator.com/item?id=38981639

You explain the evolution of CPUs but then don’t explain Rank Polymorphism.

It's just like polymorphism, only stinkier

"Rank" means the number of dimensions of an array.

So "rank polymorphism" means being able to write expressions that work correctly regardless of how many dims the arrays involved have.

For example, in numpy you can write a function that handles both lists and matrices automatically, by taking advantage of broadcasting. (The J language takes this idea a lot further -- numpy is a fairly minimal implementation of the idea.)

That just sounds like someone needing to feel smarter than 'multidimensional arrays' sounds. Which if you ask the average unskilled laborer, already sounds pretty damned fancy.

It's not the same thing as multidimensional arrays though. You can have multidimensional arrays without rank polymorphism. Rank polymorphism makes working with multidimensional arrays much easier because you can write one function which works over input arrays with different shapes.

  ⊢×0≠∧˝˘∧⌜∧˝           # Marshall & Dzaima (tacit!)
  (≠⥊∧´)˘{×(⌾⍉∧)0≠} # Dzaima & Rampoina
  {×(∧˝˘∧≢⥊∧˝)0≠}     # Dzaima

Call me old fashioned and stuck in C style syntax but I can't imagine anyone describing this as beautiful art.

I see it as beautiful the same way Galadriel would be beautiful as the Dark Queen. Utterly captivating and powerful, and yet something that should never be.

"All shall love me and despair"

When the junior programmers start saying "Turing complete" or the academics build a DSL in Julia that uses RegEx to parse Logic Symbols and stuffs the result in variables that use ancient characters that don't appear on your keyboard, it's a sure sign of imminent progress. Bonus if the PhD with nine years of schooling and five months of PHP experience at Facebook starts using emoji in commit messages.

“irony! Oh, no, no, we don't get that here. See, uh, people ski topless here while smoking dope, so irony's not really a, a high priority. We haven't had any irony here since about, uh, '83, when I was the only practitioner of it. And I stopped because I was tired of being stared at.”

It's giving APL: https://en.wikipedia.org/wiki/APL_(programming_language)

It is BQN, a descendant language

Why is it BQN instead of BQM? Clearly the idea was to increment every letter from APL, but then they had to go one further on the third letter.

It's just. So gross. Say it. Sudden interruption of slime coming up your throat. Like walking out the door into a spiderweb. Alphabetically I was mistaken but in every way that matters I was right.

Hmm. I guess it if was BQM, it would be pronounced “bequem” which means comfortable in German.

And a comfortable APL is clearly an oxymoron.

Ordinarily I'd make fun of the Germans for giving such an ugly name to a nice concept, but I've always found "comfortable" to be rather unpleasant too (the root "comfort" is fine).

I’m hoping they pronounce it “beacon” but the off by one error jokes also just write themselves.

No, it's 'bacon' :-)

They were following a Fibonacci sequence.

Supposedly it stands for "Big Questions Notation", but that could just be a backronym.

I'm somewhat sure the author actually mentioned that that was the intention, "Big Question Notation" and basically "apl" + 1. But he realized that it didn't match up

This actual answer according the the author realized after he already liked the name.

He created it intending to be +1 of APL. Accidentally came up with BQN instead of BQM. Sat with that for 1hr, really liked the name, then realized that it should be BQM which he hated, so he stuck with BQN.

That said, it's and incredibly designed language. I honestly have never read any language (especially not designed by a single person) with the level of creative thought as he put into BQN. Some really incredible insights and deep understanding. It's amazing reading his posts / documentation about it. The focus on ergonomics, brand new constructs and the consistency/coherence of how all of his decisions fit together is really impressive.

So, you write bequations in it? ;)

Well, do you know how it works? Don't judge a book by its cover and all. Although none of these are entirely aiming for elegance. The first is code golf and the other two have some performance hacks that I doubt are even good any more, but replacing ∧≢⥊ with ∧⌜ in the last gets you something decent (personally I'm more in the "utilitarian code is never art" camp, but I'd have no reason to direct that at any specific language).

The double-struck characters have disappeared from the second and third lines creating a fun puzzle. Original post https://www.ashermancinelli.com/csblog/2022-5-2-BQN-reflecti... has the answers.

Array Programming is an acquired taste, but once you do, solutions can be extremely simple, both to write and to explain.

Think about using matrix to describe geometric transformations instead of using standard functions.

You mean abstracted stuff like this instead of two lines of code that actually uses x and y?

  [ cos(θ)  -sin(θ)  0  0  ]
  [ sin(θ)   cos(θ)  0  0  ]
  [ 0        0       1  0  ]
  [ 0        0       0  1  ]

Yes, absolutely. Once you've expressed data as appropriate tensors (not all languages make this super convenient, unfortunately), it makes the implementation very readable, and easy to ensure that it's bug-free. It lets you see what is going on. Much better than futzing around with tensor components!

This type of syntax allows rapid iteration when looking at different implementations and experimenting with array problems. It should be thought of more as math notation than general programming.

Dlang does not has rank polymorphism and it handle array just fine with crazy speed in both compilation and execution.

It can be faster than Fortran based library that is still being used by Matlab, Rust and Julia [1].

It will be interesting to compare Mojo moblas BLAS library with GLAS library performance in D.

[1] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen (2016):

http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

If I understand correctly what is meant by rank polymorphism, it is not just about speed, but about ergonomics.

Taking examples I am familiar w/, it is key that you can add a scalar 1 to a rank 2 array in numpy/matllab without having to explicitly create a rank 2 array of 1s, and numpy somehow generalizes that (broadcasting). I understand other array programming languages have more advanced/generic versions of broadcasting, but I am not super familiar w/ them

Unfortunely, while the community is great, what it doesn't have is a direction, thus keeps pivoting every couple of years, and with that lost the adoption opportunity window it had a decade ago.

Form is temporary, class is permanent.

Reply on news.ycombinator.com