APL compiler targeting a typed array intermediate language | Hacker News Reader

APL compiler targeting a typed array intermediate language

github.com

123 pointsAthas29 comments

This is awesome.

I always recommend that people at least look into APL, to get a taste of how different some paradigms of programming are from what they're used to. Unfortunately, the de factor implementation of APL (Dyalog) is proprietary, and last I checked, the process of getting a trial license takes too much time to be worth it for people with only a passing interest.

It's great to see another open source implementation.

Here's my favourite video of APL in action. It'll blow your mind: https://www.youtube.com/watch?v=a9xAKttWgP4

not to take away from this, which is quite cool but:

GnuAPL is coming along nicely, and of course libre.

Dyalog has a 32-bit windows (though it works on wine middling well) only unregistered version http://dyalog.com/download-zone/unregistered.htm that other than a splash screen works great. Also if you're a student: I sent them like a 2 sentence email(from a .edu domain) that basically said "hey I'm a student, I wanted to play around with APL for data analysis" and they got back to me with a student license in like a couple hours. I am wary of proprietary environments I'm not going to argue for them, but getting a hold of a working+legal copy of Dyalog is actually pretty painless

> I'm not going to argue for them, but getting a hold of a working+legal copy of Dyalog is actually pretty painless

That's good to hear. Last time I tried it took a few days for them to get back to me.

I'm also excited to give GnuAPL another try too. It looks like it's coming a long way since the initial release.

If you're serious about investing time in something, 2 days is not much, although it didn't take that long for me.

Dyalog has great documentation, videos, and resources. I tried NARS2000, a free version of APL, but it was not as polished.

I mainly use J, and since Ken Iverson created APL, and then went on to create J with the help of a bunch of other smart people. J has certain things not found in Dyalog APL, and is very functional with trains and forks and general function composition. J is damn fast for an interpreted language, and the Jd database, which is a bit slower than kdb+/q, still is faster than many Spark/Shark setups for big data.

I am waiting for something like Futhark mentioned below, which marries J or APL to the GPU or FPGA or other hybrid card. A true vector/array language running on true array hardware, hopefully ridding the CPU-GPU bottleneck back and forth between host and device.

Plus I just like playing with J. I have used Lisp/Scheme for years, but languages like Forth and J/APL/K/Q really blow my mind, more so than the legendary Lisp ,'epiphany' at least for me.

> GnuAPL is coming along nicely, and of course libre.

From what I have been able to determine, most APL performance comes from hand-tuned implementations of the primitives, and of specially recognised idioms[0], for which the interpreter generates specialised code. This seems to imply that building an efficient APL interpreter requires a significant amount of manpower to implement all these cases, rather than being able to get away with having a few simple good ideas. Are you aware of any GNU APL vs Dyalog performance comparisons?

[0]: http://aplwiki.com/FinnAplIdiomLibrary

Your implication does not follow from your correct observation. APLs (including J and K) work from the simplest principles, and from then on it is just optimization.

K was implemented by Arthur Whitney, who implemented all the primitives (and their special cases) himself. At some point, if I am not mistaken, Niall Dalton rewrote some parts to be even faster - most primitives are easy to do efficiently (though a stunt programmer can definitely do them more efficiently), and most importantly - most patterns are too. E.g. "a@<a" means "sorted a". "*a@<a" and "&/a" both mean "minimum value in a" (each is a different recipe to compute it). It's not hard to implement, and beyond the primitives, 200 patterns cover a very significant percentage of use cases, and are definitely within the realm of possibility for a single person, let alone demigods like Arthur Whitney, Niall Dalton, Mike Pall, or a small team.

Silly question: Historically, did anyone ever attempt to create the APL primitives as separate utilities? The crazy idea that keeps recurring in my mind is that one could have a UNIX userland made of APL primitives. k's mmap approach, avoiding I/O, is preferable. But even if one used a named pipe or UNIX socket to do IPC, perhaps it could still be fast enough to be useful or fun. Feel free to dismiss this idea, but please kindly explain why it would not work.

In some way, IBM APL2 [1] was trying to achieve that with all of their APs (associated processors, APL's own way to do IPC) and built-in workspaces. You can use APL as your login shell, since it really has all the things a normal shell/CLI provides and more. Mainframe APL had all the account management capabilities for people to login remotely through their phone lines, almost the same time as Unix systems start flourish.

Unix and later linux/BSD's took off largely based on their open source and easy to duplicate and deploy (c compiler everywhere), but APL had it's corporative baggage and remained a niche (very much like kdb+ wouldn't go open/libre, and IBM is still charging a fortune for APL2).

[1] http://www-01.ibm.com/software/awdtools/apl/apl2-v2.html

As jxy noted, APL2 actually was used that way in the mainframe days (I actually had the pleasure to use it, and it WAS a pleasure compared to any other facility offered on that system).

However, it's hard to replicate unix userland as K primitives - Unix pipes work well because every participant streams along implicitly; however, one of the subtle and often invisible truths of APL and K programming is that everything sort of assumes the length of every vector/list/axis/dictionary is accessible. It's not something that cannot be removed, it's just something that AFAIK has never been removed -- and the implications of removing it are likely to make APL or K less concise. Streaming in APL and K tends to be explicit.

K and APL are ridiculously useful, and once you've actually grokked the idea, it's hard to look at conventional programming languages (Java, C, C#, even Python) and not feel disgusted at the useless extreme verbosity and fragile building blocks. Some people use them as their shell, and have no issue delegating to "standard" unix userland. In fact, an older version of K IIRC had its "create directory" operation implemented as {system "mkdir ",quote x} or something like that. It might not look that way to an outsider, but extreme pragmatism is strong with the K community.

Any particular reason you can't output the length before the array stream? (I don't know APL, but want to learn it)

The problem is that unix pipe can be blocked and you can see it as a lazy partial evaluation, so grep a HUGEFILE|head returns when 10 lines are found. In APL the semantics is not builtin, though things like ⍴↑↓ can be lazy, it's the internal optimization, rather than a language specification.

What if someone took c.c, k.h, linenoise.c and linenoise.h and made a tiny k4 REPL for UNIX with commandline history?

Would the k community appreciate this as useful, pragmatic solution?

I am a BSD /emul/linux, /bin/sh, vi-mode, ESC / user. I need searchable commandline history for q)k) and I really do not like rlwrap as a solution.

sources:

http://kx.com/q/c/c/ http://raw.githubusercontent.com/antirez/linenoise/master/li... http://raw.githubusercontent.com/antirez/linenoise/master/li...

For q/k in particular, you can just write a wrapper in q/k. Nothing stops you from doing

    while[1;0N!. 0:0]

you can save the history in a table and do whatever you want.

"Nothing stops you..."

Except my lack of creativity using k at this point.

Thanks for this.

As a k noob, I wonder: is thinking of solutions using iteration and control structures a bad habit, at least until I have command of the rest of the language?

Currently I'm using recordio to save my sessions; crude, but it works well enough:

  recordio q 2>log

Here's another solution, a little better than rlwrap:

  socat readline,history=file exec:q

As of June 30th, when Dyalog v15.0 was released, Personal/Non-Commercial licenses are free of charge for 64-bit Windows, Linux and OS/X - and on the Raspberry Pi. We are in the process of streamlining the registration screen, please bear with us for a little while longer until it catches up.

I'm a huge fan of J, which is a descendant of APL, and has a libre implementation. I think it offers many of the same advantages in learning the array programming "paradigm", and has a few tricks of its own.

An interesting project was kicked around here / proggit the other day, using apltail as the middle of a pipeline transforming APL code into optimized GPU code (Futhark), instead of targeting C. It is a pretty good display of ML's suitability for this sort of task.

I've always been impressed (amazed?) at the performance APL and its descendants get in an interpreted environment -- I'd be interested to see performance comparisons between the output of apl-to-apltail-to-C / apl-to-apltail-to-futhark and an APL interpreter.

I believe this is the paper in question!

https://futhark-lang.org/publications/fhpc16.pdf

Oh wow, hard numbers, thanks! That's very impressive, but strangely misleading in the graph towards the bottom that appears to clock TAIL->Futhark->OpenCL and Futhark->OpenCL exactly the same in more than half the cases (if you look at the raw numbers if appears that the second approach blows the first away in most).

That's because the speedup graph cuts away at a 100x speedup, but some of these (somewhat silly) benchmarks are compute-bound and hence gets these crazy 1000x speedups on the GPU.

Shashi Gowda presented on APL implemented in Julia at JuliaCon today: https://gist.github.com/shashi/9ad9de91d1aa12f006c4

This is quite interesting, especially the llvm jit bits. Are there any performance figures?

It's just a shallow transformation to julia, so it's fast ;). Would be fun to compare with a production quality APL implementation (I hear they have some fancy optimizations that we don't yet have in julia).

I used APL professionally for about 10 years. It absolutely changes the way you see and think about solving problems with computers.

Give it a shot: http://www.nars2000.org/

Relatedly, I highly recommend anyone read Notation as a Tool of Thought¹ if they haven't yet.

If I may inquire, what did you work on with APL?

¹ http://www.jsoftware.com/papers/tot.htm

A range of applications from inventory management for electronics manufacturing to DNA sequencing and robotics.

If APL is your interest and you like Smalltalk, the take a look at F-Script. It is an interesting take on array programming.

This is one of a few GPU capable compilers currently targeting APL as a source language. I am the implementer for another system, called Co-dfns, which is designed to integrate GPU compilation into Dyalog APL. A third system is the APEX compiler that targets SAC written by Bernecky. We get together at the ARRAY workshop and PLDI each year and usually go over and compare our systems to one another. Each system is bringing some unique features and capabilities to the table, and it will be interesting to see these systems evolve over time.

More information on Co-dfns can be found here:

https://arcfide.github.io/Co-dfns/

Reply on news.ycombinator.com