Array Languages: R vs. APL (2023)
jcarroll.com.au
jcarroll.com.au
I think software engineers often get turned off by the weird idiosyncrasies of R, but there are surprisingly unique (arguably helpful) language features most people don't notice. Possibly because most of the learning material is data-science focused and so it doesn't emphasize the bonkers language features that R has.
In pseudocode:
f =
let x = 1 in # inner vars for f go here
arg -> arg + 1 # function logic goes here
# example one: no external value
f (x+1) # produces 3 (arg := (x+1) = 2; return arg +1)
# example two: x is defined in the outer scope
let x = 4 in
f (x+2) # produces 5 (arg := 4; return arg + 1)? Or 3 if inner x wins as in example one?https://blog.moertel.com/posts/2006-01-20-wondrous-oddities-...
Of course, playing with magic sounds fun until you remember you're trying to tell a computer to do a specific set of steps. Then magic looks more like a curse.
That was at least true when I was looking at it. I didn't get it, but the data guys came away loving it. I came away from that whole experience really appreciating how far you can get with an "unclean" design if you persist, and how my gut feeling of good (with all the heuristics for quality that entails) is really very domain specific.
`%in%<-` <- function(x, y, value) { x[x %in% y] <- value; x}
x <- c("a", "b", "c", "d")
x %in% c("a", "c") <- "o"
x
[1] "o" "b" "o" "d"
`<-<-` <- function(x, y, value) paste0(y, "_", value)
"a" -> x <- "b"
x
[1] "a_b"
The many recommendations for J here are a great nudge for me to give it a proper go. I've taken quite a liking to the traditional APL glyphs ( see a photo of the stickers on my laptop keys in this post https://jcarroll.com.au/2023/12/10/advent-of-array-elegance/ ) so I'm not looking for a way to avoid them.
Another detraction I've seen around is about the ambivalence of APL glyphs (taking either 1 or 2 arguments and doing something different in each case). I don't particularly mind it because I think it becomes more natural to "understand" how a function is being used the more familiar you become with it, but without the limitation on the number of glyphs, I can see the benefit of separating those.
(⊢⍳⌈/)+/x maxrow←(⊢⍳⌈/)+/
⊃(⍒+/)
Just for a short comparison, In J the analogous code is </ +. >/
Where / is for reduce, +. is for the GCD, the LCM is *.
The basic idea of J notation is using some small change to mean the contrary, for example {. for first and {: for last, {. for take and }. for drop (one symbol can be used as a unary or binary operator with different meaning. So if floor is <. you can guess what will be the symbol for roof. For another example /:~ is for sorting in ascending order and I imagine that you can guess what is the symbol for sorting in descending order. In a sense, J notation include some semantic meaning, a LLM could use that notation to try to change an algorithm. So perhaps someone could think about how to expand this idea for LLM to generate new algorithms.
m ; (+/ m) ; >./ +/ m
┌─────┬───────┬──┐
│0 1 2│9 12 15│15│
│3 4 5│ │ │
│6 7 8│ │ │
└─────┴───────┴──┘ </ +. >/ *.To understand this you need to know that >. and <. are the min and max functions, and that in J three functions separated by spaces, f g h, constitutes a new function mathematically defined by (f g h)(x) = g(f(x), h(x)). An example is (+/ % #) which applied to a list gives the mean of the list. Here +/ gives the total, # gives the number of elements and % is the quotient.
Based on the examples, no, I cannot. It could be either of <: and >.
An alternative test for this hypothesis might have been using the language J, which is an array language based on APL and by the designer of APL but only using ASCII characters.
Well, at its core R is a Lisp (specifically, Scheme) but with a more traditional syntax (infixed operators, function calls, etc). And it’s fair to say the adoption of R has, indeed, been more widespread than that of Lisp.
1) The language is so frikkin massive. Common lisp is a huge language with hundreds and hundreds of built-in functions etc and the standard came very late in its evolution so there is a bunch of back compat cruft and junk that everyone has to live with. The object system is a whole epic journey in itself. You could probably kill or at least seriously injure someone with the impact if they were lying down and you dropped a copy of Guy Steele's excellent book[1] on them from a standing height.
2) The ecosystem is so fragmented. First you have Common Lisp, which isn't very common at all. Then you have all the vendor lisps. Then you have whether they have or don't have clos to contend with. Elisp is a lisp but is not common lisp and differs in some important ways that I don't quite remember. Then there's scheme, and guile scheme (which isn't quite the same) then clojure, etc etc.
3) That meant that the tooling was basically all simultaneously amazing and awful. As an example my uncle wrote a tcp/ip stack in lisp for the symbolics lisp machine[2] for a project when he worked at xerox. He told me in the late 80s about features in the symbolics debugger that just totally blew my mind and are only now available in IDEs for other languages, like being able to step backwards, alter variables, then step forward again, jump to any stack frame and just resume execution from there etc etc. On the other hand he had to write the TCP/IP stack himself because they didn't have one. I think that perfectly encapsulates the lisp experience for me around 2000 when I last used it - some things worked amazingly and were way better than anything else (eg I remember at the time the things you could do with serialization being just extraordinary compared to other languages) but a bunch of basic stuff was painful, janky or just completely missing.
4) Some of the concepts are very powerful but result in programs that are incredibly hard to understand. Macros, continuation passing, multiple dispatch.. etc etc. This puts a lot of people off because they just hit the learning cliff face-first and give up.
This is part of why python saw such wide adoption in my opinion. Not because it was in any sense the best language, but it was a very easy, practical choice for doing a bunch of things.
[1] https://www.cs.cmu.edu/Groups/AI/html/cltl/cltl2.html . Paul Graham (yes that Paul Graham) wrote a good lisp book also, although for me Steele is the one.
I've looked briefly at R and found the syntax and semantics to be less than stellar. Obviously there's going to be some bias in that sentiment due me not generally doing "array programming", but I don't believe the things that irked me were entirely as a result of that.
The more annoying stuff for R is entirely second hand. As far as I can tell R (or at least R studio) maintains implicit state between runs which means you can get to a position where the same code works on some runs, and then not on later runs. My friend was having to do a lot of bioinformatics processing (many of the libraries for this are in R) and was constantly fighting to have code she wrote to process the data or produce charts (publications in bioinformatics have an acceptance bias for "looks like it came from R" that is similar to what CS [used to?] have for gnu plot). But you could run the same scripts on the same input and have it fail where previously it worked. This is before you deal with inter-version compatibility problems which also seemed frequent.
What was irksome to me looking at a lot of the stuff that were doing is that it was fundamentally mostly basic scripting stuff you could do in other languages trivially (and more cleanly imo) but there were a bunch of functions (builtin or from libraries?) that did the work, but those functions weren't in R, so the claims that R was "necessary" seemed fairly bogus to me.
[1] https://en.wikipedia.org/wiki/APL_(programming_language)#/me...
> [R/RStudio] maintains implicit state between runs...
> This is before you deal with inter-version compatibility problems which also seemed frequent.
R is nicely cogent in syntax and largely "just works" once you accept its idiosyncrasies.
R is VERY VERY good at handling tabular data. Python can get kind of close with Pandas but IMO, it's still more awkward than base R data frames and way worse than data.table.
R also has a lot of built-ins geared for statistics and built by statisticians. If you're do it statistics there's value in not having to find a library or libraries that do that.
It's fun to translate terse APL to somewhat terse numpy. The result still can be very compact and you can parse it easily if you're used to looking at numpy:
s = arange(2, 50); p = outer(s, s).ravel(); sorted(set(s) - set(p)) /:~ s -. p [ p =: s*/s [ s=: 2+i.48
test =: (* = *. * +.) & ?
*./ test~ 1000 # 100 def d(x): N = arange(1, x + 1); return N[x % N == 0]
def m(x, n): return x * arange(1, n + 1)
def gcd(x, y): return max(set(d(x)) & set(d(y)))
def lcm(x, y): return min(set(m(x, y)) & set(m(y, x)))
def test(x, y): return gcd(x, y) * lcm(x, y) == x * y
all([test(x, y) for (x, y) in randint(1, 100, (1000, 2))]) # True
⎕io←0 ⍝ to match J
test ← (× = ∨ × ∧) ∘ ?
∧⌿ test⍨ 1000⍴100 : find-gcd ( nums -- gcd )
[ infimum ] [ supremum ] bi gcd nip ;
: max-wealth ( accounts -- n )
[ sum ] map-supremum ;
: which-max-wealth ( accounts -- i )
[ sum ] supremum-by* drop ;
primes-uptohttps://www.jsoftware.com/indexno.html
https://code.jsoftware.com/wiki/System/Installation <- install
https://code.jsoftware.com/wiki/Guides/Getting_Started <- help
> APL isn’t on the list of [Exercism] languages but I’ve seen it in codegolf (codegolf.stackexchange.com) solutions often enough that it seemed worth a look.
In other words, if you were to write the same text in Chinese and English, the Chinese version would take much less room, but native readers of each would take about as long to read them.
In CompSci terms, Chinese gets more done per instruction, but English has a much higher IPC.
I think we'd find the same between APL and Python. You could express ideas much more compactly in APL, but someone skilled in APL would take about as long to interpret a bit of code as a Pythonista would take to understand the Python equivalent.
If we take into account break of flow due scrolling up and down trough text, APL probably has an advantage after a few hundred Python-equivalent lines. One screen worth of APL holds many screens worth of Python.
This is folding several claims into each other, and they're not all true.
I would tend to associate "looks like noise" with writing where the units are not separated from each other. Arabic has this feature and this makes it appear more forbidding to the untrained.
Devanagari has it too: भारत गणराज्य
Ge'ez script doesn't: የኢትዮጵያ ፌዴራላዊ ዴሞክራሲያዊ ሪፐብሊክ
English does, if you choose to write in cursive.
By this standard that I just made up, kanji are clearly pretty far toward the "organized" end of the spectrum. An untrained person looking at them is going to be able to tell you what the main principles of the script are. And while I suspect that people complaining that a script "looks like noise" are mostly just saying that they can't read it, I also think that if people were forced to rank scripts based on how confusing they look, Arabic would be rated a lot more confusing than kanji.
(For a parallel to the above examples: 大日本帝國; الْخِلَافَة الْعَبَّاسِيَّة.)
This belies reality; in fact, reading kanji is so much more difficult that your claim that some people can do it as easily as we read Roman scripts is not defensible. (Whereas it's fine for Arabic.) The problem isn't in the appearance of the elements, it is that there are too many of them. This means that (a) learning to read kanji is a multi-year process; and (b) even those who are considered "fully educated" nevertheless need help in doing things like reading museum plaques or technical documents that use words which modern people are not expected to encounter in day-to-day life. If all you have is "full" training in the writing system, these words cannot be read at all, so there is a system of phonetic annotation for just this purpose. (For Japanese kanji, ruby; for Taiwan, zhuyin; for mainland China, pinyin.)
Kanji in specific has an additional problem that Chinese characters do not have, which is that the Japanese interpreted it as being primarily a logographic script rather than a phonetic script. To predict how a kanji is supposed to be pronounced, you need to see a particular use in a particular sentence and know the Japanese language. This is not true of Chinese characters - but it's not so much of a problem for your claim that "some people find this just as easy as we find reading Roman characters"; native speakers of Japanese don't have a problem with knowing Japanese. It does mean that it's hard to predict how personal names are supposed to be pronounced.
Furigana helps if one or more of the kanji used is rare (outside of the jōyō kanji), or when the word follows some common variation (guessing which is possible, but wastes the reader's time) or when the kanji spelling is arbitrarily assigned ("ateji"; impossible to guess).
E.g. suppose the reader sees 帰納. It is very likely they can read it correctly as kinō. The problem is not knowing what it means, that it refers to (mathematical) induction, or inductive reasoning. The dictionary lookup is trivial, though.
Basically, someone fluent in Japanese can read unfamiliar words like that most of the time. They will know a ton of words in which 帰 is ki and 納 is nō. In situations like that, it's not much different from someone encountering an unfamiliar word in English or Spanish text.
I think their claim is still defensible, and I think to make a fair argument you'd have to reduce the scope of kanji down to something the size of APL's vocabulary in any counterclaim.
This is plainly untrue; if you go to a museum and they're displaying an astrolabe, you can read the word "astrolabe" despite the fact that you've never learned it. In Japanese and Chinese, this is not true,† and the text describing the exhibit must separately inform you of how the word is pronounced.
Similarly, if you look over your Chinese medical record, it will be full of unfamiliar characters that you need to look up by shape if you want to read them. An English speaker may not know what a biopsy is, but they won't have trouble reading the word.
† It might be true; I have not looked into "astrolabe" specifically. But for various premodern devices, it is not true.
For this topic, I have in mind two exercises:
First, get people to nominate "experts" in a language, and assign them a straightforward task, and see who is actually faster using their native environments with code completion, language servers, etc. by systematically running these trials.
Second, get people untutored in each language of similar experience, randomly assign them to a language, give them a not short but not long period of time, then do a similar exercise as above.
The first has problems with assuring there is skill balance between the groups, but would I think tell you about how productive one can be in each language.
The second would have good causal inference properties, but probably not tell you about the downstream effects of learning something hard with great rewards (e.g., vim or emacs).
I'd bet there are marginal gains to APL (or vim) but probably nothing that matters in long-term productivity.
However, I do think people underrate how fun it can be to "get good" at things like vim or APL, and I think that has really positive knock-on effects.