NumPy another Iverson Ghost (2018)
analyzethedatanotthedrivel.org
analyzethedatanotthedrivel.org
The J is inscrutable! But the Python is intuitive. Analytics code I’m writing for work needs to not only operate on arrays, but also get the hell out of the way! Understanding and communicating numerical routines is hard enough without obfuscating basic procedures.
It's understandable after a time, like all else.
Do you remember how hard it was to memorize, or at least become familiar with, all of the API function names? Or with various language syntax, like generator comprehensions and decorators? Over the years, and with practice, it's much easier to remember the details. The same is true with the terse notations of the APL-like languages - once you learn them, they are not hard.
And there aren't all that many symbols. Some of what you see are composeable combinations of basic symbols.
Or, think of shorthand, which takes time to learn, but makes it possible to, for example, take dictation at a speed that longhand cannot manage. While I don't think most programming is limited by typing speed, I do think that for some sorts of exploratory research it is. For a related essay, see https://steve-yegge.blogspot.com/2008/09/programmings-dirtie... .
Finally, APL fans will (correctly, IMO) point out that (in)visibility is part of what makes something hard. That is, code which fits on the screen is easier to read than code which flows over several pages. Thus, the effort of learning a more concise language may pay off in the long-run.
(Note that I don't work with numeric arrays and have little experience even with NumPy - I've just read about this debate often.)
I'm not for overcryptic nor making people suffer.. I simply believe that people are underestimating how short syntax + repl + operators can be memorized and enjoyable.
learning the names of the greek and latin symbols (because then you can pronounce them in your head!)
I have to admit this was not obvious to me before I heard it a few days ago but it makes so much sense.
Code we can read (as in pronounce) seems less intimidating and if there are words we might be familiar with can help (and hurt) because we associate ideas with them.
Now I think this just underlines the beauty of APL, J, k and so on and why they both seem so un-approachable and loved by those who immersed themselves.
obligatory link to Notation as a Tool of Thought
https://www.eecg.utoronto.ca/~jzhu/csc326/readings/iverson.p...
if you are not familiar with the symbols and their meaning they might as well come from a dusty old book hidden away in the depths of a dungeon guarded by tiny dragons
yet if you are familiar with them they are not intimidating nor inscrutable (anymore)
J is an abomination. I have written about this before on HN and won't repeat myself too much here.
J came out of Iverson's failure to understand that the hardware limitations imposed by personal computers of the time would go away as platforms evolved. While he understood the value of notation (hence the famous paper and presentation I got to see live), he succumbed to wanting to come up with a way to make APL accessible to all those IBM PC users without having to resort to hardware tricks nobody would believe today (install a custom character ROM on your graphics card with a toggle switch on the case to switch between APL and standard character sets, etc.). The result was J, a transliteration of APL symbols into combinations of standard ASCII characters found on any keyboard of the time. The result was a complete loss of one of the most valuable aspects of APL, and, as I said above, an abomination.
I think that I can say that it is because of this that I was very aware of what was going on with APL in a commercial sense. It was very difficult to evangelize the language when, in the 80's, the friction points to adoption were many --the character set being one of them. It took a major effort on my part to convince a biotech company of the era to adopt APL for analysis of human DNA sequencing data.
All of this to say I believe I fully understand the decision to remove one of the impediments by discarding custom symbols and coming up with whatever could replace them that was present in computers of that era. This meant ASCII characters that could be typed by anyone on any machine. Almost anyone desperate enough to make the language gain a foothold in the market would have considered such a solution.
That was the driving force behind J. Nothing else. I never heard Iverson say "We moved away from notation because this is better". Never. Because it wasn't. It took a wrecking ball to what he had been teaching and talking about for decades, the foundation of his career, really. They needed market (and financial) success and made the wrong choice.
It wasn't long until custom character sets were a non-issue. Anyone who stopped to think about this a bit would have understood that transition was imminent. APL had the same problem that, say, Japanese had. IBM PC's required altered character ROMs and keyboards to be able to use Japanese. If you had to work with both English and Japanese you likely ended-up with a solution similar to early APL, a modified graphics card with a toggle switch mounted in front of the computer to change character sets.
And so the pressure to build computers that could do non-ASCII character sets with ease was there and it was very real. Remember that IBM never thought the PC would go anywhere. The machines only became more than the sum of their parts once mass adoption took place and clones got faster, cheaper and better.
Had Iverson stuck with APL as it was he would have been able to see it flourish on new more capable hardware. By doing what he did with J he likely bifurcated (or more, because there were other pure APL factions already, like IBM APL2) the community and likely contributed to the language languishing.
It only makes me angry in the sense that all of this has caused a situation where modern software engineers will likely never experience what notation can bring to the domain. APL used to be taught at a number of universities. I don't know if any even bothers to mention it yet.
First, I think that J was indeed an improvement over APL. There were several irregularities in APL that were corrected in J (eg. indexing operations) and some new features that allowed new possibilities (eg: function trains and tacit definitions). This is discussed in more detain in "A Personal View of APL" (Iverson, 1991).
And, second, there are some advantages of using ASCII. Although it does not require modifying your hardware any more, using a special character set still is inconvenient. The capacity to write a J expression in any mail program, for example, is important. Being able to use any keyboard (including mobile phone keyboard apps) is important. Easily adding J code to any paper using any text editor without needing special fonts is important. I do not think these advantages are enough to compensate for the loss of expressiveness, but they are not negligible.
All that said, I understand (and partly share) your frustration.
In my experience it is difficult for non-APL-ers to truly understand just how bad it is to lose the symbols. The only approximation I have been able to find that clicks with some people is to compare it to musical notation. Anyone who reads musical notation understands the value of symbols very well.
I can't relate to the idea of writing code on a mobile phone. I have never done anything that useful, even with my iPhone X. I barely want to reply to emails with it.
I wrote a paper for an APL conference back in the dark ages. Printing the paper with APL characters as well as normal text entailed a double printing process where I had to change the ball on the IBM printer between APL and standard characters. I actually wrote a custom print program (in APL, of course) to manage the entire process and, of course, wrote the paper in APL using a custom text editor I wrote as well (Wordstar couldn't handle the task). Think of it as Jupyter Notebook (but not as sophisticated) for APL. Ah, the good old days when you had to create all the tools you needed yourself!
Another way to look at it is: Had Iverson focused on advancing APL to the next stage, computer systems would have caught-up with the language. My guess is that the economic viability of his business forced his hand. I can understand this, of course.
# numpy - matrix product np.dot(a, b)
NB. J - matrix product a +/ . b
He's also right about Numpy. FWIIW Wes McKinney was also inspired by J when he wrote Pandas and Arrow.
Do you have a source for this? I always assumed it was based off of R, just from the syntax.
Anyway, ask him; he'll tell you!
data.tables in R has some strong resemblances to J notation as well; might be one of those inevitable things (Greenspun's 10th applied to array programming/APL), but I never talked to the author.
Thanks for your kind words "The author of this, who I know a little, maintains what I think is one of the most important pieces of code nobody ever talks about; JOD[0]."
If I was selling JOD, I would seek to use your quote in promotional material. However, I am not selling JOD; it's open source for all to use - see:
https://github.com/bakerjd99/jod
It's even going on ice in the Arctic Code Vault.
https://analyzethedatanotthedrivel.org/2020/08/17/jod-goes-i...
Thanks again.
> At the end of my programming day, I want to look on something that is beautiful. I don’t particularly care about how useful a chunk of code is or how much money it might make, or what silly little business problem it solves. If the damn code is ugly I don’t want to see it.
I can't relate to this at all.
> b = np.arange(12).reshape(3,4)
> b.cumsum(axis=1)
or, would you prefer the following:
> b =. 3 4 $ i. 12
> +/\"0 1 b
Yeah, no thanks ...
a) a*(b+c) = a*b + a*c
b) equal(multiply(a, add(b, c)), add(multiply(a, b), multiply(a, c)))I have no idea what the first line does in either of these examples. And the fact that NumPy uses an English word, axis, as a named argument means nothing at all to me
I think it is an exaggeration to say that the NumPy is any clearer to the uninitiated than the J. Once one has learnt J then surely it is as easy to read as the NumPy? The NumPy just looks easier because the syntax is more familiar.
I prefer ergonomics and having my colleagues grok what a line does: eg. 'reshape' calls will change the shape of your array.
Bless you if you're in an environment where you can always pick the tools that work for you. But I'm never going to convince my (java, scala, python, R) adept colleagues that they need J in their life.
Here lie my gripes with the linked article, it's just not such a good sales pitch.
In that case, you owe it to yourself as a programmer to go and get familiar with some APL-isms, for they are for numerical arrays what regexes are for strings and are rapidly becoming ubiquitous[0]. I wowed my boss once because I was able to mold an array into the shape we needed it in a few seconds... in Mathematica, a language I had never used before that moment, because I knew what to search the documentation for. It doesn't matter where you get it from, just pick a language with APL-isms - I recommend NumPy if you like popular stuff, and Q'Nial if you like esolangs - and work through Project Euler without writing any loops. It'll be fun!
And of course, if you're unfamiliar with the core concepts, then naturally both syntaxes will look equally opaque. That doesn't mean they are though. Once you understand what they are trying to express, then you will be in a better position to judge.
[0] In Julia:
b = reshape(1:12,(3,4))
cumsum(b,dims=2) #Julia is 1-indexed b←3 4⍴⍳12
+⌿b ⍝ maybe, don't know if numpy/J starts at 0 or 1 -- assuming 0
can you share a bit about how you came to use J?
We primarily focus on algorithms that require rapid sequential iteration rather than parallelism. However: 1) we run an enhanced version of J that is implicitly parallel on CPUs, and 2) it is possible to call GPUs (both CUDA and openCL) through J (e.g. j-fire). We do not call GPUs or other accelerators currently, though we may do so in the future.
Our goal with Monument is to create an Excel-like substitute for ML / AI. In comparing across languages, we selected J as "the right level of abstraction" for ML / AI because we wanted to write fast-running code, quickly.
J runs on virtually all CPUs and OSs, and it is highly tuned for minimized cache misses and mis-predicted branches. It is well-documented, well-supported, and the community is highly supportive.
Like Python, J makes it possible to run highly-optimized code without a great deal of boilerplate in an interactive development environment. Unlike Python, J is composed of reusable functions that readily express math. Our dev productivity is extremely high, and often we're able to engage with mathematicians who have little or no programming background.
Our team comprises polyglot programmers, and I strongly avoid "the language wars," but overall we have found J to be well-suited to our needs.
People who do things are recognizing this.
...