A look at the J language: the fine line between genius and insanity (2012)
scottlocklin.wordpress.com
scottlocklin.wordpress.com
I think it's an idea whose time has come in some ways: hardware is more array-like than ever. J has evolved a lot from 806 to the present 902 beta; lots of performance increases from moving away from the single core paradigm (where it excelled anyway). Lots more to unlock there as well, and some interesting J-based startups brewing[2]. I also think it's one of the best user communities I know of.
Oh yeah, Art's new language Shakti[3] should probably be mentioned. Lots of people excited about this.
[0]https://code.jsoftware.com/wiki/Jd/Overview
[1]http://lush.sourceforge.net/
[2] https://www.monument.ai/ for example
Questions are answered in no time, the folks are very friendly and treat everyone on eye level. A great role model for any programming language / software communities!
Recently I have been trying to learn Futhark which is a ML like language generating parallel code (E.g. OpenCL or CUDA) from pure functional code with nested array structures. I find this exciting because when using the GPU with a thin layer like OpenCL directly, you have to write kernels from scratch without much abstraction possible. Higher level APIs like PyTorch or Jax give you easy ways to flexibly structure the computation but you just have to hope for kernel fusion. In Futhark this is in principle explicit and well handled.
ISPC is another example of a C like language where SIMD style parallelism is unambiguous while staying relatively high level.
I do wish we had some better cross-platform solutions though, instead of cementing nVidia's hegemony. There's no technical reason why GPGPU should be any more difficult on AMD hardware.
Then look no further than GPUCompiler.jl. Tim factored out all the platform agnostic parts of CUDA.jl (successor of CuArrays.jl and CudaNative.jl) into GPUCompiler.jl and that is now being used by the AMDGPU.jl package. It works today!
That just takes developer time to fix. Currently it’s basically just a one-man show.
Now, all vendors seem to have dropped support for OpenCL, though it will live on nicely in PortableCL, which implements it atop LLVM, allowing it to target anything LLVM does, which finally goes further to achieve the open-ness the original standard aspired to, than the vendors themselves at the time.
https://github.com/Pascal-J/Jfire
Lush: it just felt comfy for a data science workflow. No DB adapters, or even real csv readers, but you could do whatever you needed in the interpreted language and go pretty fast with the compiled bits. Still feels like the future.
Monument runs on an extended version of J that is implicitly parallel, which enables devs to focus on math rather than optimizing C code.
The incomprehensible programs just seem to hint that there is some sort of maximal information density where every keystroke provides maximal productivity.
There's also something artistically appealing about starting at a program and making a deliberate one character change that makes all the difference ... although on the other hand, that's kind of scary too.
Here's a video that really illustrates the power of this paradigm: https://www.youtube.com/watch?v=a9xAKttWgP4
[1] - I learned (in this approximate order): C, Java, Lua, C#, Ruby, Python, Lisp, Forth, Prolog, Clojure, OCaml, Haskell, Erlang, F#, R, Racket, and Rust. I also took a look at languages like: ATS, Cyclone, C++, Idris, Agda, D, and Perl 6. Of all those languages it really feels like APL/J/K are the strange ones.
Amazing game of life code!
APL has a kind of mystique, but all it really is is a bunch of NumPy functions with a quirky DSL (of course, APL came first and heavily inspired NumPy). But we've quietly been doing the APL experiment in the background, by pressuring programmers to avoid pure Python in favor of NumPy functions for speed, and the results are in - you can indeed push the array paradigm very far. But not quite far enough to be general and efficient.
The jury is still out on whether the quirky DSL is worth anything. My suspicion is that having names for array operators is very valuable indeed, but dedicated symbols for them much less so, and cryptic ASCII digraphs are barking completely up the wrong tree. But if you really want the weird syntax - well, it's not exactly a difficult language to parse, and Unicode is a thing now. If the notation is worthwhile, there's no real reason why any language couldn't have APL built-in as a DSL, just like regex.
Cryptic symbols, I agree, less so. If your language doesn't arbitrarily restrict identifiers then you can just define the APL operators as functions and write code that looks like APL - I tried doing this in Scala once, and it worked ok. But it's hard enough to convince people to take the time to learn what "map" or "filter" or "<* " does, never mind anything further in that direction.
I like compilers, systems programming, distributed systems... but math? statistics? I'm admittedly terrible at that and find it really uninteresting. It doesn't help that most literature on J tries to teach its language concepts showing you how to solve math problems. I have no need for a fancy calculator.
For what it's worth, I like K and Klong and found them very approachable. Basically because you can get away by writing Scheme with m-expressions.
And a variety of spreadsheets, from 2 to 100 lines.
They were sort-of-ok for API access, but Shakti’s FFI is actually incredibly straightforward - comparable to writing a C extern definition only much terser (‘cause it’s Arthur...)
q especially looks mostly like a normal procedural language, and can be used like one too.
:If databaseResult.Rows.Count > 5
:AndIf warnOnLargeData
:For Row :In databaseResult.Rows
logToFile 'Careful, that's a lot of data, e.g. ', Row[1]
:EndFor
:EndIf
⎕USING←'System' 'System.Collections'
arr←⎕NEW ArrayList
tmp←arr.Add ⊂'Test'
tmp←arr.Add ⊂'Hello'
tmp←arr.Add ⊂'World'
arr[0 1 2]
┌────┬─────┬─────┐
│Test│Hello│World│
└────┴─────┴─────┘
arr.RemoveAt 0
arr[0 1]
┌─────┬─────┐
│Hello│World│
└─────┴─────┘
The core of APL symbols more or less /are/ an embedded language in the wider APL, just that wider APL is not showy for codegolf and is not unusual and pretty symbols, and you can't use it /without/ the symbols.
[1] See the examples shipped in the install folder like "C:\Program Files\Dyalog\Dyalog APL-64 18.0 Unicode\Samples\asp.net\webservices\"
[2] I expect not /everything/ C# can; C# is the first-class .Net language, but if you have things written in C# you can use them.
I'd be more interested in a J/K demonstration that plays to the language's strengths. How about live-commentating e-sports?
* Player gets first blood. Commentator executes search query in J across DB. *
"We've seen $PLAYER_NAME contribute to 65% of first blood situations across their last 10 matches."
It originally seemed like magic to me, but really - there is no magic. It is simpler in K and J to process more things, in more stages, but with completely predictable and very pipelineable operations. The J/K program often does 2 or 4 times as many operations, but has virtually everything prefetched to L1 before it’s needed so it runs as fast or even faster.
CPUs are constantly converging towards the K model (GPUs were always there).
As A. Hsu say, a java method name is long enough to encode a complete algorithm. the drag of having to maintain MLoC of boilerplate with all the opportunities for bad naming, bad style, NPE and such is so huge.
Fwir, Kdb+ is written in k which is sorta APL, and then there's q for writing kdb+ programs which is easierish. But still an adventure.
https://aplwiki.com/wiki/Array_model#:~:text=In%20APL%20it%2....
For example, C# and Java implement the same paradigm , they are different, many argue which provide a better set of feature, but I would not call the differences fundamental
C#, F#, implement completely different paradigms
F# and Clojure, same paradigm, but have fundamental differences, F# being static and come from the ML family , Clojure dynamic and come from the Lisp family
So APL to J, is more like what to what, C# to Java, or F# to Clojure ??
Both J and APL work on the same type of things (principally, arrays and nested arrays) with functions being applied over the arrays or across multiple arrays. And both languages encourage a point-free or tacit style of programming (though J, from my limited experience with both, seems to push this a bit further sometimes). For a functional paradigm example it's probably more like the difference between SML and Ocaml, or between two similar lisps like Common Lisp and Emacs Lisp. There are clear differences in focus in the two language designs, but much more in common (again, setting aside the choice of characters, you can find an equivalent for most APL symbols in J and vice versa without needing to create too many new definitions).
K on the other hand, is really different! K doesn't have true multidimensional arrays, but lists of vectors. So it's ideal for its use case: finance (1d/2d numeric tables).
Interestingly enough you can fairly easily program your own language (written in q/k) on top of that, and set the interpreter to evaluate that language by default. Kx offers a Python integration layer called EmbedPy which (among other things) exposes a Python evaluator as the "p" language in kdb+.
Any truth to this?
I suppose it might be like telling someone, "you know conditionals and loops, now you know how to program!"
Keep in mind that it's not just about the verbs (operators) in isolation, it's also about sentence structure. While you can get quite far in the language without learning how to write a fork or a hook (two related ways of combining verbs), it will be near impossible to read someone else's code until you practice these forms yourself.
Having said that, if you're anything like me you will have the NuVoc page [1] open basically all the time while writing or reading J. It's hard to keep them all straight, esp. the verbs that you don't use very often.
Then combine that with more explicit memorization/recall cards like I use for Spanish. "In Common Lisp what non-destructive function filters a sequence based on a predicate?" "remove-if".
I think this should work well as a way to develop an understanding when paired with books/tutorials on the language and its libraries.
I'm basing my experiment on what I learned using Anki for language learning. Much like you said, you need larger statements and to develop an understanding of them. For Spanish I have some loose writing/speaking prompts and some reading prompts (not copying the text, but a prompt to go to today's newspapers and read some articles and explain them to my wife or to read some paragraphs from a book). How well I can complete these without needing a dictionary or to ask for assistance determines how I mark the cards. It seems to work decently, they come up almost like pop quizzes in school classes since the spaced repetition system means I don't see the same prompts each day. I think it should work to have project prompts for programming languages in the same manner.
Usr Pri JfC LJ Phr Dic Voc !:
All these are interesting; some primers, some the old version of the dictionary (I think it's depreciated for now). I end up !: a lot for the foreigns.
Understanding how you can build that iteration space as reified data gives you a lot more expressive power than you get from remembering that `|.` means "rotate." It's easy to look up which symbol corresponds to a chosen operation (and you'll memorize the ones you use a lot anyhow).
Slight tangent, Supercollier is one of the best langs I've ever played with. Smalltalk + J. Totally blew my mind when I first wrapped my brain around it!
http://jsoftware.2058.n7.nabble.com/Spatial-trees-td25879.ht...
Link in article was dead
Plug: I wrote an open source version of q/kdb that runs on the JVM: https://github.com/timestored/jq
You can give it a try online in the browser here: http://www.timestored.com/jq/
Also, Java's "File" object is -- for the most part -- by its nature cross-platform, which is why the forward-slash version right below your change already worked as expected: https://github.com/timestored/jq/commit/4b94ce214452cf2b2fb8...
I don't claim to understand J, I just added it to the site.
Good introduction. Its always nice to see a "community review".
Please take a look at https://ciumei.ca/blog/2020-07-25/j.html and do let me know if the code is at all cryptic.
2018 https://news.ycombinator.com/item?id=16393873
Discussed at the time: https://news.ycombinator.com/item?id=4543038
Moreover, it's entirely possible to write regular procedural code in q and k, so it's easier to transition to the array language mindset.
However, the few real world projects that I've seen in these languages won't use any kind of DSLs and the argument usually is "for array-based language programmers, this is actually very clear and concise". I need to improve my J to verify that :)