J Language – New Website | Hacker News Reader

J Language – New Website

jsoftware.com

76 points4thaccount42 comments

I've long been a big fan of APL, since I learned it many years ago. I started by studying on the classic book by Gilman and Rose and trying to use the scarce freeware/shareware implementations available at the time. At the time it was hard or impossible to get anything working for free on a Linux computer.

Nowadays there is a great, free, and actively maintained implementation: GNU APL. That, combined with the widespread use of Unicode and the APL support in Linux/XOrg keyboard definitions, make it so much easier to get started than in the past.

I have used APL mostly for fun, that is, for learning the language itself, solving mathematical puzzles, and taking part in "code golf" games. I find the language exquisite, especially its programming model based on multi-dimensional arrays, its graphical symbols, and its grammar. But it is somewhat "impractical" to use nowadays.

J is a great successor (designed by the same mastermind behind APL) but I found it harder to learn (and to read) due to its ASCII "line-noise" look and to the prevalence of point-free or tacit function definitions, which I find inferior in readability.

I'm only now learning NumPy and TensorFlow, and I bet their authors were inspired by APL.

Does anybody have any benchmarks of J's performance against NumPy?

As for K, I wouldn't recommend picking it up. It's closed-source and documentation-less, meaning good look if you run into any trouble; its error messages are notoriously infuriating; it overloads its operators with a ton of meanings, depending on context, even more so than APL and J; and its fundamental programming paradigm is not based on multi-dimensional arrays.

On the other hand, writing an APL-like frontend for TensorFlow looks like a nice project!

Update: I ran a quick benchmark (large matrix multiplication) between GNU APL, J, and NumPy (standard pre-compiled linux amd64 packages, all running single-core.) Here are the results.

GNU APL (1.7)

    ⍴+.×⌿?2 3000 3000⍴1e10

    - size 3000: 65s, 9GiB RSS
      (crash on bigger sizes)

J (8.07)

    $(+/ .*)/?2 3000 3000$1e10

    - size 3000: 2s, 0.2GiB RSS
    - size 10000: 60s, 3.7GiB RSS
      (crash on bigger sizes)

NumPy (1.13.3, blas/lapack 3.7.1)

    import numpy
    a=numpy.random.randint(0, 1e10, (2,3000,3000))
    print((a[0,] @ a[1,]).shape)

    - size 3000: 25s, 0.2GiB RSS
    - size 10000: (>15m, I killed it) 2.3GiB RSS

Conclusions:

J's implementation is surprisingly performant! Easily beating NumPy on speed alone by a factor of 10 or more! (And I thought blas/lapack were already heavily optimized libraries!) J's memory usage is comparable to that of NumPy. GNU APL had the worst memory and cpu profile of them all.

Just thought I'd add the same benchmark on the latest Dyalog APL.

Dyalog APL/S-64 (17)

    ⍴+.×⌿?2 3000 3000⍴1e10

    - size 3000: 6.6s, 0.5GiB RSS
    - size 10000: 101s, 5.4GiB RSS

Conclusions:

J's implementation is still the fastest at this particular task (matrix multiplication of huge matrices on a single CPU thread--granted, not the most significant of benchmarks.) Dyalog APL comes close behind. GNU APL and NumPy lag much more behind that.

I'm trying this with Dyalog 16 (Mac OS X) and I'm getting WS FULL. Do you know why that might be?

BTW, what kind of machine are you testing on?

Have any data scientists here who were primarily working in R or Python, learned J? What did you think?

I love programming languages that teach you new ways of thinking about programming and I'm very curious about the APL family--thinking of learning J or q/kdb+.

I do most of my work in query languages on column-store databases, then some visualization and hypothesis testing on top. So I'm wondering if I could find J more expressive than R/Python + some SQL-like language.

I use Python a bunch and have used R in the past. I've dabbled in Julia and both APL & J. I know just enough to say that APL & J are much more expressive as long as Python doesn't have a fix_problem() method built-in. I believe ScottLocklin says something similar below, but he has a lot more experience in R and J than I.

Now q/kdb+ might be expensive, but I can only dream of doing analysis using it. SQL is great, but having the full power of K And Q-SQL would be pretty awesome.

Yeah it really seems to me like a highly expressive data analysis language with a tightly integrated column-store database system could have saved many, many millions of dollars and person-hours compared to Hadoop and Spark.

I know just enough to say that APL & J are much more expressive as long as Python doesn't have a fix_problem() method built-in.

This is pretty much the same impression I got, though I came from the other direction. I was kind of surprised how many StackOverflow posts ask about adapting a fix_problem() method to fix some very_similar_problem.

Yep. It's one reason why I prefer to write my own code where possible instead of using 3rd party libraries. When you need to change something, it is nice to not read tons of other code.

I don't know about J, but on occasion I have to deal with q/kdb+ and I hate it with passion. That system is pure crap. I'm convinced that the primary purpose of q/kdb+ is to extract as much money as possible on support from financial firms that happen to use it. Support contracts must run in millions of dollars if not more. The system is designed to be incomprehensible by human beings. All in a name of "performance." But is it really any faster than, say, Pandas/Numpy? (Numpy can automatically parallelize computations, whereas q/kdb is inherently single-threaded.) Q error messages drive me crazy. Basically, there are just a few error messages, like "'type" or "'rank" with no further explanation what caused an error. The Q language is full of idiosyncrasies you have to remember. I code in multiple languages including functional ones, but Q is completely illogical. There are arbitrary restrictions like number of local variables or the precise place you put a closing brace that ends a function. It's also impossible to find any help for it due to its abuse of special symbols, incomprehensible function names, and its very low popularity. There are C and Python interfaces for Q/kdb and they suck too. The C interface is actually on Github; you only need to take a single look at it to understand what I mean.

why does no programming language website ever include an example of the language? just a hello world or whatever would go a huge way. this annoys me so much.

To be fair, most people probably wouldn't be able to read a serious example of J, and hello world is just 'hello, world', so I think it's ok.

that just makes me more curious about what J looks like. and maybe a hello world isn't the best example for every language, but there should be some small example you can show.

I dug around and found Advent of Code 2015 solutions, in J: https://code.jsoftware.com/wiki/Essays/Advent_Of_Code

That should be plenty of examples for you.

In APL and J, hello world is just 'Hello, World' I think if I recall correctly.

What makes the languages (APL, J, K) impressive is that you can simplify what is sometimes pages of code into a single line of symbols (APL) or standard ASCII characters (J and K).

Keep in mind that these languages are the definition of line noise to most coders as having prior knowledge of things like C, Python...etc will do you zero good as the building blocks are completely different from a high level. They might have if-statements in the language, but they're hardly idiomatic. J emphasizes tacit programming which is kind of like a data flow model made up of function application. The languages are quite mathy in some ways, but still approachable.

I find them to be lots of fun, very interactive, thought provoking, and pretty cool. I've only written a little APL & J, but enjoyed how I built my command up from a few built-in primitives.

A problem to me involves reading other people's code which uses primitives I haven't learned yet, or common idioms (Ex: you can make your own average function with a few characters avg=.+/%# so no need to add it to the stdlib). The average function on the tryapl.org site looked a little funny to me...then I realized it worked for not only lists, but multi-dimemsional data structures of n dimensions like tables or lists of tables. I imagine it takes awhile to become proficient.

Surprisingly, both J and APL have some decent support for things such as graphics, but they are still small communities compared to languages we know and love like Python.

In short, give it a try. J has some built in documentation called "J labs" that is submitted by the community. Dyalog APL has a free version for non commercial use and the excellent tryapl.org. Try not to be turned off by the bizzare symbols in APL. They have a long history and are easy to enter with either the IDE or keyboard (especially if you buy one from Dyalog with the symbols on the keys). The inventor of APL got the Turing Award awhile back (later invented J). It was also a big IBM product back in the day and is still used by companies like Volvo.

I'm not discounting the language at all, by your comment here perhaps some application of a common mathematical function such as an average or the fibonacci sequence would be a good front page example.

That would make sense. Maybe someone on their mailing list will take note (I'm just a lurker). If you want to see something really cool, look at the APL video for Conway's Game of Life on YouTube. If I recall correctly, Jon Scholes made that video a few years ago. I don't know him, but he was supposed to be an APL wizard and an all around good guy from what I've read. I was sad to read that he passed away recently.

That was brilliant! Thank you for making me aware of that video.

It would be great if someone could translate that into J, then I might have a chance of understanding it properly.

There might be a version on Rosetta Code.

I tried, although I don't speak J so it's a rough transliteration. I couldn't do the last GUI animated part without more knowledge of J's tools, but it can compute multiple generations.

Code summary:

   r =. (3 3 $ i. 9) e. 1 2 3 4 7
   R =. _1 |. _2 |."1 (5 7 {. r)
   RR =. 15 35 {.  _10 _20 {. R
   life =. verb : '+./ > (1;y) *.each 3 4 =each +each/ +each/ 1 0 _1 |.each/  1 0 _1 |."1 each <y'
   gen =. dyad : 'life ^: y x'

The 20th generation:

    (RR gen 20) { '.X'

    ...................................
    ...................................
    ...................................
    ...................................
    .........X....XXX..................
    ........XX.....XX..................
    .......XX.....XX...................
    ........X.X...X.X..................
    .........XX....XXXX................
    ..........X...X..XX................
    ..............X.XX.................
    ..............XXX..................
    ...................................
    ...................................
    ...................................

Transcript of the whole talk with line by line translation here: https://dpaste.de/OoEM/raw (expires in 7 days).

Thank you! That was great. Thanks for putting in the effort.

I don't know that J/APL have a good front page example - if it does nothing, it's not interesting. If it does a lot, it's intimidating off-putting gibberish.

APL, rather than J, but here's a beautiful Prime Numbers up to 20 example:

    (~R∊(R∘.×R))/R←1↓⍳20

Reading right to left, iota 20 makes numbers 1 through 20 inclusive, 1 drop takes that list as an argument and removes the first element, R left arrow takes that shorter list as an argument and stores the numbers 2 through 20 into variable R.

In the nested parens, the outer product multiplication pattern (R___R) makes a multiplication table of the vector R against itself, and because we dropped 1 from the list we avoid all the 1xN and Nx1 pairs so it's a table of composite numbers. Then outer parens R epsilon compares each item in R with the matrix of composite numbers and generates a boolean result vector the same size and shape as R, but each input number has become 1 if it was found as a composite number, 0 if it wasn't (aka prime). Negate that bitmask so 1 indicates where a prime number was in the original R numbers, and use / to filter (reduce) the original list to pick those out.

Primes: Numbers 2..N which aren't members of all the products of 2..N.

No explicit loops, no if/else branching, no proliferation of names for working arrays, no need to wrap it in a function, give the function a name, document the function parameters and their assumptions. And even though it is N-squared runtime, it's short enough that an interpreter could pick out that pattern in code and implement it faster behind the scenes, if that was beneficial, without forcing the programmer to interrupt neat code to appease the machine beneath.

I can't APL or J, but I transliterate it to J as:

    (-. R e. ; R*/ R=: 2 }. i. 20) # R

J starts counting at 0 so this drops two numbers from the list. (Yes, J has p: for making Prime numbers).

> A problem to me involves reading other people's code which uses primitives I haven't learned yet,

I think this attributes to why people (not you I believe) find j/k write-only unless they are experienced. Most seem not to be able to imagine that because it is so terse, to compare reading an a4 sheet of k/j compares to reading 50 java files; if you think of it like that, and imagine those java files using some design patterns you did not know before, you can have at least some form of comparison. In that case I find k/j far easier and faster to read. I need to figure out what those few primitives do and how they work but that will take me less time than unravelling all the abstrzctions across all files to an extend I can start customizing code, while in k/j I am ready to jump in. ymmv ofcourse but I think too many people see the code and compare it to a lovely clean OO class in one file, or some a4 of Python and think wtf this is line noise!

I think your comment does an excellent job explaining the reality. I also agree that I could probably figure out what a well written page of APL/J/K does quicker than the equivalent 20 pages of OO indirection which I find quite irritating. With APL/J/K you can get to the macro level understanding of what is going on pretty quick and then drop down to the micro. With C, I can always read the Main() method, but it fractals out passed that. Aaron Hsu describes this pretty well in some HN posts and YouTube presentations.

This blog gives you a taste of J for mere mortals. After reading that I at least understood the gist behind J:

https://kukuruku.co/post/j-can-be-readable/

NB. You can get J for Apple iOS, free: https://itunes.apple.com/us/app/j701/id1255235993

and it's apparently got a new in-app keyboard to make it easier, since last time I looked.

J Is free for non-commercial use under the GPL. What does a commercial license from jsoftware cost?

Jd the J database is free for non commercial use and listed as $600 for a single user to use commercially on their site somewhere last I looked. They have fancier enterprise support contracts as well.

As for the language itself, it is free to use in commercial projects I think (someone please correct me if I'm wrong), but the source code to the J language is either GPL3 or you can pay. I think that means if you wanted to use the J source as part of a commercial project, you can pay or make the source available under the GPL3 license. I'm honestly not sure though and would appreciate some clarification.

For a second I thought this was the return of J Sharp https://en.m.wikipedia.org/wiki/J_Sharp

I like to play with algotrading bots and I'm really tempted to try out J or K in the near future. It has been in my backlog for a while :)

I suggest they update the logo since it now looks very out of place.

Does anyone use J for anything aside from experimentation (and math)? I know the same question is asked of everything that deviates from the mainstream, but the APL (and by extension J) world seems really far out from my mental model of computers.

I struggle to understand why you would want to use it instead of array orientated languages like R, Julia, Matlab or Python/numpy.

When is terseness a feature? It must make maintenance a nightmare.

Takes this with a grain of salt, since I haven't done much more than dabble with J, but I've heard that it (and APL) basically forces you to carefully read and fully understand the code, rather than skimming, which is generally what happens with "easier to read" languages.

And since it's so dense, you don't need a lot of lines of code to do most things.

Those are only partially array oriented languages. I use J for most of my personal projects because it's terse and highly productive. From empty emacs buffer to something that works; 5x faster than developing from raw R, and I'm bad at J and decent at R. Of course if R has a package which does what I need, it's easier to use that. There's also the matter that for medium big data problems, J can do plenty that R will never be able to do.

Terseness is a feature. It's also a good design in general, even though it breaks most people's brains.

When you say that J is faster than R, are you referring to using Jd versus dealing with R's read_csv, or are you just talking about the calculations in general for the same size dataset and just using vanilla native language constructs?

Not OP but I think he is referring to speed of development, not execution?

Thanks, that seems obvious now. I should not skim so much.

All of the languages you mentioned are watered-down and inferior versions of the programming models invented by APL and J.

By that I don't mean that APL/J are unilaterally better -- there are plenty of practical reasons to choose R/Matlab/numpy -- but they are unquestionably more beautiful and visionary. They hang together at a deep level in a way those languages do not. They are also much harder to learn, and much more fun.

On the readability of APL (terrific video): https://www.youtube.com/watch?v=v7Mt0GYHU9A

Alan Perlis on its beauty: https://www.jsoftware.com/papers/perlis78.htm

I write quite a lot in j/k ; unfortunately usually not for work related things but more and more so. I am far more productive in them than other languages for certain (mostly to do stocks and financials) fields. But I cannot expect my colleagues to go from C# to j ;)

Seems to be a lightweight shell that links to their wiki: https://code.jsoftware.com/wiki/Jd/Overview

Yea...sorry! I got excited and posted the link before realizing the majority of the site was still using the old wiki layout. Please forgive. I think the new landing page does look nicer and more modern.

Learn J here:

https://www.jsoftware.com/help/learning/contents.htm

Reply on news.ycombinator.com