A look at the J language: the fine line between genius and insanity (2012) | Hacker News Reader

A look at the J language: the fine line between genius and insanity (2012)

scottlocklin.wordpress.com

166 pointsjxub46 comments

This is a really old blog of mine. I was so influenced by J I tried to do a startup in 2015/2016 making a more legible APL/TSDB. At present I'm more or less running a small cryptocurrency HF in J and using the J database engine jd.

Did the story of that startup make it into your blog? Building a business to sell a new language always looked to me like too much of an uphill battle in a world where programmers generally expect the tools of their craft to be available for free.

Enterprise sales is an uphill battle. FWIIW I happily pay for my Jd license.

Hey Scott...didn't know u were on HN. I've enjoyed most of your writing tremendously and am sad Kerf didn't work for you and your business partner. I hope you find time to write more about J in the future. Your comment on not wanting to write another single For-Loop has really resonated with me. I've played with J/APL/CommonLisp/Clojure and nearly another dozen languages, but Python has traditionally been my daily driver. Recently though I've been using Mathematica and have been very impressed with the terseness. There are zillions of functions that are very easy to combine to get things done. It is kind of like a Lisp on the backend as most functions consume and create lists. I've found having primitives for charting, graph-theory, numerical solvers, and OS level stuff to be very enjoyable with no explicit looping. They also have plenty of primitives for things like FFT and Linear Algebra...the price is quite high though.

I guess I use R for the sorts of things I might use Mathematica for. Mathematica is kind of the opposite of J; with J you only have a few primitives. Mathematica is a swiss army knife for data and math. Python is fine; I was an early adopter, I just don't have any reason to prefer it to R (package quality perhaps).

Kerf was a tough slog, but Kevin and I landed on our feet. We'll do something with the code one of these days.

Agreed on that front. I've used R & Mathematica as a swiss-army-knife for those things as well, although the stats stuff I do is a lot more basic than the stuff you do professionally. I'd like to get more proficient at J & APL, but that is a long road I've found.

Glad to hear y'all found something after Kerf. I would've guessed there would be some market for a tsdb cheaper than kdb+ (what I assume y'all were trying to do). Your blog post mentioning overhyped tech really gave me a chuckle. In college I started to focus on nanotechnology before going to a career fair and talking to some companies. Thousands of students getting degrees focused on that and there are only a handful of startups hiring researchers and only so many grad-student type jobs. I had a physics professor tell me a niche field comes up like that every decade which gets a lot of funding, but essentially under delivers and dies out. I think there is some benefit to nano materials like high strength cables, but not the miracle sauce that is getting preached everywhere.

Material science is real; nanotech, not so much. Glad you found your way out of that dead end. I have a friend with a Ph.D. in nanotech. He is a pretty good GPU programmer.

Programmers continue to discover and re-discover APL and array-oriented programming.

If someone is serious about array programming/APL and wants to make a lucrative career in it, study KDB+/Q/K. There's a small but active community around them.

Just to relate an anecdote: I used to be a fairly active KDB+ programmer in finance but left both finance and programming ~8 years ago. Just yesterday, I got an email from my friend who's building a statarb fund around cryptocurrencies. They were going to use KDB+ and offered me a job for "300-500k base with incentives"...

My friends clearly don't know that I now work in sales & marketing :D

Aside: Roger Hui, the creator of J, is a programming sibling of Arthur Whitney, the creator of K/Q and the founder of www.kx.com

> finance

Did you think finance programming industry (specifically around the KDB+/Q/K niche) fostered a "bro" culture (think wolf of wall street)? It always seemed like an interesting field, but every once in a while I hear discouraging stories. Curious about your experience with it.

I think kdb / q developers are possibly as far from wolf of wall street as it's possible to be.

One guy in London who worked in kdb was a huge arse, the tens of others I've encountered have all been lovely. It's an insane little language, though I'm only a basic user.

I recall there was an article about Arthur Whitney on HN a while back. It contained a small C program that he wrote for a college assignment, which I preserved here: https://gist.github.com/lukechampine/e759098e717e42bc11c5

Looking at this "C" code, it's pretty clear that he was destined to be an APL-style programmer :)

He learned from Iverson.

I understand the advantages of array-oriented programming, but why the J/APL syntax? There are plenty of column-oriented databases available. https://en.wikipedia.org/wiki/List_of_column-oriented_DBMSes

Also related is range-based programming. https://wiki.dlang.org/Component_programming_with_ranges

    return
        // Start by generating all dates for the given year
        datesInYear(year)
        // Group them by month
        .byMonth()
        // Group the months into horizontal rows
        .chunks(monthsPerRow)
        // Format each row
        .map!(r =>
                // By formatting each month
                r.formatMonths()
                 // Storing each month's formatting in a row buffer
                 .array()
                 // Horizontally pasting each respective month's lines together
                 .pasteBlocks(colSpacing)
                 .join("\n"))
        // Insert a blank line between each row
        .join("\n\n");

Much more readable than something like "(~R∊R∘.×R)/R←1↓ιR".

okay but in J thats:

  require 'dates general/misc/format'
  4 3$ calendar 2018

4 3$ just re$hapes it from a big as line to a 4x3 calendar. If you wanted to print it to a file you'd want to convert it to a string of course so:

  ": 4 3$calendar 2018

Is it more readable? I'm even less familiar with D than APL, and that D code only looks more readable to me right now because it's mostly comments.

Wouldn't adding 8 lines of comments help newcomers to APL, too? This seems like a rather unfair contest.

Familiarity counts for a lot. 25 years ago when my primary language was 6502 assembly, I would have said 20 pages of assembly code look more readable than either of these.

In the long run, though, in every field I've worked in, being able to turn 20 lines of code into 20 characters has turned out to be a huge benefit. Once you familiarize yourself with the vocabulary, you can operate at a higher level. Nobody fears concise names and symbols when it's in a context they understand. I think even the most APL-phobic would admit that "a←b+c" is more readable than "assign(a,sum(b,c))".

Everything you know how to read, and peppered with comments you know how to read is more readable than everything you don't, without such helper devices.

What’s the catch though? If learning KDB+/Q/K nets you a nice comp why not everyone and their dog are learning it yet?

You don't just have to learn it (that's the easy part); you actually have to grok it, and that's the hard part.

The piece of code |/0(0|+)\ (all 9 characters of it) efficiently computes the maximum subarray sum[0] of an array using Kadane's algorithm. You'll either have to learn K or trust me on that.

While it is possible to break it down to multiple parts and document each, it is idiomatic to just use it and document the whole line. Or not document it at all, because experienced K people know that already. Why call a function "average" (7 chars) when an implementation (+/x)%#x is only 6? Furthermore, you know from reading it what the average of a zero length list is (NaN). Do you know what a function called "average" would return in this case?

I would say the answer to your question is that K is very different than other programming languages, and requires a different mindset. Somehow, attempts to give it a more mainstream face (e.g. article author's "Kerf" project) do not seem to take off.

[0] https://en.wikipedia.org/wiki/Maximum_subarray_problem

Yes, I was wondering the same thing. Some guesses: some people may not be comfortable with the somewhat un-conventional syntax / semantics (e.g. no operator precedence, just right to left or as they call it, left of right :) (I'm fine with it myself), or might think you can only get jobs using kdb+/q/k in big cities like New York / London, and may not want to move there. Possibly low market demand (compared to mainstream languages), although high comp, may be another reason. Also people may think you need finance knowledge too, and may not have it. Those are some guesses, that's all. Interested to hear if anyone else has some ideas on this.

Edit: Just saw beagle3's comment after posting mine. Some interesting points there. Still wonder about the non-tech factors though, as listed in above part of my comment.

I don't know either party but I'd hazard a guess that they're not really offering 500k for a K developer.

They're more likely offering 500k for someone who has a lot of experience managing financial tick data in a KDB environment.

The catch will be that not everyone who has read the KDB for Dummies book (OK, I made that up) will be able to walk into a stat arb hedge fund and add 500k of value.

Heres the .q file I use to read HN. Hobbyist programmer; corrections/improvements welcome. Better yet, how would I do this in J?

cat 1.q

   \c 2000 200
   k)-23!t:select from .:`:t.kdb
   k).Q.fs[{`t insert +:`url`title`user`item!("SSSS";",")0:x }] `:hn.csv 
   k)-23!t:?:select from t 
   `:t.kdb set t 
   \\

As for how to get hn.csv, I use shell scripts to download all the pages of HN html (in accordance with robots.txt specified delay). Then I use a simple lexer (made with flex) to transform the html to hn.csv with the above selected columns.

After some years, the file hn.csv can grow quite large, larger than available memory.

To read and select articles from t.kdb, I use less(1) and some small shell scripts.

This lists all the articles, starting after a given offset, or zero by default:

      echo -e '\\c 2000 2000\nk)select i,title from .:`:t.kdb where i>'${1-0}'\n' \
      |exec q > 1;
      exec less 1;

Then, to select and read an article or its comments I use the ":!cmd" feature in less(1) to invoke an appropriate shell script (actually, an execline script). Quit, and I am returned to less.

Have seen lots of systems for reading HN made by HN readers, using various databases I guess. This is mine, using kdb+.

Neat, but why don't you use lynx?

Let me guess... You work at a bulge bracket

how much did you pay for your kdb+?

/After/s/hn.csv/t.kdb/

Shameless plug but I recently picked J back up as a hobby and it's been super fun. You can read some of my findings here: http://thatjdanisso.cool/j-fibonacci/ http://thatjdanisso.cool/j-pascal/

I encourage you to give this language a shot - it's got some really cool ideas.

These are cool, but how practical would it be to produce a blog post akin to, "Here is a web app running J in the back end"?

Somewhat tangential, but some concepts and operators from J have been integrated into the audio programming language sclang for Supercollider [1]. Pretty neat! (I'm currently learning SC [2]).

[1] http://doc.sccode.org/Guides/J-concepts-in-SC.html

[2] https://github.com/Engid/superfun/blob/master/first-impressi...

I've read a bit about APLs and I've seen them popping up in HN a few times. They do seem incredibly awesome and the products around it (K/KDB+ etc) amazing.

However, in case I ever felt like learning one of these, I'd always wanted to know: the syntax crazyness, "a la brainfuck", is it a feature? a bug? Is it really needed? Can't they make an APL "look like Python" but using the same concepts, etc? Or is it "a thing" / something ultra-core to these languages, to be extremely terse / cryptic and intensely dense (each line packs a ton of punch!)?

e.g. just read http://www.jsoftware.com/help/dictionary/didot.htm and some of the stuff in there can be translated quite well:

i. 5 --> list(range(0, 5))

i. _5 --> list(range(0, 5))[::-1] or list(range(4, -1, -1))

i. 2 _5 --> [list(0 + j, 5 + j)[::-1] for j in range(0, 10, 5)]

P.S. any pointers to content / books about the philosophy and ideas behind these programming languages are welcome! :D

The density is very much a feature. Some previous discussion from an APL'er about this:

https://news.ycombinator.com/item?id=13565743

https://news.ycombinator.com/item?id=13797797

> Can't they make an APL "look like Python" but using the same concepts, etc?

Perl 6 has some APLish array-oriented features in something arguably closer to what is common syntactically. Of course, it's not an APL with alternative syntax, but an extremely multiparadigm language with that as one of many influences.

People make fun of Perl6 for adding Unicode operators, variables and such like having them would just be totally insane when APL has been doing that since the 60's and it works well lol.

Certain things translate, but not everything. In J when almost everything is an array and you're using points free form, you can pretty much combine everything without having to build classes...etc. Those python examples are much longer even for simple stuff. As far as books go, Dyalog APL has "Mastering Dyalog APL" for free and although Dyalog costs money, a hobby license is free. There is a guy who writes programming languages and books for a living. He recently wrote an array language called "Klong" and a book on it. J has several books. There's not a whole lot on the philosophy, but using it teaches you what you need. It looks like line noise at first, but you start to recognize idioms and phrases. Aaron Hsu did two HackerNews discussions on here and a YouTube broadcast on his "co-dfns" GPU compiler for Dyalog APL. That was pretty amazing. He talks about trying to write it in Scheme, Java,etc...but APL lets him put it all on a few pages to where he sees everything at once. At his level, abstractions aren't necessary.

I did a cursory Google search for Klong after your mention. The author of the language (and its book) posts his writings to t3x.org

Thank you for the brief mention -- I've been perusing his website and writings and am enamored. There's something about niche, obscure tools (and their authors) that give me some profound joy.

I was thoroughly impressed as well. It would take me years to read all his stuff. Klong us actually written in his other language. That is impressive.

I prefer J for this specific examples. The python versions rely on 'non-obvious' syntax with the ::, and argument ordering to the range function.

> The python versions rely on 'non-obvious' syntax

...really?

Look, I like APL and J much more than the next guy, but I can't find a single angle from which that is a reasonable argument.

For the ideas and philosophy behind J have a look at the paper [Notation as a Tool of Thought](http://www.eecg.toronto.edu/%7Ejzhu/csc326/readings/iverson....) by Kenneth Iverson. He originally presented this paper as his Turing Award Lecture in 1979.

As someone who has written APL professionally, Numpy and Nd4j are much better choices to spend your time on. It might not be exactly the same thing, but thats the point. I only miss the linear algebra from APL and there are very awesome libraries that do better.

similar background (started my career in APL, only language i've ever used that had its own keyboard)

also NumPy, and Nd4j via scala bindings

of course NumPy and Nd4j are libaries

i strongly recommend Julia as an array-oriented programming language

particularly because it has quite a few other features to recommend it

Julia follows the Matlab syntax conventions more closely than NumPy (or R) which i think was a good design choice

Numpy does have much better scientific capabilities, so you can use vanilla Python for the basic coding and Numpy routines when you need them.

APL uses arrays for everything though. I just wish it had taken off more on the scientific front instead of the financial side as the linear algebra stuff isn't nearly as fleshed out as Matlab/Numpy/Mathematica.

I started writing a program that transformed J source to numpy. For just the basic verbs it was surprisingly easy. I think this is because arrays are arrays and we only have one way to think about linear algebra.

read through the OP and scanned a few examples on the J Project Wiki

i suspect that, to capture the full value of J, one needs to already know the array-oriented paradigm (or at least have a strong motivation to learn the two in parallel)

knowing the AOP paradigm also helps understand the programming patterns as well as the syntax

still, the dense syntax has extraordinary power

for instance, n-queens in J (from Rosetta Code) is just 4 short lines:

    perm   =: ! A.&i. ]             
    comb2  =: (, #: I.@,@(</)&i.)~    
    mask   =: [ */@:~:&(|@-/) {
    queenst=: comb2 (] #"1~ mask)&.|: perm

a fundamental AOP patterns is evidenced here:

line 1: create the 2D array (n x n chessboard)

line 2: populate the array with candidate solutions

line 3: eliminate invalid solutions ('mask' them)

i suspect that, to capture the full value of J, one needs to already know the array-oriented paradigm (or at least have a strong motivation to learn the two in parallel)

Yes, that's the power that's specific to J. Having a library of functions like anagram index, nub, antibase, matrix inverse, etc. can be done in pretty much any language. So can using really short names for everything (have a look at the J implementation). What J has that isn't so easy to export is the way first-order functions work on arbitrarily high-rank arguments, and even a lot of APLs don't let you mix argument ranks in the way that J does (e.g., vector-matrix addition). Without the inherently rank-polymorphic function application semantics, it would just be an idiosyncratic pointfree programming syntax for people who don't want to use anything higher than second-order functions (and maybe have a grudge against static parsing).

The talk "APLicative Programming with Naperian Functors" shows an approach to APL-like programming in Haskell https://www.youtube.com/watch?v=D1sT0xNrHIQ

Many APL idioms, when translated to Haskell, are related to applicative functors. A smidget of dependent types is needed as well, to express the dimensions of vectors, arrays, and the like.

Writing J code feels a bit more like you're interacting with a database, in that the code is being executed in a way that isn't immediately obvious based on the logical instructions you give it. It seems like the higher level and terser you can make the instructions, the faster it will generally run, because J can more ideally optimize the operations.

Reply on news.ycombinator.com