Manipulate Functions with the J Language | Hacker News Reader

Manipulate Functions with the J Language

adamtornhill.com

112 pointsnephrenka31 comments

Interesting article. I'm playing with J currently and it is one powerful, yet difficult to learn language. Simple things are simple, but being able to efficiently chain trains of verbs together and think not in terms of loops and normal data structures, but arrays and math/matrix operations takes some training and time to get used to. I'd like to keep it up to obtain a powerful data analysis tool in my arsenal, but I want to make sure there are enough J & APL developers to make sure it is worth it. The language itself is amazing, but if there aren't enough developers you start to suffer from lack of libraries and you end up back in JS/Java/Python/Perl land.

Yes, it looks like they dropped library support for xml apparently because of some problem with the C implementation of sax (presumably some of the same things that prompted others to migrate to sax2)?

(But apparently they can support libraries that have C shared library interfaces, so... there's that?)

I didn't know about XML being dropped...interesting

J is a beautiful, beautiful language. It is quite hard to learn -- as a benchmark, I'd say Haskell is easy by comparison -- but once you reach a critical mass of knowledge it can more addictive, and more fun to program in, than anything else I've tried.

While J is fast, and can be practical, I would never recommend it for that reason, if only because it's so niche. But if you want to glimpse what a small sliver of all possibility the world of your day to day programming is, you will love J.

I love J. It's elegant, expressive, and concise. Just like regex, it looks like line noise at first, but that is because it's designed to get its job done as efficiently as possible.

I think it would be a wonderful way to write deep learning modules, since it is such a powerful tensor manipulation language. Unfortunately however, even although it should be perfectly suited to SIMD and GPU acceleration, this work hasn't been done yet.

So it ends up not really being fast enough for modern numeric programming - but I hope one day someone smarter than me will invest in making this happen.

I should add that the J community is wonderfully helpful. If you decide to take the plunge in learning this great language, be sure to join the mailing list.

If you get the chance, check out Aaron Hsu's work. He wrote a compiler that converts Dyalog APL to C++ for high performance stuff...I can't remember if GPU or what. He goes over the code in several YouTube videos and HackerNews posts. First class stuff. I think Dyalog sells it as a product.

APL (and J by extension) are more tricky to parallelise than you might expect. The frequent reliance on boxing leads to irregular pointer structures, and the absence of compile-time type information makes it hard to generate code at all. APL is usually based on efficient implementations of primitives, but that is certainly too fine-grained to be sufficient for bandwidth-starved devices such as GPUs. I contributed to an APL-to-GPU compiler[0], and it was hard to make it work on more than a small (well-behaved) subset.

[0]: https://github.com/melsman/apltail

Dyalog seem to have done amazing work on this in the last few years. Talk about Dyalog onging performance work https://video.dyalog.com/Dyalog16/?v=2AeONlTj1aY. Latest version performance info https://www.dyalog.com/dyalog/dyalog-versions/160/performanc...

> it looks like line noise at first, but that is because it's designed to get its job done as efficiently as possible.

Myth: line noise syntax required for efficient machine code.

Teenagers believed this about C versus Pascal in the 1980's. "Stuff like while (* s++ = * d++); looks faster so it must be."

Or else, what do you mean by efficiency?

Cheaper software engineering life-cycle, end-to-end?

Fewer keystrokes?

The problem I've seen with these languages is they are tough to debug syntax and semantics errors. A typed array programming language would be interesting. When are J or APL expressions well-formed and would some kind of type system help incrementally build and compose expressions?

J syntax is very strictly enforced; the rules for verbs are unambiguous and fairly easy to parse as a human. The difficulty arises in learning and committing these rules to memory.

For example, every verb is infix, but the right side is evaluated before the left. thus expressions like '2+34' is unambiguously different from '43+2' (the former evaluates to 14 while the latter is 24). J is a language similar to C in that it will oftwn assume you meant what you typed, but has much less undefined behavior.

Re: types, J has a type system, albeit most are numeric (support for complex and rationals are built in). I dont, personally, imagine a type system akin to Haskell or Ocaml would benefit J greatly. Almost every verb in the language is overloaded with respect to the numeric types, thats why addition works as it should with ints, floats, complex, and rationals. Whats more, it handles all that stuff in the underlying system, so that everything works together correctly (coercion and whatnot). A type systen would introduce complexity and virtually nothing more to an already complex language.

If you add spaces around your (single) asterix, hn won't interpret it as signalling itallics: * itallics * .

'2+34' is unambiguously different from '43+2' (the former evaluates to 14 while the latter is 24)

I really hope the markdown has mangled this.

Me too. I'm trying to decipher what was typed, and not sure if I get it right. I'll try with spaces.

So, 2 + 3 * 4 is indeed 14 in J. The reason is J does calculations from right to left, so first 3 is multiplied to 4 making 12 and then 2 is added to that making 14.

The expression 4 * 3 + 2 would evaluate in J to 20 - not to 24. That's because first 3 + 2 makes 5 and then 4 * 5 makes 20. All verbs in J are of equal priority and evaluated right to left - no exceptions for arithmetical + - * % they are all equal and only right to left order matters.

May be I got it wrong and it wasn't 2 + 3 * 4 versus 4 * 3 + 2 but something else which would make 14 and 24 as results. But may be it's just a typo.

You may be interested in some recent research[1] on “static rank polymorphism”, a way of statically describing & enforcing the implicit structure in an array language.

[1] http://lambda-the-ultimate.org/node/5329

Thanks, that is exactly what I was looking for.

> Function inverses are the kind of construct that changes how we view code.

> [..] For example, consider pairs of malloc/free, acquire/release, and open/close.

> All of them follow the pattern captured in the Under idiom: We do something to

> create a context, apply some functions in that context, and then leave our context

> by undoing the initial effects with a logical inverse like free, close, etc.

How do function inverses interact with error handling? What happens when a file cannot be opened because it doesn't exist? Or when the file can be opened but subsequent reads fail because the file is empty?

How do you typically work with errors in programming languages like APL / J / K? (I don't mean how do you work with errors in the REPL but how do you catch and handle/report errors in your program and recover/continue gracefully?)

I had a similar experience when I first started learning machine learning; thinking in terms of, and manipulating data as, matrices and vectors was not easy at the beginning -- even if the rest of language (Python) was 'classic'.

I guess pushing the envelop with a language like J, where everything is an array, plus the unusual function manipulation, would make it even more alien; it's like touring another planet. At least that's what I felt while reading this.

---

That said, it's really boggles me that every time someone tries to explain a new paradigm or idiom, they feel obliged to make it look as superior way of doing things. (Such as in : "J helps us get better at expressing that universal pattern.").

Can't we just express difference just like that; a different new thing. It doesn't have to be better, or worse.

Otherwise, I really enjoyed the article.

The author programs in a lot of languages including Lisp and Smalltalk, so I doubt he feels J is all around superior. You have to admit the syntax is pretty universal (meaning it covers everything) and consistent. With Python you learn a bit of the language/syntax and then how to apply all the functions and methods where they belong.

To me the most interesting part of this article is the part that nobody else seems to be commenting on: the ability to automatically invert a function, even a programmer-defined function. That seems magical! What if I define a hash function? How can it possibly invert that? Clearly there's something interesting going on behind the scenes, and I wish the article discussed what it is, and how (and when) it works.

Everything else in the article seems like plain old functional programming to me, with a little syntax sugar here and there.

The only other language where I’ve seen this is Factor, with its “undo” word. I presume they work similarly, so I can speak to the general idea.

It’s not that the compiler can invert a hash, or automatically derive a logarithm function from an exponentiation function, or anything like that.

Essentially, if you have a composition of functions in a “pipeline” (h ∘ g ∘ f):

    [ f g h ]

Then to get the inverse of the composition, you just invert each function and reverse the whole thing (f⁻¹ ∘ g⁻¹ ∘ h⁻¹):

    [ f g h ] undo
    [ \ h undo \ g undo \ f undo ]

“undo” is defined for a set of primitive words. If there isn’t an inverse defined for some function you used in the composition, then it fails; however, you can just define a custom inverse for your function. And obviously this relies on having some kind of reified representation of functions available.

“undo” can be used for things like pattern-matching: a destructuring function, which takes some value and produces its fields, is the inverse of a constructor, which takes the fields and constructs a value.

You could do something similar in "normal" functional programming languages, but you usually won't get the nice syntax.

Basically, you can represent each invertible function by a pair of functions (pseudo-Haskell, because I don't use it very often and can't test on mobile):

  data Invertible a b = Invertible { forward :: a, backward :: b }

  apply :: Invertible (a -> b) (b -> a) -> a
  apply f a = forward f a

  invert :: Invertible (Invertible a b -> Invertible b a) (Invertible b a -> Invertible a b)
  invert = Invertible { forward = invert', backward = invert'}
      where
      invert' :: Invertible a b -> Invertible b a
      invert f = Invertible { forward = backward f, backward = forward f}

Then you can for example invert invert itself by

  apply invert invert == invert

But because there is no built-in language support, you'll end up doing lots of parallel construction of the standard library before you can even think of using it productively. Apparently someone already did that for Haskell (https://hackage.haskell.org/package/invertible), but it likely doesn't cover everything.

There is the basic programming language, which provides a set of instructions that computer would do; then there is the meta-language (e.g. macros, templates) that provides instructions to compose instructions. I have come to realize that the functional programming essentially is the latter part.

For example, I have a meta language in which we can define a box and unbox:

    subcode: box_a
        instructions that un-boxes a type ...
        BLOCK
        instructions that boxes a type ...

    subcode: box_b
        instructions that un-boxes b type ...
        BLOCK
        instructions that boxes b type ...

    main_code:
        &call box_a
            &call box_b
                your code

That is a verbose version of J's under adverb with my meta-layer that I can use in any languages (C, Python, ...).

What's the advantage of using array programming versus using list/array types in other languages? Nothing I saw in this article seems particularly unique to J except the syntax.

I'd take this a step further and say the website for J language doesn't move a finger to entice anyone to learn the language or use it.

I don't know if it's academia or what - but if I spent the effort on creating a whole programming language that I believed was actually good - I'd have a very different front page.

It was probably fine in the 90's when I'm guessing it was written. I personally think the minimalistic design is great, which is a trait of both the language and community. The problem is different parts of the site have different and inconsistent menus, so navigation can be cumbersome at times.

Although some academics (like myself) are intrigued by them, APL-like languages (APL, J, K, Q, etc.) have very little to do with academia historically, and were developed pretty much isolated from the academic programming languages community. APL itself does have distant roots in academia, growing out of notation Iverson developed at Harvard in the 1950s, but its history as a programming language goes via IBM and a variety of other companies generally focused on enterprise and finance (I.P. Sharp, Morgan Stanley, Dyalog, Kx Systems).

As for J, it was released by a startup, J Software, founded by Kenneth Iverson and Roger Hui in 1990 for that purpose. It was later open sourced in 2011.

I think the advantage is that you end up with much terser programs because you don't have to explicitly declare everything as an array.

Probably the built in support for vectorization, which reduces the amount of code you need to write, and allows your code to be closer to mathematical expressions. Of course there are other languages like R, Julia and Fortran that also have built-in vectorization, and Python adds that with the Numpy library. It makes certain operations much easier to express than with your normal lists/arrays. That's why the Python scientific stack is built on top of Numpy arrays, and not the builtin data structures.

You can compare it to regular expressions for manipulating text.

This setup often gives APL-family languages a curious performance advantage, even though they’re largely dynamic & interpreted. All of the hot loops are in the implementation layer, and the surface language just glues them together, just like how Python can “be” fast by acting as glue for C and Fortran.

Parallelization automatically, special constructs for sparse matrixes, complex manipulation of multidimensional arrays without need for making sure they are constructed correctly. Not to mention never having to worry about an indexing error, unless you are specifically asking for an index, in which case the bug is in your design, not the program. Built in methods for sorting and searching that are very efficient, as well as filtering. Also, J is faster when you give it all the data you can at once, rather than moving through index by index. A lot of these advantages take time to really be aware of, much like Erlang's bread and butter often hiding behind much more than one article.

Reply on news.ycombinator.com