Hacker News Reader

Comment by zests | Hacker News Reader

I don't really understand these array based J/K/APL languages but from a distance it would seem like they would primarily be useful for numerical based work such as finance or calculating patterns on 2d arrays (game of life). The languages seem more mathematical than software engineering based.

It reminds me about when I was learning my first programming language. The book had a ton of example with simple numerical computations (e.g., calculate leap years) and I soon became competent with the language's facility for basic mathematical operations. But as a programmer now I have very little use for these operations so when I learn a new language I have only a rudimentary understanding of how to do math in that language.

Is this correct or am I misunderstanding the usefulness of array based languages?

View on HN

APL is best thought of, not as a "programming language", but as a DSL for array manipulation. The syntax is sadly dead. The semantics live on in the most mainstream of places, notably Numpy.

Trying to write entire full-features programs in APL should be thought of as roughly the same as doing so in Awk. Possible, and even beautiful - but not a serious endeavor. We'd be better off if other languages incorporated APL as a first class DSL, the same way they incorporated regex.

> Numpy

https://analyzethedatanotthedrivel.org/2018/03/31/numpy-anot... . https://news.ycombinator.com/item?id=16982086 .

> if other languages incorporated APL as a first class DSL, the same way they incorporated regex

April for Common Lisp et al? https://news.ycombinator.com/item?id=22225136

Numpy has a very rough approximation of APL's semantics. The broadcasting rules are quite different, and it's missing some rather fundamental things like automatically lifting functions with argument rank other than 0.

It requires a different mindset. You might not need leap year calculation often, but many things you do have a short, simple, efficient J/K/APL implementation entirely different from how you’d do it in other languages.

E.g. to check if a text string s has balanced parantheses in C/Java/Python, you would have a loop to increase/decrease depth based on character, error if negative at any point, and balanced if zero at end.

In k, you would compute the running balance at each point

    b:+\(s=“(“)-(s=“)”)

And then check the minimum of it &/b to see if it is negative, and the last element *|b To see if it is zero.

This is a text parsing problem, but has an elegant vector solution. Most things do, whether it is graphics, database, whatever - but it’s rarely intuitive if you are not used to thinking in vectors.

Also, the expression for b above can be half as long, but you would really need to be versed in k to understand it.

That’s a very trivial example though. For a more real-world example, consider multiple different delimiters for matching. The C/Java/Python version is still trivial if you use a stack instead of a numeric counter, returning an error if you try to pop an empty stack or one whose top element does not correspond to the delimiter you’re matching. How would you do this in K?

The stack makes it much less trivial than a counter in Python, and even less so in e.g. C which lacks a built in data type useful as a stack (leading many to allocate a fixed size stack and either reject valid strings going too deep or have buffer overflows)

Indeed, a multi delimiter version is harder in K as well. I am writing this on a phone without K (as I did the prev post) so will not attempt a runnable example; but, I would implement it as a state machine, taking Stack and one char as input; and returning new stack as output; then reducing it on s; if final stack is empty, it is valid. It would still be one line.

K loves state machines. When tokenizing/parsing in K, you’ll often see people write DFA tables by hand because it turns out to be simpler than writing a lex spec; also shift/reduce tables because it is simpler than yacc grammars.

I would not recommend these if you want to write a C++ compiler, or you need yacc’s error reporting and recovery. But 95% of the time, yacc/lex are for simple calc or DSL expressions that ARE easier to code by hand in K.

And that’s another place K/APL mindset differs significantly from common practice: Avoid solving the general form of your problem if you can solve the particular one in one line (which you often can, and is often enough)

It’s not very hard to code up a dynamically allocated stack in C. I think it’s of fundamental importance for a programming language to let you easily implement lots of different data structures efficiently.

Are you saying K has a stack built-in (or another structure that can easily implement a stack)? Okay then how about a problem for which K lacks a built in data structure? I would google it myself but I can’t seem to get any results on K.

K has lists somewhat comparable to Python’s, which is what i’d use to implement a stack in either (append to push, shorten/pop to pop)

You are moving the goalposts. You can construct problems where eventually K cannot be shorter/simpler - this is a consequence of Turing equivalence and kolmogorov complexity.

The claim I made was that, for problems that people actually need to solve, K And APL often have a short, simple, elegant and usually performant solution.

I stress that I am talking about the actual problems - that is, creating a mapping from an input set to a respective output set - I am NOT claiming that the solutions people reach for in language $x can be expressed more elegantly in K/APL. Another example is “flatten” - expressed as ,// in K, and which could be read “join over until-converged”. This is not how you would implement it in Lisp or Python likely - and it is likely less efficient at runtime than how you WOULD implement it in Lisp. But it is arguably short, simple, elegant and (depending on implementation and input structures) likely acceptably efficient.

Added p.s: it is not hard to code a dynamically allocated stack in list, but it is about as complicated as checking the balance, which makes the balanced parentheses problem twice as complicated in C compared to even Python; which is why so many C buffer overflows exist - the buffer Management was accidental Complexity and therefore gets less attention than the security implications of a buffer overflow would warrant.

The reason I'm asking about further problems is not because I want to "move the goalposts" but because I (and likely other people) are skeptical about the ability of K (and other APL family languages) scale beyond simple problems or specific niches (such as finance).

It's fine if it can't though. Different languages have different niches. You wouldn't write an operating system in python. I'd just like to know if there is a use case for K outside of finance (or other sorts of math on large data sets). Has anyone written a video game in K?

> The reason I'm asking about further problems is not because I want to "move the goalposts" but because I (and likely other people) are skeptical about the ability of K (and other APL family languages) scale beyond simple problems or specific niches (such as finance).

Yeah and I've become wary of languages that are an island where everything is nice as long as you can change the problem until it has a nice solution in the language. Forth is a bit like that.

The risk is that all these nice and elegant things fall apart like a house of cards once you try to merge the island with the ugly nasty real world, where legacy compatibility and rigid requirements from various stakeholders define the problem.

I think most people out there are primarily interested in general purpose languages that they can learn, master, and use to attack just about every problem, ugly or not. In that case, it's only natural to be reserved about proposals that are sold as very elegant but the demonstrations of which are quite removed from the ugly real world problems.

I'm kind of on the edge. I can tell that a bunch of cool stuff has been done in APL and derivatives, but I'm still not sure whether it'd hold up e.g. in my day-to-day job (which involves lots of networking, async i/o, timers & callbacks, parsing, message passing, crypto, etc). And I still feel like there aren't enough relevant real world examples (or I'm just not able to find them much).

> The risk is that all these nice and elegant things fall apart like a house of cards once you try to merge the island with the ugly nasty real world, where legacy compatibility and rigid requirements from various stakeholders define the problem.

I've had more of that experience with Java than with K in the late 1990s and early 2000s. Java wasn't new, and was well accepted, but the ecosystem was horrible and interfacing to other systems required great effort and was abysmal in general.

The issue you are talking about is not a language issue, it's an ecosystem issue. K recognizes it by not trying to be all things to all people - it's incredibly easy to FFI directly, and to connect through various interprocess methods (including a native efficient KIPC well supported from just about every other language environment).

Arthur also dropped native GUI in 2005, even though it was incredibly effective (by far the easiest way to get a GUI - way easier than VB6 for example; http://nsl.com has a spreadsheet, calculator, and more showing just how easy). It was incredibly effective, but also incredibly unattractive (the term "milspec" was often used by users).

This the issue you mention - users wanted beautiful GUIs, often web ones, that they could put their own touch to. The requirements are rigid and have inherent complexity that general elegance cannot help with. K is decoupled enough to not have to solve everything to be extremely useful.

I don't know if it's still the case, but in 2004, if you called Kx systems (the people who sell K) and asked for an evaluation, they would send a person who would solve YOUR problems with K on YOUR system, working 1-2 weeks. Then you got 1-2 weeks of evaluation; and then they charged tens of thousands of $$$ for the solution if you wanted to keep using it. And many did.

Most project specs today include requirements that are accidental complexity, rather than inherent. K/APL (and their philosophy) is about solving the inherent complexity elegantly. It's often possible, but rarely practiced, that you can solve the inherent complexity one way, and the accidental complexity another way, and it ends up being much simpler overall.

> The issue you are talking about is not a language issue, it's an ecosystem issue. K recognizes it by not trying to be all things to all people - it's incredibly easy to FFI directly, and to connect through various interprocess methods (including a native efficient KIPC well supported from just about every other language environment).

I'm afraid this only makes my point stronger. If I'm pushing the ugly complexity outside of K with an FFI, I'll be stuck writing the ugly plumbing (a major part of the job) in some other language. For the uninitiated it is very difficult to say whether or how much will be left for K to handle, if anything. That kind of thing would make K look like a one-trick pony, and maybe not worth investing in.

> Most project specs today include requirements that are accidental complexity, rather than inherent.

I agree, and I'm afraid that in many projects (including my dayjob) the accidental complexity is the bigger job. Even if I could solve the inherent complexity of all we do in one beautiful line of code, we're still stuck with a mountain of crap from 2-3 decades of legacy and bad design that really can't be changed.

> If I'm pushing the ugly complexity outside of K with an FFI, I'll be stuck writing the ugly plumbing (a major part of the job) in some other language.

It’s kind of the opposite - you get an easy to use escape hatch where it makes sense - which enables you to work with large complex legacy systems.

It lets you do the ugly plumbing in K.

Contrast this with Java - back in the early 2000s my company ended up rewriting huge parts of our legacy systems to Java simply because JNI was so bad.

How about a nice game of snaKe?

http://johnearnest.github.io/ok/ike/ike.html?key=Sfm0ER_9

> [...] a state machine, taking Stack and one char as input [...]

https://en.wikipedia.org/wiki/Pushdown_automaton

Will the vector solution thing still be elegant if parentheses that are escaped with a backslash don't count? Oh, and backslashes can also be escaped, so the paren in "\\(" still counts. Oh and you can have a quoted segment in the string where parens don't count. As expected, quotes can be escaped..

I bet there are elegant solutions, and perhaps they are obvious to those who are well versed in array programming. What I think the world needs is a book/tutorial/whatever that demonstrates elegant solutions to things that seem ugly and not amenable to vector processing (at least in the obvious and naive way). I think we have enough (too many) tutorials showing how to calculate means and such :(

Elegance is in the eye of the beholder, of course - a state machine is easier to code in K than in almost any other language, and is likely the most elegant overall solution.

A common kludgy solution you’d see would probably be to delete or zero out characters that shouldn’t be counted, and then do the original thing; some would consider that elegant and others won’t.

There are many examples if you look for them, many well explained, e.g. https://nsl.com ; the problem is APL/K is almost as different from C/Java as Japanese is from English ; there’s no amount of chewing anyone will do for you that will save you from going deep, and if you’ve decided to go deep, the material is already available.

Sounds good, I love state machines and I often write them in C where other people would just go with ad-hoc loops and branches and temporaries. I'll keep an eye out for examples..

Recently there was a link on simdjson, a C++ parser for JSON which aims to be record fast. It uses vector instructions of modern processes, but if you'll watch the video presentation, Daniel Lemire will also explain how they handle finding string boundaries - with all those escapes needed - without branching, purely logical operations.

Could be a good example for how APL approaches things.

The one thing that makes you think different when working with these languages for a bit, is that your mind has to move differently from when you are doing 'regular' programming.

> Will the vector solution thing still be elegant if parentheses that are escaped with a backslash don't count? Oh, and backslashes can also be escaped, so the paren in "\\(" still counts. Oh and you can have a quoted segment in the string where parens don't count. As expected, quotes can be escaped..

You name 4 additional 'features' to the issue; in a c-like language, for example, you would amend the original problem with extra code to add these in. And, usually, you can recognise the original problem solution somewhere inside the final code. In k you might find that the final solution is the same size as the original but has no (seeming) overlap at all. You might find, that when you bring the problems as managers often do; after you implemented 1 thing, they would in with 'oh, it does this as well, amirite?', that you could end up with 5 solutions that are completely different but the last one delivers the features you required.

Good or bad, that's a matter of taste, situation etc.

You build vectors of the other conditions, and subtract them before the running sum.

I'm interested in this are (building a toy relational lang with array capabilities) and one are I have not find example is CRUD apps. How exactly a simple todo or invoice work here?

I don't have a crud example off hand, but https://nsl.com/ has a lot of examples - chat server/client, instant messenger, spreadsheet, class tracker, 15-puzzle, calculator, raytracer, pivot tables, and more.

You may like http://nsl.com/k/t.k - a 14 line in-memory, quite efficient, relational-style database (no parser, just the ops).

Note most of nsl is k2/k3 ; the differences aren't huge, but if you dig deep it helps to know which version you're reading.

I agree that array based programming languages are especially useful for solving mathematical problems.

For anyone new to the field, a very approachable introduction to array programming is in the Klong book, here an excerpt: http://t3x.org/klong/ap-excerpt.pdf

They're useful for a lot more than math, certainly. APL has a lot of world-firsts that wouldn't really make sense for a language useful for nothing more than math; looking at the history of IPSA and similar will probably give you some good ideas. The user 'dTal in this thread is far from giving you an accurate story.

Not just 2d, nd, three, four, five, whatever dimensions, a bit agnostic in this, as is algebra. I can't claim to have dabbled in the language but was somewhat aware in the 80s and 90s. An APL magazine from the 90s was called Vector if I recall correctly. Yes quite mathematical, and an interesting community then of some quite smart melding of math heads, still a huge focus on performance, played with it on a VAX.

Reply on news.ycombinator.com