A Conversation with Arthur Whitney (2009)
queue.acm.org
queue.acm.org
=================================================
BC: Do you ever look at your own code and think, "What the hell was I doing here?"
AW: No, I guess I don't.
BC: Wow! I confess that I tend to write comments for my future self. I know that when I come back to code I've written, I often don't recall instantly what the problem at hand was or how I solved it. Now you've got me thinking that maybe I'm just in the wrong language. When you're at this higher level of abstraction, maybe it's easier to see your intent.
In terms of debugging your code, obviously the power of a terse language such as K or Q is that, presumably, it's easier to find bugs by inspection. How do you debug them?
AW: In C I never learned to use the debugger so I used to never make mistakes, but now I make mistakes and I just put in a print statement. K is interpreted, so it's a lot easier. If I'm surprised at the value of some local at some point, I can put in a print, and that's really all I do.
BC: That works well when you have deterministic inputs. What if the nature of the problem is just less reproducible—for example, if you were in an event-driven system where you had a confluence of events that led to a problem?
AW: It has been 20 years now that I've had Wall Street customers—they're doing 2 billion transactions a day and they have trillion-row databases—and in those 20 years, there was one time where we couldn't reproduce the bug. That was nasty. I knew the kinds of operations that they were doing and I finally found it by just reading my code.
=================================================
I can't believe, well I guess I can, and have to, that people out there can be this good. Is there something missing from the reality painted here?
In general, Mr. Whitney seems like a freakin' genius, and I'm gonna take a stab at learning some K just to see what it provides. For those interested, from a previous thread today about Mr. Whitney's work, is Kona an open source implementation of K:
However, you need to appreciate he's that good because he's able to see solutions that are easy to implement, and not that he has any special implementation talent. K itself is tiny. The programs he's written in it are an order of magnitude less, yet do a frightening amount for their size.
And I keep thinking "ignorance is bliss" might have been right in this respect - perhaps I would have been better off had I not lost my appreciation for the (now obviously sad) state of software construction. (Engineering it is clearly not, by and large).
But I wouldn't go back. Would you?
I find that especially in the valley, adherence to buzzwords and fashion of the day is a little too common for my taste now.
Even the URL is terse. The attitude is "this is so obvious, I don't know why I'm even telling you". And if you work at it, you will eventually realise: yes, it is obvious.
The product is fantastically good at what it does. Maybe with today's SSDs and hundreds of GB of RAM you don't need such highly-optimised software, but it's still something to marvel at, like Petra, or the Merlin engine.
http://queue.acm.org/detail.cfm?id=1531242
It turns out he rewrites the whole compiler from scratch in C every four years.
This is a snapshot from the source code of 'J' also by Arthur Whitney - the precursor to 'K'.
http://keiapl.org/rhui/remember.htm#incunabulum
This is it described by Ken Iverson: "The final impetus that got J started was the one-page interpreter fragment that Arthur wrote, recorded in Appendix A of An Implementation of J [29] and also reproduced in Appendix A below. My immediate reaction on seeing the page was recoil and puzzlement: it looked nothing like any C code I had ever seen. (“Is it even C?”) However, Ken counselled that I should reserve judgment. "
Without wanting to psychologise this too much, when I put the reckless coding practice together with a paragraph like "Whitney is no respecter of rules. One of the scariest things I ever did as a young man was following him through central Toronto on a bicycle." I get the impression Whitney thinks he doesn't have to follow other people's rules because he is an infallible genius. Seriously unbecoming and offputting.
There's the case for a kind of extreme pragmatism — writing code that does the one thing it needs to do, without particular concerns for things like abstraction.
We all write one-off scripts that are short and precise, but for a myriad of reasons won't scale to anything bigger or more general: we cheat and use global variables; we target one specific use case but don't allow parametization/configuration to support others; we target one OS and a certain constellation of dependencies; and so on.
What Whitney has accomplished, I suspect, is that he has found a way to reduce the problem surface by going for extreme simplicity in absolutely all areas of his system, which is by making certain assumptions. Everything is lists or simple data types, there are no abstractions (by which I mean things like interfaces, generics, abstract data types, even custom types), almost everything is in-memory.
You can do this with a Lisp or a Scheme, too. But the temptation is always to reach out for abstractions and patterns and build complicated systems out of subsystems. A database would have a cache manager, a page manager, a planner, an index manager, an SQL parser, a transaction log manager, etc. etc.
K shows that you don't need go for this clean, compartmentalized, layered subsystem approach if the whole program is small enough to make it easy to see the whole system on the screen. You can use global variables because it doesn't violate any layering. You don't abstract things like indexes and operators into abstract internal APIs because you know at any point what they are.
An example of this kind of extremist simplification is how kdb+ stores tables: It memory-maps the entire table file and treats it as an in-memory list. Instead of having to invent a paging mechanism and create functions that work on tuples in pages, they can simply reuse all the vector-oriented K operators, which work on the tuples exactly the way they work on other in-memory lists.
I don't think I could write K, but it is certainly a source of inspiration when designing systems. We developers have a tendency to pattern-match and generalize, when often "ad hoc" techniques would be better. The old adage is "make everything as simple as possible, but not simpler", but it's actually possible to make things simpler.
If you focus just on inner loops, this kind of simplicity is widespread and essential for performance. APL and its derivates are great specification languages for array manipulation. If you are writing a networked window system, on the other hand, a "high ceremony" language is more appropriate because you need to cope with protocols and abstractions. Right?
A networked window system — perhaps not the best example. With that one I think the assumptions are flawed (ie., that you can make one without introducing painful leaky abstractions) to begin with. ;-)
Perhaps the key contrast here is between the tendency toward static mathematical perfection of the APL family and the notion of dynamic languages for building systems that Alan Kay et al would propose.
"APL is like a beautiful diamond - flawless, beautifully symmetrical. But you can’t add anything to it. If you try to glue on another diamond, you don’t get a bigger diamond." --Joel Moses
There's a talk by Ian Piumarta that tries to get to the core of what the VPRI guys feel is important. I'm not sure about it but it seems worth mentioning in this context: Ian Piumarta "Building Your Own Dynamic Language" http://www.youtube.com/watch?v=cn7kTPbW6QQ
Also, this: https://www.cs.cmu.edu/~aldrich/papers/objects-essay.pdf