Smaller Code, Better Code
sacrideo.us
sacrideo.us
If people express interest, I'll run such a live session and let people judge for themselves what they think of the code and my approach to "simplicity" after they've been introduced personally to the code base.
...
rth,←' A zs;A rs=scl(r.v(0));rr##mf(zs,rs,p);if(c==1){z.v=zs.v;R;}\',nl
rth,←' array v=array(z.s,zs.v.type());v(0)=zs.v(0);\',nl
rth,←' DO(c-1,rs.v=r.v(i+1);rr##mf(zs,rs,p);v(i+1)=zs.v(0))z.v=v;)\',nl
rth,←' DL(zz,if(rr##scl){rr##df(z,l,r,p);R;}\',nl
No.
And commit messages like "Hopefully that does it." No again.
When faced with something unconventional, the reaction we're hoping for from HN users is first to pause—and then to reflect. If after pausing and reflecting you want to argue that the conventional position is right, you'll be able to do that thoughtfully and with some sense of nuance.
imagine being presented with this and tasked with maintaining this. or adding a language feature. i'm certain the author could do it without much effort, but this code is as short as to be obfuscated - i have had more understanding from ioccc entries than this.
code exists as a common language for humans to understand and collaborate. this code is nightmare-ish.
https://github.com/arcfide/Co-dfns#publications
You can read the "Key" paper that walks you through one of the core data structures in the compiler and how it works. If you see the "rn" binding in the compiler code in "e.cd" you'll see that code put into the first compiler pass of the project.
If you want to see more than that, I can email you a copy of an updated version of that paper which is currently under review (not likely to see publishing this round). It includes descriptions of the implementations of the "lf" and "fe" compiler passes.
Another assumption is that short variable names always make a program more obscure.
Another assumption is that concision makes a program less readable. This assumption runs so deep that it's hard to even see it, but consider: one concise line of code is harder to read than one verbose line, but much easier to read than a million verbose lines. That is, in the time it would take me to understand a million lines of verbose code, I could learn the language of a concise program from scratch, work with its short (and at first obscure) codebase long enough to understand it, and still have orders of magnitude to spare. At some point on the code size curve, the tradeoffs change dramatically.
I don't know the OP's program but I can tell you that the APL culture is a mature, sophisticated, and beautiful approach to programming that occupies a different local optimum than most programmers are used to. To react to this not just with disagreement but outrage ("disgusting"!) is quite interesting. There's something threatening about encountering an approach to one's area of expertise that is so radically different, based on such different assumptions, as to be outright alien. Our reflex is to dismiss it forcefully. But if you can catch yourself doing that and stay with the unfamiliar long enough to get over the "disgust" response, the reward is magical.
the author themselve states that they are responsible for the majority of commits. this should be a red-flag, itself.
I wouldn't ordinarily want to make this personal. But you, who cannot be bothered with grammar, punctuation, or the shift key, want to make a statement about this code. You must understand it is a personal statement more about yourself than the code you are looking at.
Here is someone who does something you admit you cannot do, and you say don't do that! Don't do the things I cannot do! What a missed opportunity! If I see someone do something I cannot do I say how do you do that? Why do you do that?
Are your an experienced user of array-oriented, programming languages? If so, what specific things about the code were bad other than the shortened names someone else mentioned?
Technically, the above is a snippet of C++ put into an APL variable "rth" but there's so much more to it than that, and so much more to the design that you're missing.
The design and choice of aesthetic in the compiler is a very intentional one that is arguably one of the main issues that has caused me to rewrite the compiler so many times over the years and has lead to this massive code adjustment.
There are very good reasons that the compiler is written in the style that it is, and you cannot compare it to other project's style guides.
Keep in mind that this compiler is designed to run natively on the GPU in a fully data-parallel fashion.
One major issue that I had to address, and I discuss a little bit in a thread above, is the idea of the malleability of the code base. It's critically important to this project that I be able to adapt and alter the compiler rapidly. For example, I recently had to rewrite the entire backend due to a shift in some underlying core technology. This shift lead to a shrinkage of about 2000 lines of code because the underlying supporting libraries were a better fit to what I needed than what I was using previous to this. But I might not have been willing or able to make this change if I didn't have confidence that the rewrite would be swift and fast. Indeed, it took only two months to rewrite the backend from scratch, add more new features, improve robustness, and so forth. The code also got cleaner.
This obsessive need to be highly adaptable leads me to the desire to have exceptionally "disposable" code. The cost for replacing or deleting code should be as low as possible.
This has a few follow ups. In order to achieve the above, I need to ensure that I understand the ramifications of deleting code as readily as possible as quickly as possible. This basically means that I need to be able to squeeze as much of the compiler into my head as possible, and what doesn't fit, I need to be able to "see" and "read" as quickly and as readily as possible.
The compiler is designed so that I can see as much as possible with as little indirection as possible, so that when I see a piece of code I not only know how it works in complete detail, but how it connects to the world around it, and every single dependency related to it in basically one single half screen full of code (usually much less than that) without any jumps, paging, scrolling or any movement. It means that I can completely understand the ramifications of any edit I make in nearly complete detail without any dereferencing or indirection. There are one or two places where there are some helper utilities which are on a different page, but these are part of the "domain vocabulary" which is basically in my mental cache any time I'm working with that code. I keep these "helpers" to a minimum, so that they can fit with anything else I want and not waste mental space in my head. Too many helpers leads to a failure to understand the complete macro picture and thus defeats my ability to delete code.
In order to make the code more readable, it has to be highly consistent and idiomatic. I take this to an extreme level. This code is highly regular and predictable, to an almost obsessive degree. I do this by enforcing a style discipline on the code that allows me to eliminate the use of a host of abstractions, further paring down the complexity of the programming language in which I'm working and allowing me to think in the same mental plane at all times.
The idea of semantic density is critical to this point. The semantic density of the APL code I'm using to solve the problem is at a certain rate. I maintain a consistent density rate by choosing my variable names in such a way that they visually align with the expressivity per character of the built in primitive symbols. This means that the cadence when reading the code is maintained. The "universal" naming scheme allows me to take any given name and know exactly its purpose, parentage, place, and use in the compiler without adding any additional cognitive overhead of inheritance syntax, datatypes, classes, or anything more than a name.
The C++ code above is written the way it is to allow it to stylistically align with the semantic density of the APL code. This means that I can jump between the runtime and the compiler portions of the code with minimal mental shifts between the two, because the style and approach are similar. The code can be "read" in much the same way with minimal change. I am intentionally prioritizing internal semantic and stylistic consistency over satisfying the popular expectations of how C or APL code should look. I believe the internal consistency within the project contributes more strongly to the day-to-day readability and hackability of the project.
Furthermore, I strongly restrict my use of programming languages features. This simplifies self-hosting, but it is primarily a means of maintaining stylistic and cognitive power. Since I know how I need to think about my problem "compilation on the GPU" in order to make it go, I can restrict myself to a paradigm that only allows me to think in this way. I choose a paradigm that is also exceptionally expressive to allow me to be productive as well. By selecting the right core paradigm, I can eschew further programmatic abstractions since they contribute nothing and only cost something.
One way in which I do this is to write the core of the compiler with only one or two syntactical conventions, and only one main programming method: function composition. The entire core of the compiler is a single points-free (almost), data-flow, data parallel expression. Names provide the anchor points of the "macro" level ideas, but the language is expressive enough that I need very few other anchor points. Instead, I use only function composition over the core primitives with a syntax known as "trains" to create the mental effect of working with normal expressions when in reality I create new functions with every line in the core compiler (which is 90 lines or so). By restricting myself to only writing in this style, the mental effect works. If I had to switch between expression level and trains/points-free style in the code, it would be much less readable. But because I can now treat my points-free programs as regular expressions for all intents and purposes, it actually simplifies my cognitive load, as there is only one thing to think about: function composition.
This means that in those cases where reuse is valuable, it's very valuable, and it comes to the fore and you can see it as the critical thing that it is. It doesn't get drowned in otherwise petty abstractions that assist reusability, since we don't need that anymore.
Furthermore, if I write my code correctly, there is very, very little boiler plate in the compiler. Almost none. This means that every line is significant. By doing this it means that you don't get the fun of feeling like you're accomplishing something by typing in lots of excess boiler plate, but it does mean that you have no wasted architecture. Because rewriting the architecture is so trivial, basically everything now becomes important, and you don't have petty book keeping code around. You know that everything is important, and there is no superfluous bits.
The result, as mentioned elsewhere, is code that is getting continuously simpler, rather than continuously more complex. The code is getting easier to change over time, not harder. The architecture is getting simpler and more direct and easier to explain. Because it costs so little to re-engineer the compiler, I can do so constantly, resulting in little to no technical debt.
This is an intentional synergistic choice of a host of programming techniques, styles, disciplines, and design choices that enables me to program this way. Give up one of them and you start to break things down. It allows for a highly optimized programming code base that has all of the desirable properties people wish their code bases have, and it scares people. I think that's a good thing. Because I don't want people to see this codebase as just another thing. I want them to see that this is something truly different. How can I get away with no module system? How can I get away with no hierarchy? How can I get away with having everything at the top-level, with almost no nested definitions? How can I get away with writing a compiler that is not only shorter, but fundamentally simpler from a PL standpoint than standard compilers of similar complexity by using only function composition and name binding? How can I get a code base that has more features but continues to shrink?
By chasing smaller code. :-)
I assure you, and I'll make good on this in another reply here, I could get you up and running on understanding the code and how it works faster than just about any other compiler project out there. In the end, one of the goals I want for this compiler is for people to say, "Woah, wait, that's it? That's trivially simple." The more I can push people to think of my compiler as so trivial as to be obvious, the more I win. The compiler really is so dirt simple as to shock any normal compiler writer.
But to make it that simple, I have to do things in ways that people don't expect, because people expect complexity and indirection, they expect unnecessary layers for "safety" and they expect code that needs built in protections because the code is too complex to be obviously correct.
I'm pushing the other direction. If you can see your entire compiler at one go on a standard computer screen, what sort of possibilities does that open up? You can start thinking at the macro level, and simply avoid a whole host of problems because they are obviously wrong at that level. When you aren't afraid to delete you entire compiler and start from scratch? What sort of possibilities does that open up to you?
The only problem is that I don't know where to start if I wanted to study what you're doing and take notes. Those millions of lines of changes are still lurking in the background as building blocks for an overall understanding.
https://github.com/arcfide/Co-dfns#publications
Some of that deals with the micro and some with the macro level ideas, but there are some key elements in those that will be necessary to appreciate the whole thing.
1. What happens if you get sick. You say this is a project in production and there is money on the table (I assume not only yours). What if you get sick and are unable to work for 3 weeks or 6 months. Don't you think that this code is very hard to grasp to someone else, who would have to temporarily work on your postion?
2. It is weird, that you wrote such a long essay, spanning two comments, but it has so little examples from the actual code. Usually when people explain stuff they go between the abstract concepts and how they are materialized in the code. Here you only explain the idea behind writing it and how it makes you feel/operate/gain flexibility and performance but the closest to the code information I've got from it is that it has compiler passes and that it has a C++ runtime in a string variable. Just a thought, what do you think about that?
You're right about the code examples. I figure that people were already posting some code snippets. I wanted to give the big ideas rather than any specifics. The reason for this is basically that if you take any single line of code out of context, it's a bit hard to explain why I'm doing the things that I'm doing. It's very much a macro design, which is why I am offering the live session to go through. It's sort of, but not quite, an "all or nothing" thing. if you let me sit down with you and go through the entire code base, then I can explain how it all fits together and why things are the way they are, but if you just take a single piece of code out, you're missing the picture.
If I took a single compiler pass, out, for instance, you'd have between 1 and 12 lines of code to look at. I could explain a few features, but how would I explain that when you look at this piece of code you're able to see it entirely in context? Well, I can't, because the code it completely out of context at that point. Or what about demonstrating how the naming conventions exhibit structure informative regularity? Again, I can't, because that's a visual design element of the code. It's something you have to "see" by looking at the whole painting as it were.
The naming convention is actually a great example. Out of context, there's apparently no rhyme or reason to it. But in context, it forms a key component to the visual regularity and continuity throughout the code. The names are an important part of how you can see the structure of the code. It helps to orient you in the big pie. But if I were to quote a single line here, there's now pie to look at, no sky to navigate by. It's just a single constellation. By analogy, it does less good to say, here's the Big Dipper, it's useful. But why? Because it's easy to find amidst the context of starts and its shape helps you to find the North Star. But on its own it doesn't seem as valuable. At that point it is just another constellation. The same thing happens with this code.
So I'll go through and explicate it all in detail in the live session, where I can provide the "painting" and workflow in its entire so people can see how it works. Then you can see how my comments here match up with the code.
In the large program case, the programmer feels like they can cross-cut it, install some duplication, and yes: get their change done faster, but at a cost of making the program bigger.
But in the small-program case, you only pay the cost of learning the codebase when you add a new programmer to it -- something that happens very infrequently. Your program stays small, and you gain all the benefits therein (faster, fewer bugs, and so on).
Thank you for explaining. Wow, so much to chew on here. The naming conventions and trains sound really interesting. I can see how having a lot of the code visible on one screen would be a fantastic advantage. Again thanks for writing this up. Obviously I didn't find your code transparent at first glance, but clearly if one takes the time to understand what you are doing, the approach has its benefits. I look forward to reading more of what you post. And you've got me intrigued about APL.
"The k binary weighs in at about 50Kb. Someone asked about the interpreter source code. A frown flickered across the face of our visitor from Microsoft: what could be interesting about that? “The source is currently 264 lines of C,” said Arthur. I thought I heard a sotto voce “that’s not possible.” Arthur showed us how he had arranged his source code in five files so that he could edit any one of them without scrolling. “Hate scrolling,” he mumbled."
Source: http://archive.vector.org.uk/art10500700
I suspect his code looks a lot like the J incunabulum:
Since I believe Whitney wrote the J incunabulum, I suspect that it looks very similar. The code is actually quite simple and straightforward if you take the time to read it.
That's a great counterargument, and one I fully agree with. I've noticed that over the years, there has been a growing trend of promoting "readable, maintainable, clean, insert-fashionable-adjective-list-here code" which really amounts to a lower-common-denominator, dumbing-down perspective of how software should be written. In their perspective, code that someone does not immediately understand is "bad", seemingly regardless of how much (or little) knowledge that someone possesses. I think this is ultimately a harmful trend.
The opposing view, which appears to be largely a minority in more mainstream language communities but dominates in others like APL and Asm, is that programming languages are essentially like human languages: they need to be learned, are not necessarily "easy" or "familiar", and this learning and eventual mastery is wholly beneficial to their use. As with human languages, it is not expected nor a problem that a beginner will immediately understand code written by a more advanced user. Instead, the beginner progresses by learning the language and eventually becoming an advanced, "literate" user. This can be summed up in one sentence: "The code is unreadable because you are not yet qualified to read it." ;-)
> rth,←' array v=array(z.s,zs.v.type());v(0)=zs.v(0);\',nl
I don't know APL or the Rth variant, but this goes against most standard style guides. There may be good reasons for it, but they are not obvious to an external viewer.
- What are 'v' and 'vs'? Is this quickly obvious from context (to anyone but the author) (where are the comments?)
- Why are they single characters (non-descriptive variables are almost always a code smell)
- Why does it do multiple things on one line? I think this is a limitation of use of Rth? Normally this sort of whitespace compression is verboten, even in functional languages like Lisp.
I think the parent has a good point about this being very difficult to understand code, and the OP has confused terseness with quality. If this code followed traditional coding styles it would be easier for new people to understand what the hell is going on, and would probably 3+ times longer in LoC. But who the hell uses LoC as a valid metric anyway? Besides the worst sorts of manager, of course...
That's precisely the point. The whole philosophy of APL is that it's not supposed to be obvious to anyone who doesn't (yet) know it. However, seeing as the character set is still Latin, it's not hard to guess at what it does even if you don't know the language.
- I don't see 'vs' in the snippet, but would guess V stands for Vector.
- Even in all but the most anal "standard style guides", single-character names are normal for temporary/limited-scope variable names.
- You could likewise ask why Chinese words aren't separated by spaces, or why English words need to be. It's a different language with its own grammar and style.
I don't know APL either, but at least I make an effort to see their perspective on the language, because it is clear that there are people who are highly proficient at working with code like this. (Likewise, I would guess that experienced APL'ers probably find more "traditional" languages like C, Java, Python, etc. "unreadably" verbose.)
In the case of this compiler, I take an opposite convention. Most single character names are globally meaningful, and their meaning rarely, if ever changes across the whole compiler. As names get longer, they progressively represent more local elements. This is done in a way that reveals that nesting structure, but is also done because over time I realized that it was harder to remember from one patch to the next what the local variables were meant to do, rather than the global variables, which were almost always the same all the time and were much more likely to be in mental cache. Therefore, I used more "information", that is, more characters, for local names that I would more likely forget the meaning of later, than for global names that were universal and almost always on my mind.
And yes, I did try doing this compiler in many, many other styles, including C, C++, Nanopass Scheme, ML, Java, Cleanroom, traditional APL style, and so on and so forth. They were all unreadably verbose and difficult to work with and very hard to make forward progress on.
But isn't the real question whether that obviousness is more important than ease of comprehension and maintenance for someone who does have the required skills to work on the code?
To a child who has just learned squares and square roots and who has never encountered TeX, the expression $e^{i\theta}=\cos\theta+i\sin\theta$ is probably just line noise. To a practising mathematician, it is immediately recognisable and a useful tool. Obviously the difference is that the experienced mathematician has learned the underlying concepts and the notation to represent them. The result is that while the teenager might be learning double-angle formulae by rote for their trigonometry exam in a few years, the experienced mathematician could use their more powerful tool to derive those formulae or any variations on the theme in moments whenever they need them. Their greater skill and understanding makes them much more capable.
There are certainly reasonable arguments for making the code for some projects accessible to new developers, but doing that isn't free if it also means compromising some aspect of that code for current developers. It's a trade-off, and sometimes requiring new developers to have a certain level of skill and understanding before they can work on a project is OK.
1. It matters how much code you can see and work with at a time.
2. This code is using research level new algorithms for solving certain problems in the core compiler that are not a part of the standard toolbox of most programmers, and so, even if you did have the code laid out differently, it wouldn't necessarily be any easier. An early version of one of these new algorithms has been published and is listed in the set of publications:
https://github.com/arcfide/Co-dfns#publications
3. As mentioned above, keeping this code at the same semantic density as the core compiler allows you to read this code in much the same way that you can read the compiler code, making it easier to jump back and forth whenever you want throughout any point in the compiler and not adjust to multiple languages. This applies to the use of single characters.
4. The letters v, z, and s are all semantically meaningful to those working in APL, but they are also globally standardized "for the most part" throughout the compiler so that you know what we are talking about whenever the letter "z" shows up anywhere in the compiler. There are some exceptions, but these are obvious, and local, and can be that way because the use site and definition site are within a couple of lines of each other. So, yes, it's quickly obvious from the context.
5. I have tried many, many times to introduce comments. They almost never help, and almost always hurt.
6. If I were to change the style, it might be easier to understand what a given snippet is doing, but this wouldn't aid in your ability to understand what the whole compiler is doing or how it all fits together. It would be a false sense of understanding because it would be divorced from the broader context. The point of the design the way it is is to encourage you to focus on the macro level design issues, and allow you to see the whole context of the whole compiler more easily.
I am perfectly willing to believe that I could reduce the size of my code by a factor of 10, maybe even 100, if I was willing to give up the constraint of making it maintainable independently of myself. I think that would be a poor tradeoff to make in most cases.
More generally I think choosing tools based on small examples is a big systematic bias affecting the industry; I can absolutely understand why people do it (because who has time to compare large systems) but I think it holds us back, and I think this particular programming style games that metric even more heavily than most, meaning people falsely attribute advantages to these languages that don't exist in the real world. I think the scepticism a lot of people are showing here is very healthy and frankly I'm surprised you don't share it.
Perhaps my use of the word 'maintain' was confusing. I'm not suggesting that one programmer write such a system and others then take over maintaining it. I'm suggesting that 3 or 4 programmers write (and maintain) such a system together and all be intimately familiar with it.
But APL was designed from the beginning to enable human communication. I would argue that almost all programming languages fail to be a good human medium of communication. The evidence I give in support of this assertion is that if you look at how people write when they think the computer won't need to see the code, such as in academic publications on computer science, see what they use in the paper. Almost all of the people who implement their ideas in one language or another fail to include the entire code in their papers, and they usually include some mathematical notation and diagrams to explain their ideas instead. They may include some small snippets of code, but they rarely if every include the full code. Dan Friedman being an exception that proves the rule, if you will.
If you then take a look at how APLers communicate when they have ideas, you see code all the time, all day long. The APL community is the only one I've seen that regularly can write complete code and talk about it fluently on a whiteboard between humans without hand waving. Even my beloved Scheme programming language cannot boast this. When working with humans on a programming task, almost no one uses their programming languages that primary communication method between themselves and other humans outside of the presence of a computer. That signals to me that they are not, in fact, natural, expedient tools for communicating ideas to other humans. The best practices utilized in most programming languages are, instead, attempts to ameliorate the situation to make the code as tractable and as manageable as possible, but they do not, primarily, represent a demonstration of the naturalness of those languages to human communication.
When I see people communicating in (my part of) the industry they use pseudocode, which is often described as looking like python. They use if anything fewer symbols (and more space) than a real programming language. They do indeed elide parts of the code - often things like error handling.
To my mind that says: we should use languages in which code looks like pseudocode/python (this idea was suggested in http://paulgraham.com/hundred.html , though he takes it in a different direction). And we should look for ways to elide in real code the parts that people like to elide when talking about programs: to e.g. have "ambient" error handling that's more-or-less invisible most of the time, without sacrificing the safety advantages of checking error cases (this is why I'm interested in e.g. effect systems).
In other words, people use pseudocode because it's close to the code they intend to write and represents their current notational expectations. It's an enforcement of legacy methods of thinking.
But many people have admitted that there is a problem with writing pseudocode style programming for modern hardware performance, where taking advantage of parallelism is important.
Furthermore, I would argue that academia is relevant because it's one of the few places where the ideas are more important than the executable. If the ideas are communicated clearly, then you've succeeded. If we really want to program for the human, then we want our programs to be focused on the communication of ideas, and not machine-focused. And the reality is that if you take the machine away, and focus on human-to-human communication, without any "industrial" bias (expectation of machine execution), then rigorious idea communication is almost always pictorial, visual, and ideographic. Fruthermore, the notations that people develop and have developed over time to communicate ideas never end up looking like mainstream programming languages. As people work with ideas, math notation is the quintessential notation for communicating human ideas rigorously. It is highly evolved for human consumption, and manipulation, rather than machine-focused.
I believe there have also been some studies on how people describe processes without any computing background, and it's inevitable that many of the core "serial" programming concepts are not "natural" in human though, but a very acquired taste.
Again, I would be surprised if you put a bunch of industry or non-industry professionals up to a white board and had them illustrate their ideas rigorously to one another on just that whiteboard, that they would naturally gravitate to any real programming language. And I doubt strongly that they would actually continue to use pseudocode at scale on the whiteboard.
Error handling was one example - I see concerns like serialization, permissions, transactionality commonly elided, and I look for better ways to handle them in programming languages as well.
> I would argue that academia is relevant because it's one of the few places where the ideas are more important than the executable. If the ideas are communicated clearly, then you've succeeded.
Maybe. That assumes that the successful papers (and successful academics) are those that communicate ideas clearly. I'm not convinced.
> the reality is that if you take the machine away, and focus on human-to-human communication, without any "industrial" bias (expectation of machine execution), then rigorious idea communication is almost always pictorial, visual, and ideographic.
Not my experience at all - if anything I'd say visual aspects tend to be a marker of less rigorous communcation.
> Fruthermore, the notations that people develop and have developed over time to communicate ideas never end up looking like mainstream programming languages. As people work with ideas, math notation is the quintessential notation for communicating human ideas rigorously.
Mathematics is one such notation; "legalese" is another, and philosophical terminology a third. I'm wary of generalising too much from mathematical notation alone.
I would point to the field of combinatorics, the traditional proofs of both the ancient Chinese mathematicians as well as those of the West, both of which took on various elements of geometry and spatial reasoning for a significant number of their proofs when other tools were not yet available. The development of algebra I see as a chiefly visual and ideographic one, even tangible or malleable one. The development of UML diagrams another. Flow charts another. We have the abacus and Chinese counting sticks, as well. And finally, while poetry is not specifically rigorous, it is efficient in a way that few other communication methods are. And we find a great deal of "visual cue" elements in that field. In physical sciences and statistics, visualization is a very important tool. Mathematical notation itself is largely spatial and visual at scale.
As for legalese, I would argue that legalese is perhaps well designed for experts to be complete, but not for clarity. Comprehensiveness is different that clarity of rigor. And as for philosophy, vocabulary is not enough. And you'll note that some of the best notational systems to arise came from the philosophy departments in working on logical systems. Those are all usually notationally represented using ideographic, rather than natural language forms. And even some Eastern philosophers who wrote very verbosely tended to make their arguments from visualizations in the mind to make their point.
Musical notation, again, has evolved into a spatial, visual notation. A large number of traditional writing systems were ideographic, including ones we now consider alphabetic/phonetic.
I'm glad you think my compiler is a small system. The problem I'm solving is one that people said was simply too difficult and impractical to pursue. If I have made it so simple as to be dismissed as trivial, then that's good. :-)
I'm happy to walk you through the compiler in the live session and let you decide for yourself just how maintainable it would be if you had to pick it up. But this code base has been designed with maintainability in mind from the beginning.
How big is a big system? You've called this a small system, but it's a compiler with commercial backing/funding that compiles a language used in production systems, and is, to my knowledge, the only compiler able to express core compilation algorithms in an efficient manner on the GPU. It's rapidly moving to the self-hosting point, and at that point we will have a complete compiler that compiles a real language that runs completely and entirely on the GPU, from parser to generator.
To give you an idea of this task. A basic scan primitive implemented efficiently on the GPU in the neatest and cleanest code that I know of published in the literature is 100 lines of code. If you compressed it, you could probably fit it into 50 - 70 lines of code. That's for one simple operation that takes anyone a single line of C code to write.
This project has taken a real compiler (it's not a C++ compiler, of course) and is putting it on the GPU. Is this a small system?
I would put it in the realm of the sort of problem that can only be meaningfully solved by simplification.
However, this isn't the only code base around. There's another company who has a larger team of APLers who maintain over 1 million lines of APL code in production. At that scale they have to make different design choices than I do, but they also say, if they can do it in APL, they do, and they wish they could do everything in APL. They are one of the only groups, to my knowledge, who has been able to see a net gain in value from implementing a static type system on top of APL's core. So, in terms of scalability, yeah, maybe you need something more (like a static type system) as your code grows, but if you manage to need 1 million lines of APL for your problem, then you're in a good place.
Still, just come to the live session and we can discuss all of the issues that you see with maintainability. If you can see a way to make the code simpler and easier to reason about at a macro level, I'll be all for it!
I figure anything being done by one person is necessarily that trivial. Maybe you're doing the work of 100 people. Maybe the work of 1000. But you can't scale arbitrarily far; at some point you'll hit your limit. The amount of work one programmer can do is, ultimately, O(1).
> they also say, if they can do it in APL, they do, and they wish they could do everything in APL.
Fair enough; where I'm working there's a rather different view of the APL parts of our codebase.
> Still, just come to the live session and we can discuss all of the issues that you see with maintainability. If you can see a way to make the code simpler and easier to reason about at a macro level, I'll be all for it!
I can't/don't do audio/video/"live" I'm afraid (and if that's the only way you can explain the code then that itself reflects badly on its maintainability). I'll read a transcript with interest.
I do think the value of conciseness is real and underrated - at the same time it's very possible to overestimate it if you're looking right at the transition point between a project being small enough to keep in your head at once, because if your project is very close to that line then you can reap huge gains from small conciseness improvements but not in a way that scales. I once looked at implementing a lot of the APL operators (Scala supports unicode identifiers and has a very flexible syntax, so you can actually get pretty close). But I've found that, at least in the context of a large codebase moving incrementally (and I firmly believe that's the one that ultimately matters, for the reasons above), the conciseness gain isn't worth the cost of not having clear English names for all the operations. Indeed I now try to move away from symbols and short names in general as much as possible.
And part of the problem is that people aren't thinking of the whole context and picture when they talk about conciseness. Conciseness is not the end goal in and of itself. It is a design constraint that creates pressures to help mentally force you to achieve the really important things. You can't blindly chase conciseness. You have to have a reason for choosing conciseness and fit that within a broader aesthetic with the big picture always kept in mind.
I think this is one of the reasons that people balk at APL and fail to integrate it into their code bases. We don't teach people how to design or write "poetic" style code. I believe one famous author (Perlis? Kay?) talked about as "lyrical" programming. It's not something we teach in schools, and it's not something that most people ever learn intentionally. So, people see APL and they get the take away that the reason this code is so neat is that it is so "short." So they often do one of two things. They try to make their own code short, or they see the APL operators and think, "I could do that in language X" (I did this at first with Scheme). But as you've discovered, that fails. It doesn't really seem to work well when a lot of people try it. But why? Why do we get such great results in some cases, and such poor ones in others? Why does it seem so hard to have "APL in Language X?" There are two or three major ideas that I think contribute to this.
The first is what makes APL code unique. It is not its shortness. The tersity of APL is just the surface characteristic of a set of synergistic design aesthetics that result in short code that can actually be useful. Yes, it is short, but it's not shortness for shortness sake. That brevity of expression is a part of a bigger picture. Iverson described some of how they saw this aesthetic himself in this "Notation as a Tool of Thought" Turing Award lecture. However, I think he misses a few things that were just "obvious" to him. One of those ideas would be the concept of the design tension between generality and specialization.
The second idea is this concept of semantic density. One of the reason, I believe, that APL code is unique, is because of the ability to remain largely at a single abstraction level and be so productive with minimal abstractions. The regularity of semantic density is really important. It allows you to make your variable name choices higher impact than they might otherwise be. But this semantic density issue means that by definition, inserting some small bit of APL code into a larger code base is fundamentally breaking semantic density. It's easier to deal with uniformly verbose code or uniformly terse code than a mixture of the two together.
Finally, the contribution of APL style coding is not primarily one of semantics, but one of design. That's why it doesn't work to just implement and APL semantics with terse naming in some other language and expect it to work. The semantics and domain and abstractions available to the APL programmer are a part of the whole, not the whole itself.
So, I'm not surprised that you struggled with integrating APL style programming with other styles. It's a case of trying to have your cake and eat it, too. It's also a reason that some groups can come to loathe the APL code base, because very often they are used to doing it another way, and the APL code base interferes with their approach. Of course, it's always possible to write bad APL code, too. But the style I'm talking about isn't strictly about just APL.
What most people encounter with integrating APL into other code bases, IME, is a fear of deleting their code. There's incremental development, and then there is glacial development. When I've worked with a few people on code integration, one of the things that trips them up is that they tend to think so much at the micro level, that they don't get any benefits from APL, because they don't understand how to leverage it. What they end up doing is trying to fit APL into their architecture, because they are too afraid to change the way that they do something. They want to try little "piecemeal" integration here and there. But that doesn't work, because having some mysterious three line piece of APL surrounded by hundreds of lines of other code doesn't work nearly as well. You get almost none of the benefits of APL style coding and all the potential social issues.
You can always write verbose, old style code in APL if you want, and then it will integrate fine, but you are just writing Python in APL then, and your gains will be minimal, at best.
Instead, you have to shift your granularity of integration. You have to think of replacing whole, independent units in your code base at a single time. This sounds scary when you first mention it, because people are imagining some massive change in the system. However, if you can take a single unit or module in your system at a time that is truly independent of the rest, then you can choose to have a separate style for that code and a separate aesthetic without incurring the same costs that you would have elsewhere. It would be like having part of your system written in Python and the other written in Haskell. Sure, you've two languages to work with, but as long as you're not switching back and forth all the time in the middle, you can focus on doing good design for each of the languages.
If this is done right, then the selected code base should go from a large piece of code to a very very small piece of code, and the time it takes to maintain that code should be minimal, requiring maybe one or two people to work on maybe a hundred lines of code instead of thousands or tens of thousands or something like that.
And really, to do it really right, this group needs to be working directly with customers on this code. Then you start to see the wins in APL style.
So, the shift in APL style programming is as much aesthetic and cultural as it is semantics and technical. If the methods aren't fundamentally forcing a change in the way you think about your code and the fundamental complexity of your code, then you're not using APL correctly.
I don't see how this is unique. There's a similar idea of keeping each function at a single semantic level in e.g. Clean Code. (The connection to a consistent level of terseness isn't there, but in my experience it isn't true; lines that use terse expressions of common concepts and verbose expressions of unusual concepts are more readable than lines that are uniformly at either level).
> Instead, you have to shift your granularity of integration. You have to think of replacing whole, independent units in your code base at a single time. This sounds scary when you first mention it, because people are imagining some massive change in the system. However, if you can take a single unit or module in your system at a time that is truly independent of the rest, then you can choose to have a separate style for that code and a separate aesthetic without incurring the same costs that you would have elsewhere. It would be like having part of your system written in Python and the other written in Haskell. Sure, you've two languages to work with, but as long as you're not switching back and forth all the time in the middle, you can focus on doing good design for each of the languages.
You can make that change piecemeal though - I've done so repeatedly, for various pairs of languages, and also for quite radical stylistic shifts within a Scala codebase. Yes, you have to define a border and gradually expand it rather than replacing random lines in the middle of other things, but it's very doable.
> this group needs to be working directly with customers on this code
Doing that is such a huge win in any language that it could easily explain all the advantages you're claiming for APL.
I find the poetry-of-code stuff unconvincing, and the "if it didn't work for you you must be doing it wrong" even less convincing. If we've ended up doing APL wrong then it's not for want of trying - the right way of doing it must be hard to communicate to people, which I think comes right back to my original issue. I'm very skeptical of anything that claims the only way to try it is a big migration and a huge raft of integrated changes - that very conveniently makes it hard to fairly compare the direct advantages of the thing itself, and ensures that anyone in a position to compare has already made a substantial investment/commitment to the thing.
You're right that you can change piecemeal, but without knowing your specific case, it's hard to say what you focused on integrating, and thus, whether you were just integrating the language or integrating a style.
If you're willing to talk more about this, feel free to email me, I would love to see more details of your experiences trying to integrate and get more details on your project. This is a part of my future research agenda, and so gathering information on what has worked and didn't work for you, and making an attempt to understand all of that in your case would be very valuable.
As for the issue with big migration, I absolutely agree that it makes it hard to study. However, that's where academic research can come in. I'm specifically taking a look and building a set of research studies to isolate specific factors related to the APL phenomenon and understanding their relevance and how they place a role in the HCI of programming languages and developer culture. There's almost no one doing these kinds of studies, so the main problem at this point is that we have anecdotes, but not a lot of large, well done studies to deal with the issues. I hope to change that.
As for working with customers, are you aware of any other large projects that directly involved customers in their programming? That actually expect their customers to read source code, when the customers are not programmers themselves? I'd be interested in seeing the results of such an effort in other languages, as the only places where I've seen this level of customer integration used for non-computer science/programmer type customers (that is, those with no programming background) is in APL.
I think a big problem with people trying to work with APL code is partly training. There is very little out there on how to do "good APL" code. You get trained all through your development career on how to work with traditional programming languages and what are considered best practices there, but it is not at all clear or obvious to me that the best practices that you learn are actually the right ones once you start moving far from the traditional programming language bent. I would include not only APL in this, but Prolog and Agda, for instance.
I would even go so far as to say that some best practices in other languages are anti-patterns in APL. If your team has made a concerted effort to avoid and educate developers to avoid anti-patterns in APL and they have actually worked at changing that aspect inside of your development team(s), I would be very interested to learn more about this. It's hard to find good case studies in this material in the wild, and so if you'd be willing to share some of yours, I'd be very interested in getting good data.
Please contact me by email (arcfide@sacrideo.us) if you would be willing to discuss further.
After that familiarity of code comes first. And by familiarity I mean: common patterns, common solutions, ability to bring new people into the fray.
Small terse languages tend to breed long-running small teams with an insanely high bus factor.
See my other reply here about Direct Development. One of the better ways to do APL development is to write your codebase in such a way that you don't just onboard other developers, you onboard the customers as well. You don't just talk about business processes and then have some IT team turn that into code, the customers themselves work off the code together with the developers directly on solving the issues of business process. The result is that the code literally reflects the needs and expressions of "why" because that's the document or artifact in which the customers write their business processes down. And at that point, tersity matters. It's important that semantic density of the code keeps computer science-y stuff away from the customer, and brings the domain vocabulary of the customer to the fore in the code itself. You can't do this if you have your code littered with words and vocabulary that the customer has to parse and work with. The words that the customer sees, i.e., the variable names, should, ideally, only come from their own domain. In this way they can follow the code and see how the data flows.
That's the way to make business processes the core component of everything. You're reducing the development cycle down to a singular point, or very close to that.
There are a number of papers on this, but this idea of "user pair programming" has been successfully used in a number of APL projects, and is often the basic interaction/development methodology through the community.
And I would argue that the most important thing about the code base is that it accurately expresses the solution to the needs of the user, that it is capable of shifting with the needs of the user as quickly as the needs of the user change, and, finally, that it be as easy as possible to verify with the user that the code in fact is an accurate solution to their needs.
You could say I care more about whether the users can get what they need out of the code, including verifying that the code is what they intend, than about how easy it is to onboard other developers who are separate from the customer. That's best achieved by making the code easier for the user to read, rather than easier for the broad range of developers to read.
What additionally interests me is the combination of points-free style and the kind of data structures you're processing in an array-biased language, could you give an insight on what that is like to work with?
In particular, I presume from your description, and only a conceptual familiarity with APL, that most or all of this code is "functional", i.e. all data structures exist as values passed between the composed functions, and nowhere else (no globals or similar). I'd love to hear more about the predominant data structures and what shape they take.
Somewhere else you mention Quad-XML, which seems to be a way to represent trees as arrays, with each element pre-fixed with its depth. I presume you use this for the AST? What kinds of operations are simpler on these arrays, and which are harder, compared to tree data structures used in other languages? For example, addressing the Nth child from a parent could be harder, since you have to search past the other children? I could imagine that operations like "set all fields X of the tree to Y" are a lot easier since no tree traversal is required.
Does your ability to quickly refactor rely on this functional nature?
The challenging part, which is part of what the research is one, is doing tree transformations in an efficient manner. Because this is a pointerless representation, you have to be careful in how you design the structure to ensure that you maximize locality and parallelism. If you read my "Key" paper in the publications:
https://github.com/arcfide/Co-dfns#publications
You'll see how I manage one of the most tricky elements. By using the techniques I describe in that paper, I can perform arbitrary computation over any group of sub-trees in the AST selected by their parent-child relationships in a data-parallel fashion. This is largely accomplished by converting that depth vector in the Quad-XML format into a "path matrix" which allows for computing and reasoning about the parent child relationships of any two arbitrary nodes in the AST without reference to any other parts of the tree, without pointers. I can then optimize that path matrix representation for either ease of construction or for performance over certain types of common operations.
That's really one of the most significant elements and sort of blows the whole problem wide open and actually makes it possible to do what I'm doing so easily.
Once I replace the standard recursive transformation idioms with this new set of Key/Path Matrix idioms, I use the Nanopass architecture to allow refactoring. Nanopass is a style of compiler construction that builds on the idea of functional programming. So, yes, the compiler itself is very very functionally oriented, and that's a very big part of the refactorability of the code. But also, since I have done so in a way that results in so few variable names, that's also a major component, and it means that code is often highly "independent" and can be removed or deleted easily.
This is argument from analogy, and with a certainty nearing 100% it doesn't apply to programming languages.
If you want to take this argument all the way though... Why not use Japanese instead?
It's ugly, it's unreadable, it takes countless hours to master the language, the grammar, the writing system. In the end you arrive... at yet another language[1]. Which may or may not express some things that English can't. By the time you've mastered Japanese, you'll have achieved near perfection and all your goals in English :)
[1] I speak four and currently am in the process of learning a fifth language (Russian, Romanian, Turkish, English, Swedish). I can say with some "expertise" that you can't make direct comparisons between natural and computer languages.
While I could have chosen Japanese, it wouldn't suit the purpose as well.
Moving this into the domain of programming languages, the point is that working with APL as a notation fundamentally changes the way you see problems. It facilitates a style of thinking and working with code that promotes the ends I'm working on.
I'm sorry that you don't like analogy, but analogy is important to me, and I'll have to throw another one in here. To me, it's more about constraints that shape design than anything else. I'm not writing "English" style stuff in Japanese or Chinese, I'm writing, say, Chinese Poetry in Chinese. It would be exceptionally difficult to achieve the same result in English. Could you literally express the same content? Sure, maybe, but it would depend on what you counted as important. And the result would be very very very verbose indeed, and would no longer have the value that the same thing in Chinese poetic style would have.
In PL, I could have written the same compiler in CUDA, and I would guess that it would take at least 10,000 lines of code or more to make it work.
I could try to write it in any other number of methods, and I would argue that not only would the results have been more ugly, they would have been much less maintainable. I could have implemented the same literal algorithms with the same literal content, but to get the Human factors that I want, I would have a very very hard time of it.
This is a big problem I see in the PL community. We don't, as a whole, understand or care to study the impact of notation on our thinking. It's all just "syntax" and we can choose what we like. But that's not really true. Just because I could write an APL library in Scheme does not mean I will be able to duplicate the efforts of APL in Scheme. Going the other way, I've for a long time tried to imagine some way of getting syntactic abstraction a la Scheme's syntax-case into APL. I can't find a way that wouldn't basically be useless, because the human factors involved change the game.
So yes, you an translate Chinese poetry into English, and no one will consider the translation to be as good. Now, what most people are suggesting is that you can create Chinese poetry by starting from English and writing the same thing there. There are a host of reasons why that doesn't work.
It's not just about meaning/algorithm/semantics, but about the experience of working with the code day to day and how that affects your ability to work in your problem domain. The design of this compiler enables better, simpler collaboration, and more adaptability and flexibility than other designs I have tried. It obviates certain documentation burdens within the team that is involved in working on this code. It simplifies deployment, maintenance, and all sorts of other things.
Perhaps the biggest boon to working in this way within the team involved in this code, is that everyone, from the managers down to the users of the compiler, are able to discuss and have conversations about making things happen, handle bug reports, and deal with architectural design issues, all without anything else in front of them except for the compiler code itself. We don't need other documentation, we don't need diagrams. We can all literally, from the top to the bottom level, work off the one single code artifact. Everyone can get what they need from that single code base, without needed extra levels of "human documents." Because the code itself is a sufficient entity for human discussion.
There's a reason why readable and beautiful code is favoured: it's so that anyone else that opens the source and tries to understand it doesn't have a difficult time, and therefore, anyone that tries to contribute doesn't have a difficult time either.
Looking at the project's Github page, I can see that there's no contributor even coming anywhere close to the project owner. Whether that's because of the obscurity of the codebase or for another reason, I can't comment. However, it does stand that the project owner is the only real contributor, and so the minimum that he himself has to consider is if he can understand the code.
Having said that, looking at the code does make me cringe. I'm sorry if that offends anyone but it is what it is: the code is not very nice to look at. It seems as though it has been engineered to be as obfuscated and shrunken as possible, without any regard for readability. I mean just the file names themselves: was there really any need for single-letters?
Now the author claims that (and a lot of other people agree with him on this) it is not for the purpose of what I outlined above, but rather, as mentioned before, so that he can understand it all easily and rapidly modify it. Whether or not that's the case I do invite you to consider the fact that the post that we are all replying to is somewhat bragging about the extremely small size of the codebase.
Personally, I think this sort of code would fit in rather well on a code-golfing forum or something similar, not on a production system. Then again, it is a personal project so ¯\_(ツ)_/¯
Please omit such offensive/defensive rhetoric from your posts to HN. It adds no information and is bad for conversation.
The problem here isn't "daring" to criticize, it's rejecting the unfamiliar. This is like traveling to a new country and complaining because they cook everything wrong and say everything wrong. Unfamiliarity is relative—it's not a property of the thing you're reacting to. Same with readability: it's relative to the reader.
In some contexts this is obvious. If you don't know German, you wouldn't reject a German text as unreadable or poorly written. But in other contexts, when we unconsciously assume or were taught that there's only one valid way to do something, we react with shock and distaste at work that violates known conventions. Such work may in fact be organized around different conventions for reasons we don't yet see. Good conversation across such boundaries requires a bit of distance from our own assumptions.
Programming is like the world of art this way. There are countless examples in art history of sharp departures from convention provoking shock and distaste, and people saying things like "There's a reason why readable and beautiful [art] is favoured". Riot police famously had to be called to the early shows of the Impressionists, yet the beauty of their paintings is obvious to us now.
Let me however just say that what you're doing in your reply is attacking the straw man. You're setting up a version of my argument and then attacking that, instead of responding to my argument directly.
I'll respond to your comment anyways. Code is art, just as you alluded to in your response. And from a solely artistic point of view, there's a certain glee and wonder at seeing short and smart code. When I browse codegolf on SE, I never fail to be amazed at the frankly fucking brilliant solutions some people come up with.
Having said that, my main point was that code like that, in my opinion, does not belong in a proper project. I get the point of it being practical for a single person, but that codebase is above and beyond what is reasonable. It's simply not nice code.
You're saying that I should not think that the code is not nice because I don't understand it, but you seem to be missing the main point in your flowery metaphors and analogies of art: art is subjective. You might find that code to be beautiful in its own way, and I'm sure that's justified to you, but I do not.
Argument from analogy?
Please omit such rhetoric from HN posts
Firstly, about this being a personal project, it's actually a bit more than that. It serves as a research platform for a research agenda around the usability of programming languages and the HCI and pedagogy of computation, yes. However, it's also a commercially funded compiler that is commercially licensed and distributed. The compiler is still in early stages, so it's large a boutique offering at this point, but that's scheduled to change this year or maybe the next. And yes, the development team that works with me on this compiler has read the code, and while they are not as fluent in it as I am, they understand how to work with it and we can talk about the compiler and work through issues in the compiler that comes up.
Indeed, the fact that the compiler is so easy to track through at a macro level has allowed us to avoid needing extra documentation throughout, because when we have a question about some level of architecture design, we can usually pull up a page of the compiler and work through it without needing any other documentation.
And, the point of the post above was that small code is a useful metric for pushing for simplicity. There is a difference between obfuscation and small code, but my code is not obfuscated to those who need to work with it. It is obfuscatory for anyone who expects to read it like a normal program.
At the heart of this is the meaning of readability. You've implicitly defined readability as being a state for code bases that allows anyone else to read and understand the code. That's a high bar. If I write a standard proof of the uncountability of the real numbers, it's a rather high bar to say that everyone should be able to read that proof.
Also, if you look at the way that the Clang codebase is engineered, for instance, if you take any one snippet of 50 lines or so of code, it's all nice, neat, and readable. But when it comes to understanding the entire compiler as a whole, the codebase is completely unreadable. It requires external documentation to understand almost any part of that code at a macro level.
But Clang uses best industry practices and is, on the whole, what most people would consider very cleanly written code. And yet, it is essentially impenetrable from a macro level without other documentation.
Instead, I'd submit that readability is something that we should consider valuable for those who have the relevant pre-requisite understandings of key ideas and concepts when looking at a new code base.
Part of the problem is that this code base is introducing new, research level ideas into the coding space. There is a fundamental difference between it and other compilers in the approach that it is taking, and thus, you can't just look for the same patterns.
I've already touched on malleability elsewhere. SICP has a classic quote about the importance of malleability in code (amoeba vs. pyramid programming).
I'll give a simple description of the architecture. If you understand everything said in the following sentence, then you'll have no trouble understanding how the code is written, and if you don't, then learning these sub-domains of programming skillsets will go a long way in helping to clarify the design.
It's a three part dfns->C++ offline batch compiler overloading the standard Quad-Fix system function in APL interpreters to compile whole, closed namespace scripts built of pure functional dfns on the Dyalog 15.0 primitive vocabulary sans-guards through a PEG parser to a core compiler written in a Nanopass compiler architecture over a linearized Quad-XML style matrix AST representation where each pass is written as a data-flow, data-parallel function train leading to a single dispatch code generator with a runtime library header prepended to each output file containing implementations of each implemented APL primitive.
Those would be the basic set of techniques and skills that are being put to use in the compiler. If you already understand Nanopass, PEG parsers, the Quad-XML tree linearization format, function trains, and so forth, then the structure and format and design of the compiler is obvious and easy to work with after about 5 minutes of orientation. If you don't have that background, then understanding that part of the compiler is rather a difficult one. In addition to this, there are new techniques being used and applied to solve problems in this compiler itself, and those are being documented through the papers that I'm publishing on these techniques:
https://github.com/arcfide/Co-dfns#publications
Most people don't have a strong data-parallel, array style programming background, which makes the micro-level code the hardest part to understand for them. However, if you are experienced in that background, then working with the compiler passes is not difficult, provided that you take the time to understand the core idioms in play.
So, in summary, I'd say that you're right that the code looks horrendous, because your heuristics are designed for code that is completely, almost assuredly, fundamentally different than this code. However, like I said, come to the live session and see me explicate the architecture of the compiler. I'll explain a lot of the ideas I mention in the above sentence enough to allow you to walk through the code easily. If you still think it's scary, okay. I'd appreciate some ways to make it easier to work with it on a day to day basis.
I still disagree with the premise that the code is clean, beautiful or readable, but I could concede that this may be due to me not having an in-depth understanding of it.
You speak about a small code base being a metric for a simple code base, and while this is true sometimes, it starts to fail as a metric when the code is more obfuscated than concise. This is the kind of code I'd see in a webshell, not a compiler.
I'll try and make it to your live session if I can; I look forward to it.
P.S. I don't know if this was intended or not, but you really came across as trying to make everything seem more complicated than it actually is in the "sentence" where you went over the architecture. Dumping a load of complex-sounding words in a giant sentence just sounds as if you're trying to justify your design decisions by further obscuring understanding of the project. I'm fairly sure this isn't your intention since you are hosting a live session, but it's just how it looks. :P
The sentence was a bit of a tongue-in-cheek sort of rhetoric. In particular, if you look up most of those words in the relevant domains, they're all standard practice ideas that are well understood around their various parts. Importanntly, none of the words or anything said in the sentence is really complicated, and if you were familiar with all of those ideas, then it would be an easy sentence. However, I wrote it in a way so that it appears to be obtuse and a bit ridiculous on the face of it. Basically, attempting a bit to mirror the code itself. The sentence very concisely and neatly describes the architecture of the compiler, but only if you know what you're reading.
One of the biggest issues with reading the compiler is that most people will have a "part" of the picture based on their backgrounds. If you have written compilers before, the overall compiler design and the strategies at a macro-level used for it will make perfect sense, but you'll balk at the data-parallel programming style. If you are an APL programmer you'll be very familiar with the basic tricks being used in the compiler and you'll easily be able to see at a micro level what's happening with the code, but not having the background in Programming Language Design (such as you might receive at Indiana University's PL course path), the overall design and the intent of the whole system won't be intuitive to you. This means that likely for anyone new coming into the code, there will be parts of the code base which feel very foreign to them. Of course, I don't know of a way of doing something new without making people learn a thing or two to understand it.
Fortunately I'm not the only one in the world that has experience in all of these areas. And even better, the code is straightforward enough that once you do learn the basic skills, it's easy to work with. But there are precious few APL/Array oriented implementers out there, and probably less than a handful of compiler writers out there that specialize in this area of languages. I hope that will change and that I can convince people that this approach is actually a very neat and easy one to work with. But that's not easy to do.
The use of single letter names in the files here is a bit of an inside joke, referencing back the style of programming of Arthur Whitney, signaling a bit of a historical "stylistic" or artistic connection, while at the same time being the first "alert" to the programmer that they are likely to see something along the lines of Whitney style C code inside of the files. It serves both as a chuckle to the APL community as well as a documentation of how you might want to prepare your mind before reading the code.
Y0←{⊃,/((⍳≢⊃n⍵)((⊣sts¨(⊃l),¨∘⊃s),'}',nl,⊣ste¨(⊃n)var¨∘⊃r)⍵),'}',nl}
I'm quite open to people providing ideas on how to properly document this project outside of "big idea" documentation that I'm currently doing with the paper publications. I've yet to be able to find anything that works. The standard best practices don't seem to flow well at all, but maybe I've missed something.
A dense expression can still take a while to puzzle out sometimes, but certainly no longer than the equivalent logic spelled out in a more verbose language across many lines.
As for "other things", here are a few K sketches I've written recently:
- generate random island shorelines:
http://johnearnest.github.io/ok/ike/ike.html?gist=629a86c257...
- animated holiday card:
http://johnearnest.github.io/ok/ike/ike.html?gist=e5fdbb8724...
- an implementation of the arcade game "Snake":
http://johnearnest.github.io/ok/ike/ike.html?gist=ead275c529...
And that's just me! There are many interesting examples and essays here: http://www.nsl.com
APL really encourages code golfing. It's barely readable in the best of circumstances, so there's a lot of temptation to compress code. Er, it's readable, but it's like reading a regular expression. You read it character by character, not line by line.
The other thing APL encourages is writing dimensionality independent code. Just as a good C programmer will write a function to concatenate two strings that works independent of the strings' sizes, a good APL programmer will write a function to accept arrays of arbitrary dimensionality where that can make sense. That's because it was not uncommon to have intermediate results of 3, 4 or 5 dimensions. Also, most of the built-in operators do something useful with higher-dimensioned inputs.
So yes, it becomes second nature, more or less. But at the same time, it restricts the kinds of problems you think about solving with it.
Today I would not recommend APL for any purpose except studying its place in computing history.
And APL has been and continues to be used to solve problems from web programming to graph processing and everything in between. While it finds immediately obvious advantage in numerically dense calculation heavy code, my research is specifically interested in demonstrating and scaling APL beyond the "status quo" of what people think APL is good for. The Co-dfns compiler mentioned in these threads is precisely such a counterpoint, demonstrating how to leverage APL in a domain space traditionally dominated by OOP and Lisp style languages, and more recently by certainly highly typed languages such as Agda or maybe Haskell.
The difference is that good APL is often more or less "parallel ready" out of the box. Good code in other languages is often anything but. I am working through making the case that "parallel first" languages such as APL are actually pedagogically and programmatically better than "serial first" languages. To this end I have conducted at least one exploratory pilot study on the matter:
// src/addclass/addclass.js
// Add class(es) to the matched nodes
u.prototype.addClass = function () {
return this.eacharg(arguments, function (el, name) {
el.classList.add(name);
});
};
addClass: function( value ) {
var classes, elem, cur, curValue, clazz, j, finalValue,
i = 0;
if ( jQuery.isFunction( value ) ) {
return this.each( function( j ) {
jQuery( this ).addClass( value.call( this, j, getClass( this ) ) );
} );
}
if ( typeof value === "string" && value ) {
classes = value.match( rnothtmlwhite ) || [];
while ( ( elem = this[ i++ ] ) ) {
curValue = getClass( elem );
cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " );
if ( cur ) {
j = 0;
while ( ( clazz = classes[ j++ ] ) ) {
if ( cur.indexOf( " " + clazz + " " ) < 0 ) {
cur += clazz + " ";
}
}
// Only assign if different to avoid unneeded rendering.
finalValue = stripAndCollapse( cur );
if ( curValue !== finalValue ) {
elem.setAttribute( "class", finalValue );
}
}
}
}
return this;
},In my opinion it is the number of requested features shipped minus the number of bugs introduced. Weighted by the importance of each, as collectively decided on by everyone or client.
Of course, it still has problems. The metric probably has to be calculated long before the number of serious bugs is even known. On the other hand, good developers anticipate needs; there may be features implemented that the users haven't even realized they wanted yet. And of course, it's hard to come up with numerical weights for the features and bugs.
But the worst problem with this metric is that it doesn't count the maintainability of the code. That's an even harder thing to measure, of course.
Combine that with a code review process that will surface commits with excessive or insufficient tests and it was one of a couple ways I validated my feel for the work of my direct reports when it came time to fill out reviews.
Profit? The trouble is it is so far removed from the day to day work we do it's almost impossible to draw any direct conclusions. So instead we start trying to use proxies like function points, bugs or lines of code. In the most dysfunctional organisations worse metrics get used like time keeping a seat warm or political skills.
If your users have the confidence that they can walk up and get any feature they need implemented with you, that's about the best metric of success I can think of.
Never seen that. Thanks for sharing as it might be a great way to drive point home to business types. The amount of code certainly turns [for developers] from asset into a liability as it's size and usage grows. Hmm. Maybe such a presentation could always consider the amount of code a liability or neutral asset that produced benefits on the other side or reduced them. A well-stated connection between the two might justify reducing technical debt.
In a strange new project straight-line code is usually easier to follow than lots of indirection and abstractions.
I would argue that indirection and abstraction can always be harmful to code readability if the amount of complexity they hide is less than the added complexity from using them. For example, if your abstractions are leaky and you use them to break a long algorithm down into a hierarchy of very short functions, a reader probably still often needs to look through the implementation to figure out what is really happening, but now they have to follow several levels of indirection to find that information.
I would also argue that if you choose levels of abstraction that really do hide a lot of complexity most of the time, this can be helpful for beginners learning a new system as well. For example, this can happen when the abstractions in question have intuitive meanings, such as representing real world objects or other recognisable concepts from the problem domain you're working with. It can also happen when the abstractions represent common patterns of behaviour in some reasonably clean and concise form, such as navigating a data structure in a particular way.
In short, while I agree with you that straight-line code can be clearer than lots of indirection and abstractions, I think that is often because of the poor choices of the latter rather than the experience level of the reader with that particular project. If they were very new, they'd have to learn what the key concepts and common patterns of behaviour were anyway, and once they do understand those ideas, good code using them should be easier to follow than code written using more primitive concepts.
To create good abstractions, constantly look for opportunities to "hallway usability test" [1] them with newcomers.
[1] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
There's definitely a tendency for that to happen, I agree. If nothing else, we often don't know what good abstractions will look like yet in the early days of a project.
Even if we do, the most useful abstractions might change as the project evolves. Fortunately, the kinds of abstractions that tend to be most useful for hiding a lot of complexity also tend to be quite stable, in the sense that while they might be extended or generalised as the project develops, it is rarely necessary to severely break backward compatibility.
In an ideal world, we could definitely have more natural tools for things like refactoring, source control and diffs/merges, though, to lower the kinds of barriers you mentioned and promote willingness to refine the design and keep it clean as a project develops.
At this point I should just plug my current project: http://akkartik.name/about, https://lobste.rs/c/rue8pf. Any comments most appreciated. (My email is in my profile.) Thanks for a fun chat!
I think you have interesting ideas and laudable goals. It's nice to see some promotion of good coding style that goes beyond basics like how to choose good names (which are also important, of course, but well covered by existing material). It's also nice to see some recognition that what may be good for working with code in the small may not always be good for working with code in the large, which I think is not as well understood as it could be in our industry, to our cost.
Curiously, while I agree with many of the goals and much of your analysis of the problems, my own search for solutions has gone in almost the opposite direction. I tend to emphasize good modular design and clearly specified interfaces more as a result and to rely on ad-hoc examples and test suites less. It's not that I have a problem with the latter or don't use them, just that I don't find them sufficiently general and unambiguous on their own. So instead, my own thinking has often been about how we might improve the way we define interfaces and then connect modules using them, including evolution and being able to migrate code using a module from one interface to another efficiently and reliably if and when that becomes useful.
It's fascinating, and perhaps a little scary, that despite being interested in similar areas and having reached some similar conclusions about problems that it would be helpful to solve, we can still look in such different places for those solutions. I suppose that's a sign of how "young" our understanding of how to build software still is and how much we still have to figure out. Perhaps your project will bring some of those answers, and I wish you luck with it.
Also consider: "In anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away, when a body has been stripped down to its nakedness."
Lately in my programming career, I find myself simplifying code, distilling it to solve the problem at hand, then clarifying the code (with good variable names, explicit comparisons to NULL/nil, fully demarcated if/else, small well-named functions, etc) so that future me can grasp it faster. This has the added benefit of pleasant peer review and getting new devs acquainted with the code.
But some people have also told me that was a great learning experience -- they spent many hours understanding this half page of code, and felt they'd really grown once they'd mastered it.
This compiler has gone through many core paradigm shifts in an attempt to find an appropriate way to express a solution to the problems that it encountered. Each iteration revealed some new insight into how to solve the problem, but inevitably lead to a need to rethink the system.
Now, the system is so expressive and capable that reusability isn't even an issue. At this point reusability is about as useful in the compiler as having a new word to represent the word "the". Why? Why not just write the? Anything else you could write is likely to create confounding layers of indirection and distance between definition and use in the code that will actually obscure clarity.
Instead, I take the intentional approach to make the code as "disposable" as possible. Why change a compiler pass that is two lines long when you can just rewrite it from scratch in less time? By leveraging a different aesthetic, architecture, and language, I'm able to have more expressivity by removing unnecessary abstraction and making it as easy as possible to re-engineer the whole thing at the drop of a hat. This means that I never have to "live with" code bloat or some design decision that's annoying me. The cost to re-engineer is so low that I have almost no technical debt. If an architecture fails to scale, replace it and move on, without any loss of productivity, and a net gain since the code gets easier and easier to work with on each iteration.
I guess there's no such thing as "good enough" with a compiler?
Those are staggering numbers to me. Kudos to the author.
I think 'simpler' would be a better term than 'smaller'.
Also - every line of code has cost. A lot of cost. Maintenance of code and complexity is not only expensive, but it adds to the maintenance of other code.
So less code to solve the problem is almost always better.
By forcing myself to ever greater degrees of ascetical code sizes, small, cute micro hacks in a given function don't work. At that point the "fancy pants" hacks fail, and I am forced to create macro simplifications that obviate the need for whole classes of programming techniques.
So, yes, we want simple, but it's about how we can push ourselves and our minds to get there.
One more point: I find that there are a lot of very common things that we, as developers, have not 'standardized' on - but if we did, it would be beneficial.
The underscore/lodash JS libraries are great examples of this.
They are not just a bunch of 'helper functions' - they are really a series of new 'functional keywords' that in a way represent a new paradigm in software: we all get used to these 'mini patterns' and call them the same thing, and when used in code they can make things a lost simpler.
Map, reduce, find, each, pull, filter etc. etc. - at first glance it would seem compulsive to jam all these into some code - but once the developers are familiar with them ... guess what - they become almost part of the programming language itself.
So I think this is a pretty good example of a 'meta' way to facilitate simplicity: agree on names for very common patterns, and abstract them away with tools or linguistic constructs.
I find it unfortunate that these ideas are only now beginning to find some general acceptance in larger, more mainstream languages.