The Tragedy of the Common Lisp: Why Large Languages Explode
medium.com
medium.com
The result is that programmers introduce complexity lightly, and when they walk into a new language/organization/etc are inclined to add random abstractions that they are used to from past experience.
And yes, I am not immune.
'Lisp programmers know the value of everything and the cost of nothing.'
Here’s the footnote:
“Perlisisms: EPIGRAMS IN PROGRAMMING by Alan J. Perlis”
Like real language, you probably prefer certain idioms, you have your own style and you know certain words better than others. This is similar to how we program, we know certain algorithms, yet are oblivious to others, we have our favorite abstractions and have our etc.
There are downsides to these habits for both spoken language and computer programming languages, no doubt, you can get stuck in a rabbit hole without seeing you're in one. In the end, a similar approach works for both, read and write widely if you want to be great at these, meet different groups, work on uncomfortable projects, you know the drill.
Just like how you expand your language and therefore your world; socially.
What abstractions?
What costs of these? In human comprehension, in runtime, in reliability, in mem/cpu?
Can you give examples which I can usefully learn so as to avoid?
TIA
After discovering the eye-opening expressivity of functional programming years ago with Dylan and Haskell, I had a substantial project in javascript. I found JS supported first class closures and lambdas, which allowed me to make iterators.
The novelty was they were iterators not over lists but over trees (DOM trees in this case but of course any tree could be made iterable).
I understood there would be a learning cost for anyone who took over from me so I left plenty of docs and pointers. It would not be a small cost either, my successors would have to learn to think differently, at a higher level and possibly rather alien (to them) way.
But was it worth the cost, bloody hell yes! Not having that iterable and rather declarative abstraction over DOM trees would have greatly bloated the code and consequently brought in bugs by the bagful. I would do in a couple of lines what would have taken half a page to do, in many locations.
So it had a human cost but if you could overcome that, a seriously huge human benefit.
Because of the abstractions I'd done (it wasn't only tree iterators) I could largely work around the bugs invisibly - I pushed the bug-special-case-handling code down to make it invisible when using my abstractions.
That made it all too viable to continue using an crappy, flaky product far longer than would have been possible - or sensible - without those abstractions. I'd turned the abstractions' value into a liability!
I finally told them it wasn't worth continuing with H, they junked it AFAIK and I walked away.
Examples that I commonly encounter include OO, closures, dependency injection frameworks, complex configuration systems, various code generation systems, and on and on and on.
In general the tradeoff is this. For those who have internalized the abstraction, they can think about more complex things. Those who have not internalized the abstraction find it hard to figure out how the system works at all until they internalize it. So when you work on code that has a lot of abstractions under the hood it becomes either a black box (that occasionally you dig into) or (very often) a requirement that you understand X before you can even start to work on the code.
A rule of thumb that I use is how long the stack backtrace is. If every bug creates a stack backtrace that is dozens of frames, there are a lot of abstraction layers in place. And when all of the layers are actively being worked on, it adds up - quickly.
Deciding whether given abstractions are worthwhile for a given problem involves a judgment call. Unfortunately the people who are in the best position to make those calls tend to be the most senior, and tend to be the least aware of the costs of the abstractions that they add.
I would note that if you start your list with OO then you're including abstractions that almost all professional programmers would consider not abstractions but basic tools. It's impossible to work without such abstractions, except by working in purely procedural code, and even then...
I would add that if "a requirement that you understand X before you can even start to work on the code" is the case then your abstraction has - arguably, I may be wrong! - failed. My DOM iterator abstraction would take some effort of understanding to maintain, which was my concern, but it was a very simple black box to use.
And yes, some abstractions truly are basic tools. And the more experience you have, the more basic they seem, and the more such tools you have.
I am not arguing against abstraction per se. What I am arguing is that abstractions bring a cost, and that cost adds up. A project needs to find an appropriate balance, and there is a tendency for experienced programmers to draw that balance at a point that may or may not be appropriate for the people who have to maintain the system after them.
This is part of what you get with a language, but it depends on the language. (We've seen Javascript users split into several such communities, I'd say).
Or it can be what you get with a framework -- one of the main values of Rails is that people who have learn it's abstractions can look at each other's code and understand it. This applies to third-party extensions to Rails that get especially popular too, like say Devise. (The downside is when those abstractions aren't good, and you want to use something else instead... now everybody finds your code difficult to understand).
When we talk about a language having a good stdlib, a lot of what we're talking about is providing a good set of abstractions that everyone will learn, making their code understandable as well as interoperable with other developers'. JS's lack of much of a stdlib may be not unrelated to JS schism into several communities of shared abstractions...
I don't think it's really about "minimizing abstractions", it's not even possible to do so -- it's about which abstractions how. The important point you make is that abstractions that are understood by a community of programmers, from which those who work on your code are likely to be from -- actually practically have less cost than abstractions that will be unfamiliar with them.
I rememeber when OO was super confusing to me...
Also beware of that kind of pseudo-wise thinking. Is there a point in climbing up a learning curve anymore ? Maybe you have a constant and massive stream of fresh bodies to throw at your "simple" solutions before laying them aside once those abstractions have clogged up their heads ... And maybe if you can unskilled workers at such a scale, it's because you have massive fundings as well...
More a question of financial optimization than software engineering I think
In the case of Go, the core team saw the pain of indirection-masquerading-as-abstraction in complex Java/C++ codebases and considered the whole thing to be a boondoggle. As a result of this we’ve been saddled with a popular language in which two massive projects (gvisor and kubernetes) have had to hack their own expressivity into the language just to build complex software (i.e. codegen’d generics)
I worry about the cyclic nature of progress in our industry, where wonderful advancements can be made and then walked back or under-utilized because we aren’t patient enough to learn them thoroughly.
If it's enterprise adoption ever goes beyond Kubernetes and Docker, expect GoEE and Go Design Patterns to make their appearance.
Worse, since its plugin support is really cramped down, expect any enterprise grade CMS to be built on hundreds of processes.
This happens all the time with simple languages, tons of library boilerplate code.
In C++, the cost of abstraction is a fundamental consideration, and while you can still ignore it - most resources about using C++, and gradually even the language itself, tend to nudge you towards avoiding abstraction costs in various ways (sometimes ugly, sometimes elegant). Plus, core language designers and compiler architects bend over backwards to reduce the cost of various abstractions to nothing or very little.
Plus, it happens that sometimes, you can use stronger abstractions to _reduce_ cost rather than increase it.
Those languages are also not very popular at all today. Perhaps "small and beautiful" are not the right metrics to optimize programming languages for.
In many ways, keeping the core JavaScript language small has led to the JavaScript ecosystem being too large and sprawling. For example, look at module importing, something which is a fairly normal part of most programming languages. It still works in a completely different way in Node.js and in browsers. I think if there had been a standard way to handle JavaScript imports years ago, the practical experience of working in JavaScript would be simpler today.
But yet, C (arguably 'current algol') and Python (arguably 'current scheme') are..
yes: python is verry loosely like scheme, this is meant in the sense of a 'dynamic loosely typed language you can interact with in a repl'
Both Python and Scheme are dynamic but strongly typed.
yes, they have strong core types, but there is nothing preventing you calling some function with completely invalid arguments..
even within the 'interpreted fp' world, there are better examples (e.g. ML family)
but then, scheme is heading in the same direction if you look at the latest standards development
The small core exists, because that was the only subset their were willing to agree on.
i don't know if/when that point will be reached. my comment was hinting at the potential for scheme to become larger.
Scheme in particular deserves so much more attention, it is so capable and it is a joy to write.
I doubt even veteran C developers know by heart its 200 UB documented use cases on ISO C, or the specificalities of any C compiler other than the one they daily use, and which features of it are actually part of ISO C.
Unless they happen to do ISO work all day long.
Perhaps popularity isn't.
So being popular just means having the luck to be on a platform that is doing well.
Usually most programming languages fade way when that platform stops being relevant, as history has proven a couple of times.
Alan Kay's "The Early History of Smalltalk:"
http://gagne.homedns.org/~tgagne/contrib/EarlyHistoryST.html
This is how I felt about Racket, and I felt I had to do an opinionated pruning before I could use it for teaching, but the problem is that the docs don’t have such a boundary for learners so you feel tempted to rewrite the docs.
? show (map [[x y] [output se :x sum :y #]] [a b c] [10 20 30])
[[a 11] [b 22] [c 33]]
Racket itself arose from building a cross-platform toolset to support building those programming tools. It also became a toolset and testbed for PL research.
I appreciate what you're saying about Racket having a big library. I was recently talking about how to to teach Racket to experienced programmers. I'd put them down in front of a full `#lang racket/base`, but start with teaching them only a specific subset of R7RS, and incrementally introduce a few more concepts and exercises to try for a day or more (things you can practice while writing your own real code, even if it briefly seems harder than something we told you that you could use before). If instead I just dumped the full Guide and Reference on experienced people, they'd become productive pretty quickly, albeit colored heavily by what they previously knew, and they might have a much longer path to becoming strong in some unfamiliar fundamentals.
BTW, another educational thing that Rackets `#lang` does is support students working through SICP use DrRacket. It emulates the particular older Scheme variant used in SICP, gives an IDE maybe a bit easier for new students, and runs on the student's own modern computer.
Common Lisp isn't even close to the most complicated language out there. Every
I also think pythonic and theoretically best performance might not need to correlate.
I would personally stick to for-loops for general but tricky code and leave things complications like nested comprehensions to places like the guts of libraries or classes that make the tradeoff to have simplified externals.
(for example argparse - very nice externals, tricky tricky guts)
Okay.
> it's resulted in a simpler, easier-to-use language.
Does `print(len("ẅ"))`[0] still produce a value (2) that is neither the number of characters (1) nor the number of bytes (3)?
0: "print\x28len\x28\x22\x77\xCC\x88\x22\x29\x29"
Well, yes. To be fair, it's not like any of them make a secret of the fact that they're mistakenly counting unicode code points instead of characters.
Consider, for example, the wonderful piece of writing in the answer to this question.
https://stackoverflow.com/questions/1732348/regex-match-open...
How many characters do you suppose are in this string?
.
"TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚ N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ"
.
And what should Python tell you the length of this string is?
Unicode is complicated in some ways because the domain it is dealing with (representing all possible human written communication, basically) is complicated. Unicode is pretty ingenious. It pays to invest in learning about it, rather than assuming your "naive" conclusions are what it "should" do (and unicode's standard docs are pretty readable).
Unicode does offer an algorithm for segmenting text into "grapheme clusters", specifically "user-perceived characters." https://unicode.org/reports/tr29/
It's worth reading that document when deciding what you think the "right" thing to do with "len()" is.
The "user-perceived character segmentation" algorithm is complicated, it has a performance cost... and it's implemented in terms of the lower-level codepoint abstraction.
Dealing with codepoints is the right thing for most platforms to do, as the basic API. Codepoints are the basic API into unicode.
It's true that they ideally ought to also give you access to TR29 character segmentation. And most don't. Cause it's hard and confusing and nobody's done it I guess. It would be nice.
If you want to know "well, howe come codepoints are the basic unicode abstraction/API? Why couldn't user-perceived characters be?" Then start reading other unicode docs too, and eventually you'll understand how we got here. (For starters, a "user-perceived character" can actually be locale-dependent, what's two characters in one language may be one in another).
It is specifially a abstraction that is not useful.
> It's worth reading that document when deciding what you think the "right" thing to do with "len()" is.
Technically not - the right thing to do is return the number of characters[0] - but the character segmentation parts are worth reading when deciding how to decode UTF-8 bytes into characters in the first place, so the distinction is somewhat academic.
> a [character] can actually be locale-dependent, what's two characters in one language may be one in another
[citation needed]; ch, ij, dz, etc are not examples, but I'm admitted not exhaustively familiar with non-latin scripts[1], so I would be interested to see what other scripts do.
0: or bytes, but that's trivial
1: Which is why I hate Unicode; I'd prefer to pawn that work off on someone else and just import a library, but Unicode has ensured that all available libraries are always unusably broken.
> 0: or bytes, but that's trivial
In what encoding? The utf-8, utf-32, and utf-16 encodings of the same string are different numbers of bytes.
"\xC4\xAC" is two bytes regardless of whether you interpret it as [latin capital i + breve] or [hangul gyeoh] or [latin capital a + umlaut][not sign] ("Ĭ" / "곃" / "Ĭ").
~$ python3
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> len("ẅ")
1
>>> "ẅ".encode('utf32')
b'\xff\xfe\x00\x00\x85\x1e\x00\x00'
Python 3.7.3 (default, Mar 27 2019, 09:23:32)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> len("ẅ")
2
EDIT: Your "ẅ" doesn't seem to be the same as the OP's "ẅ", although they look the same at first glance.
>>> "ẅ".encode('utf-8')
b'w\xcc\x88'
>>> "ẅ".encode('utf-8')
b'\xe1\xba\x85'
>>> import unicodedata
>>> w1 = "ẅ"
>>> w2 = "ẅ"
>>> unicodedata.name(w1)
'LATIN SMALL LETTER W WITH DIAERESIS'
>>> unicodedata.name(w2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: name() argument 1 must be a unicode character, not str
>>> unicodedata.name(w2[0])
'LATIN SMALL LETTER W'
>>> unicodedata.name(w2[1])
'COMBINING DIAERESIS'
> the number of characters should be in a string isn't always clear-cut.
This is why I use examples from latin-with-diacritics, where there is no ambiguity in character segmentation.
Still, to the original point, I think this is more of a criticism of Unicode than of Python. It seems to me that the answer is to not use combining diacritics, and that Unicode shouldn't include those.
True, although it's more specifically a criticism of Python for using Unicode, where these kinds of warts are pervasive. See also "\xC7\xB1" (U+01F1 "DZ") which is two bytes, one code point, and two characters with no correspondence to those bytes.
> the answer is to not use combining diacritics
This doesn't actually work, sadly, because you can't represent eg "f̈"[0] without some means of composing arbitrary base characters with arbitrary diacritics.
0: If unicode has a added a specific NFC code point for that particular character, then that's bad example but the general point still stands.
Keeping the humor-ish stuff aside, actually taking CL here as example it's someway ignorant. That's doesn't make any sense to me. Seems that all that opinion are based in nothing more that nothing when it concerns about Common Lisp literature experience and personal experience with the language.
Some languages are popular, but they being popular doesn't proofs that they are good languages. This is a fact based on the type of problems most of the people are trying to solve.
CL being big it's some way comparable to C++, but for big problems, sometimes we need complex tools. CL is not for kids, neither C++. Common Lisp it's a language for hackers.
There are a bunch of Common Lisp implementations which are smaller (CLISP), can create small applications (Lispworks), can be embedded (ECL) or can compile to smallish C code (mocl).
I think the main problem is there's too many implementations of the same thing in CL, so devs basically have to agree on a very specific subset of tools for every project, which adds a fair bit of overhead.
https://roswell.github.io/ https://github.com/fukamachi/woo https://github.com/fukamachi/ningle
p.s. Not the author, but a big fan.
I personally wonder if CL's "computer program as an image" instead of the Unix process-oriented architecture will see a comeback in this modern era of virtualization and rump kernels and containers. In theory a Docker image for an SBCL program would be just the Lisp image itself plus the small handful of standard POSIX libraries on which the Linux version of SBCL depends (e.g., libc, libm, libpthread, etc.)
https://github.com/bakpakin/Fennel
It's a lua based lisp. Technomancy of clojure fame has taken a shine to it and become a large contributor.
I'm not familiar with this. Anyone care to explain?
That's a good example. When an error is signaled you're able to modify the program (via recompilation of functions, changing of values) and restart at a point you select. There are other things you can do, but this is a particularly powerful option.
the goal was to unify several similar Lisp dialects as a Maclisp successor
For example, consider the new for loop syntax
for (auto item: a().b().c()) ...
C++ does have a lot of weird, inconsistent warts that make it tricky to learn/use everything. Which is why most people who advocate to use C++ talk about modern C++ which is C++ where you stay away from those warts.
No explicit link with the tragedy of the commons was implied. Implicitly, of course, it applies - there are many participants, each of which is aware of their own gain and unaware of how their actions cost everyone else. The result is that when they all get their changes in, the result is terrible for everyone.
For another example, C++ is a terrible language, but it contains many good ones.
Okay, but the article says:
> Adapted from a 2015 es-discuss thread. “Common Lisp” is not the topic. It serves only as one of many illustrative counter-examples.
So if "Common Lisp" is not the topic, and the tragedy of the commons is not the topic, then it seems the title is not very well chosen.
I don't wanna be -that guy- but ... github lists 5 contributors on that repo. "his" is doing a lot of work in your claim there.
But even with the all those 'libraries' COMMON-LISP package has just 978 external symbols.
1> (len (keep-if [orf boundp mboundp fboundp] (package-symbols 'usr)))
1713
That doesn't count any structure types or their slot names, FFI types, and local macros involved in syntaxes like awk and such.
I would say that Common Lisp shows amazing restraint, given its scope and number of people involved.
But they did not achieve widespread adoption.
Widely used languages need to apply to a wide range of use cases. For example the fat arrow(=>) class syntax that binds this.
class Foo {
bar = (baz) => {
// do something
}
}
That's always the issue with small, clean implementations. They work when they only apply to one domain. Once they need to be used for different things, they then need to add new features. It is no different for software, blenders, bicycles etc. Once something is used for multiple domains, it now needs to become larger and more complex. Its basically a rule of nature.
That's always the issue with small, clean implementations. They work when they only apply to one domain.
Sorry, but this is pure BS with regards to Smalltalk. I worked for a Smalltalk vendor for 5 years, then consulted in the language for several years after that. I had a front row seat to these issues. Having a small language does not make it suitable for only one domain. The primary reason for Smalltalk not achieving widespread adoption was due to a series of decisions which, in retrospect, were highly anti-adoption. It started with no operator precedence in the language design, which acted to alienate engineers and scientists. Then the industry went through a "we're the boutique language of the Fortune 500" phase with $5000 and $10000 per-seat licenses.
It is no different for software, blenders, bicycles etc. Once something is used for multiple domains, it now needs to become larger and more complex. Its basically a rule of nature.
That just doesn't work for software. The reason why Python is so popular with certain scientific communities isn't at all specific language features to support them. It's general utility plus specific libraries. The reason why Smalltalk got popular at one time in finance and energy trading didn't have anything to do with specific language features or libraries. It was purely rapid development!
There is something which a small language can be prone to: fragmentation. It was so easy to roll your own Smalltalk. This, combined with a toothless language standard meant that the community got balkanized into multiple communities with incompatible code.
> But they did not achieve widespread adoption.
> Widely used languages need to apply to a wide range of use cases.
A long time ago, an old-timer, even more old-time than me (and I did CS home works on punch cards in PL/I...) was a C fanatic, and was dragging his mainframe colleagues into Unix, said it best: "C does not get in your way."
He (and I) came from an era when eventually pretty much every language eventually "got in your way" and you had to learn the assembly language calling and linking conventions and implement a few nuts & bolts to get your job done. C was the first popular language that didn't eventually knee-cap you like that. That is why C won and Pascal was an intense but passing fad.
To me, that is the essential message for language designers, more important than "keeping it small". Stay out of the programmer's way. Of course, there are different ways of staying out of the way.... Python's "duck typing" is a way of keeping the type checker out of your way. Rust's "unsafe" is a way of keeping the borrow checker out of your way when you really, really, need to carve with your sharpest knife.
So I would argue that "applies to a wide variety of use cases" simplifies to: does not get in your way no matter what your application is.
As an aside: The old timer was a role model to me in one important aspect: Learning to recognize progress when you see it. He not only lived through the transition from core to solid state memory, he lived through the transition from transistor logic to integrated circuits. But he was evangelizing C and Unix when many his age thought the list of interesting programming languages had len()==3: FORTRAN, COBOL, assembly. I took that lesson to heart, and try very hard to answer the question: "Is this progress, or is this just different?".
Kotlin's another language that looked at say Scala (which is an immensely powerful language, but easily lets you get lost in abstractions), decided "We don't need that feature unless it actively helps remove friction when writing programs", and ended up with a lot more real-world programs written in it.