The Tragedy of the Common Lisp: Why Large Languages Explode (2019)
erights.medium.com
erights.medium.com
Here you go, three reasons why CL is less popular than it could be that have nothing to do with the size of the language, and these are just examples. A language doesn't need to be big but it needs to offer what programmers need in one way or another. It's irrelevant whether that's part of the language or part of an ecosystem.
For any sufficiently large organisation or lasting use cases a language is almost not at all about writing it but about reading it. Any subsequent modifications after the initial creation will require reading and hopefully understanding what is happening. Even if the code is properly documented the code is always the final specification and if you can't read, you can't understand what your organisation is really doing.
In that sense, adding a feature that is possibly detrimental to reading the language means detrimental to the whole language and its users.
This is very true, and is one of the reasons our workplace moved to Python. Readability is very, very important.
In environment with less "language X is shipped by default", it had much less impact except for crossing over of developers who did work with it on other platforms (which is how Turbo Pascal/Delphi kept on significant marketshare in PC space for a long time).
C domination really was linked more to network effects in my experience.
Naturally they ignore the size of ISO C and its implementation specific nuances, the set of extensions each compiler provides, every system that matters also brings POSIX for the ride and it isn't fully portable with plenty of room for interpretations, the amount of build systems to rely on, the tooling to actually write safe code,...
But hey, the K&R C book is small.
EDIT: or just use redis lol
I don't buy that Common Lisp is too large. I would say the opposite is true. The base language is rather simple and what you mistake for language are just standard library.
What is complex is it being Lisp language which means you need to adjust your brain structure to a different level of functioning and focus when reading code.
**
But languages do get too large.
I think the reason languages get too large is because developers, as soon as they get some experience, start demanding more features. Maybe they want to have more functional experience in an object-oriented language, or maybe they want async in one that was not designed for it.
These developers form a core of users that enjoys being able to absorb these features over time as the language evolves.
On the other hand, new users need to learn it all at once.
There will come a point where new users will just look at horrendously complicated language with vast specification and will just say "Fuck it, I don't want to spend so much time to be able to do anything, I will program in this neat new other language that is simple to start and already has most of what I need."
I have been programming Java for the past 20 years and I do enjoy new features. And I would like couple more features myself.
But I am also tech lead for a lot of new developers which I see they struggle finding sense in what is going on with Java. Not just the language, but the multitude of frameworks and other paraphernalia supposedly needed to make a working modern application.
Rather than trying to understand it they are giving up and revert to copying and pasting code that works. And wait for somebody with experience to catch obvious problems on code review.
I think this is not sustainable.
Often, though, I just feel like Doctor No.
I got new appreciation when it became my responsibility to care for productivity of a lot of people.
Often people bring in new toys just to have some change in their work environment. I select for people who like to learn new things so why would I fault them for it?
But I also see people don't understand how disruptive it is to productivity. People mostly see only gain side without seeing the outsize cost side of the problem.
Even management doesn't seem to understand it. One time they brought me into large project that had trouble delivering and huge reliability problems and they told me my first priority problem is going to figure out how to upgrade Java from 1.6 to 11.
My first question was "Why do you guys think it wasn't possible to deliver reliable applications with Java 1.6?"
I wonder how much language inertia can be overcome by surveying what's been implemented multiple times and ranking them.
One imagines a very minimal language with multiple implementations, say, of asynchronous I/O or common data structures. Frequently-implemented features would be good candidates for thoughtful inclusion.
In a proper Lisp you can take anything that in a regular language would be a language feature and implement it as a library.
Instead, what Lisps seem to excel at is allowing you to implement syntactic sugar. For instance, you can easily write CL macros that give you JSON/hashtable literal syntax, list comprehensions (pretend LOOP didn't exist), foo.bar to access struct elements, and infix notation - those things that people think of as being "part of the language" but don't give you any new capabilities (e.g. closures, a MOP, multiple-value returns, conditions).
I should have been perhaps a bit more clear, though T9 makes that hard. I mean more in the sense of standardization. In some ways, LISP makes a better case for what I've suggested simply because you can modify the syntax with such relative ease. If you see patterns emerge, it makes it easier to standardize a syntactic construction to make it less idiosyncratic.
Like in Scala it could be from a trait (and traits can define variables), or inheritance, or implicit values.
Compared to Rust or Go for example where I can read the docs for a struct, and easily see where the different parts come from (and follow those up if I want).
Of course when they design a new language they try to incorporate all the features that developers demand at the moment in a way that makes sense.
But once the language is developed it is difficult to add new features seamlessly. The new features are necessarily impaired in some way by having to be compatible with existing environment.
Rest assured, wait couple decades and the same will happen to Rust.
That is how you get AOP frameworks, code generators based on EMF, XML driven frameworks,....
Especially with things like Spring Boot, most of the behavior is driven by sprinkling annotations, but you can not use your understanding of the syntax and semantics of the Java language to understand what they do. It's just thrashing around trying different combinations of annotations and configuration until something works.
And God help you if you need to debug what's going on if it stops working on some corner case.
But that very important valuable feature was squandered over time with the biggest offender being Spring.
"To be fair, you have to have a very high IQ to understand Common Lisp..."
You need to be willing to explore and be inquisitive.
You need to be able to read more difficult books (than entry level) like Let over Lambda or On Lisp and try to understand every sentence and every example and not just the superficial side but also as questions like why this way, when this is useful, what are limits on application, etc.
I guess intelligence correlates but I know a lot of very intelligent people who are too lazy to be bothered to understand things deeply.
It would be fair to say you need at least some IQ to be able to do this. Probably about 100-110 is a limit below which people just are not able to understand complex abstractions (which is a little scary if you think about it, because that is over half of the population). But I wouldn't say you need to be very intelligent (like 140 and upwards).
Because you can't switch languages in a system for each feature, so if you want features A, B, C, D, and E, and the subset (A,B,C,D) has more value than (D,E), the language that has A,B,C,D will be preferred to the one with D,E, but you’ll still want to add E.
Even if you do get proficient at this, there is still the problem that reading CL is not enough - it's CL + custom macro/DSL systems per project. Given a bunch of different projects and teams in an org, moving people around is easier with other languages because that second part tends to be less complicated. We have one team that decided to go all in on customizing the common frameworks, inventing their own with bespoke annotations and patterns so now they are basically isolated from the rest of the org and others dread working on projects that involve changes in that codebase.
How is this any different from building an overly convoluted architecture in any other language where you have layers upon layers of objects, getters, setters, proxys, callbacks, VisitorFactoryFactory's, configuration files, multi-level build systems, virtual methods, type system hacks, generic functions?
I've been programming in CL for years now, and I have more of a problem keeping my non-macro code from spiraling out of control than my macros. What makes macros any more than just a thing you have to exercise discipline with, like "normal" code?
That is true and expected.
Because if you "got" CL it means you start any larger application by writing a perfect language in which to write that application.
But with power comes responsibility.
And this is where a lot of Lisp projects fall short.
It is much more difficult to make a mess in a poor language like Java where you can basically ctrl-c/ctrl-v your solution than it is in super powerful language like Lisp that gives you no guidance on how to structure your application. Especially if you also have intelligent people on the project.
Because intelligent people can make even more complicated constructs with which to shoot their feet.
I wish every CL project had one guy with a long beard that can bring wisdom on how to deal with an infinite power of Lisp.
CL honestly isn't a large language by modern standards. The Common Lisp standard library is tiny compared to modern C++, Java, Python, Ruby, Rust, Haskell, APL, or whatever other mainstream language one is working with nowadays. CL might have been "batteries-included" in 1994, when it was standardized, but the idea of "batteries" that a language can and "should" include has been expanded hundreds of times since then.
I understand why people might get nostalgic for simpler times and simpler computers, I'm partial to the rose coloured glasses view of the past myself.
Its been interesting using C# for the last 15 years through that perspective. Microsoft seem intent on folding every cool language feature from every programming language into .NET and it does feel overwhelming, but in the end most experienced programmers have a style so we naturally settle on a subset of language features until we need something new. I really wish somebody had implemented the async patterns in mainstream languages in the 1990s for example.
I really don't think nostalgia for simpler times is useful for solving todays problems, I much prefer C# to Smalltalk or lisp for getting stuff done today.
The language itself has adopted just about everything interesting from contemporary languages, from pattern matching to top level statements allowing you to write a hello world or simple scripts that rival python for simplicity.
It’s performant, asp.net is at the top of the tech empower benchmarks, c# is only just behind c++ on the compiler benchamrks game leaderboards (although if you actually read the code i question the value of the deceptive impls there, e.g. the regex redux example is just calling a c library rather than using native c# facilities).
It’s not as memory efficient as go, by about 5x-10x worse but it’s comparable to java. The clr has a much faster allocator than go (although go developers just pre-allocate up front as an idiomatic optimisation).
Since dotnet 5.0 the developer experience is very decent across all platforms. 6.0 with its further enhanced hot reload etc looks even better.
It’s cross platform - apple tv apps or apple watch, ios, android, windows, linux, macos, they’re all first class citizens.
Whether you want to write 3d games (ubiquity), machine learning, web apps, desktop apps, mobile apps, wasm / blazor, it has 1st class support for them all.
And yet, i can’t help but feel common lisp is probably a better language.
The developer experience in CL is superior - repl driven development combined with structural editing and easily accessible ast manipulation just put CL (and other lisps) beyond what more static languages can ever hope to offer in terms of developer ergonomics.
Both could be added to common lisp though - the vector arg list would be a short reader-macro and my lisp-foo isn't yet strong enough to say how to implement the keyword as a function thing but i'm 100% confident it can be done...)
You'll find 2 C# regex redux programs when you look at all the regex redux programs —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
— or look at all the C# programs —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
One uses "native c# facilities" —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
— and the other uses PCRE2 —
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
There's nothing "deceptive".
You are explicitly told to look at the other programs.
The language is described in about 40 pages of said standard.
Common Lisp 1994 (2004 edition): 1153 pages
C 1999: 554 pages
C++ 1998: 776 pages
C++ 2011: 1356 pages
C++ 2014: 1375 pages
C++ 2017: 1618 pages
C++ 2017(2018): 1620 pages
In comparison, the reference manual for Ada 1983 had only 336 pages and the Algol 68 report had only 146 pages, these 2 being considered very large languages at their time.
It is true, however, that the Algol 68 report would have required a double length to include enough explanations to make it intelligible for everybody.
I have a copy of MIL-STD-1815, the first Ada spec, and it's 242 pages!
At that stage, Ada had already won the DoD competition, but until it was published as a military standard a few more features were added and some sections of the specification were rewritten, for increased precision.
In any case, the June 1979 Ada version already included what I consider as the most useful Ada innovation, even if it is just a small detail: allowing underscores inside numbers, for better readability.
It always annoys me when I have to use some programming language or software tool that happens to miss this feature, even now, more than 40 years after it became known that this is a good idea.
The rationale for this surprising choice was that a number starting with an underscore could be interpreted as an identifier.
??!! Why would anyone want to start a number with an underscore? Underscores are needed only inside long numbers.
The syntax should have allowed underscores only inside numbers and there would have been no problems with backward compatibility, as such numbers would have been illegal before.
The single quote cannot have been easier for parsing as the new use must not be confused by the parser with character literals, which can appear in the same context.
The underscore was first introduced in a programming language by IBM PL/I (in 1964, when it was still named NPL) with the precise purpose of improving the readability of identifiers.
Ada has extended this use to also improve the readability of long numbers.
Why C++ has chosen a different character, which is normally used for completely different purposes, without having any decent justification for such a baffling decision, I cannot understand.
https://dl.acm.org/doi/abs/10.1145/3386323
Since D popularized it, many other languages added it. Even C is finally doing it (although using a ' instead of _, which I do not understand the rationale for).
Had you added Java, .NET or Python, with their standard libraries, C++ would look like a toy language.
POSIX 2017 has 3951 pages !
Adding those to the current C standard would make about 4700 pages, a little more than 4 times the size of the Common Lisp standard.
C++ is perhaps the most prominent example of this, I’d consider it to be at least three different languages merged into one at this point. JavaScript has also accumulated some weird stuff over the years, like var, == and (arguably) a class-based OO system that doesn’t interact very well with the rest of the language.
One “solution” is using only young languages (e.g. Rust is very large, but young enough that most of its features are still useful). I personally like switching technologies every few years, but as a field, we can’t keep doing that forever. The other idea is using a language that’s more or less frozen, like C.
R6RS was considered too big, and too influenced by one of the players, by a large amount of the community.
So, the big player distanced themselves a bit, renaming themselves to Racket, etc.
But for the next standard, two were produced. R7RS-small and R7RS-large. A core language that stays the same, that everyone has to use to call themselves a "Revision 7 Scheme", and an extended standard language that is useful, for those wanting to provide a "batteries included" kind of environment.
And no one is obliged to use every single feature from C++.
Size isn't all, what it really is is comprehensibility. Perception of its essence. If the foundations are solid, if you don't have a feeling that it might just be an ad hoc contraption mucked up by the devil himself to make people suffer, if it feels like you don't have to fight it every step of the way, if you don't get nauseous just by thinking about it, you just might as well stroll through it, no matter the pathwalk length.
Maybe a better question is, what's really extraneous? You know, with CL, I can't immediately tell. The Scheme folks have an idea. But Scheme and CL still look like a pair of siblings nonetheless.
Or maybe an even more fundamental question is: "is it well-designed"? Or maybe "does it provide depth without imposing unnecessary complexity"?
There are plenty of questions, but the size one is quite boring, honestly.
Nevertheless Common Lisp, C++, let a lone Java (I know nothing about PL/1) at least are reasonably easy to grok, read and write. At the same time LaTeX seems a language of extraterrestrial insectoids as soon as you go beyond composing formulae and substituting text in pre-made templates and and want really custom typesetting. I once felt like I would like to actually learn it, tried it and gave up completely quickly - it really feels unapproachable.
Before `class`, people built ad-hoc class-like constructs based on prototype inheritance. Often in slightly different ways, with different constructor choices and different implementation details.
For larger code bases in team environments class is a really critical standardization, otherwise you can easily end up with a mess.
It's also important for tooling, because IDEs can much better understand what's going on and present information better.
Now, if Javascript really needs things like private class members and a lot of the other difficulties around classes is another question.
Lisp is like being over a small table really drunk in the middle of the oceanic storm. The points of reference are moving all the time and the documentation system basically hates you.
apt install slime sbcl
Always stubborn like a horse
This true and really nice, as long as long as you are Okay with copy-pasting everything mindlessly without serious customizations. It's like being a wizard who would invoke some specific spells by reciting specific phrases in an ancient language without having any idea of the actual language and the nature of the magic while knowing them would let him achieve much much more, easily and elegantly.
Generics getting added is OK. I mean, almost every veteran Gopher I've talked to agrees that there's been a couple of times, maybe, that they've needed generics and not had them, and that interfaces work fine for 99% of cases. Everyone outside the language insists that it needs generics. But sure, let's add generics for those few times we need it [0].
But then "error handling is too onerous", "there's too much boiler plate", "why can't I code in Go using a functional style?", "Go needs to look more like Ruby", etc, etc.
I really hope they keep Go simple and don't cave in to demands to add stuff to it. It doesn't need more syntax. It needs smaller runtimes, a method of determining whether two slices are using the same backing array, and a few other gotchas.
I can see a future where there's a "Go classic" forked from 1.17 (or possibly 1.18) that never moves forward, while the main Go gets ever more complex and Java-fied.
[0] Except, of course, that as soon as we have generics then every new gopher will insist on using them for everything, and not learn how to use interfaces well instead.
What I think will actually happen is that Go will continue increasing in size, generics will bring a new kind of fragmentation with people using libraries everywhere to replace the base constructs of the language, and in 10 years a new "minimalist" language to replace Go will appear, and the cycle will continue.
You see that happening with pretty much everything. At first the new thing is easy to learn, and then it slowly becomes as large and hard to learn as the thing it replaced, because accidental complexity wasn't such a big part of the original thing.
My understanding of Lisp 'history' is that most CL implementations were by large vendors charging silly licensing fees and they got undercut by C and Pascal vendors...
Another theory I have is that Lisp encourages the use of macros which makes sharing code more difficult.
Or maybe it's simply that the languages that built/were built on Unix and PCs became most popular as those systems were most popular (C, C++, Perl, Python, Ruby, etc...). And of course now the language of the browser (JS).
Or maybe a bit of everything. On the plus side, SBCL is pretty fantastic and people still do things with Lisp.
These two books explain how it’s done:
https://en.m.wikipedia.org/wiki/Object-Oriented_Programming_...
https://en.m.wikipedia.org/wiki/The_Art_of_the_Metaobject_Pr...
The fact that C++ classes are part of the C++ core langauge while CLOS is a library doesn't really change this aspect.
The big difference between C++ and CL in this particular instance is that you can’t write C++ without writing or using classes/structs. For example, using STL almost always involves using or writing custom comparator “classes”. Even if you choose to go with lambdas, the whole concept of a container is object oriented.
In CL, you can, if you want, stick with lists all the way down without ever touching defclass/defstruct at all. An equivalent abstraction level of the C++ code would only use integral data types. Whereas in CL those types are part of the language.
I have always loved a tech world with many programming languages, but I wonder if this might change in a few decades if software reliability becomes more of an issue. I don’t know much Rust, but I have taken a good look at Swift as being a possible direction for a universal language (REPL development, strongly typed with fast compilation speeds on M1, and a generally pleasant language).
I have a prediction: IT security will become such a larger problem in the future that programming languages and operating systems will go through a winnowing out phase, fewer choices, and those fewer choices will have more effort in producing provably correct software.
Common Lisp has little syntax (but complicated by macros), 900+ functions in the language spec, supports many programming paradigms, and yet I don't feel it as having become fragmented as a language.
Your winnowing idea is interesting, but would seem to suggest we periodically discard entire languages. A language's library ecosystem is often a critical factor in selection, which also makes it harder to give up.
The longer a language is actively developed, it seems there's a greater risk for the language to become fragmented. I wonder if this is ever a concern for language designers (I presume it is) and what sorts of decision might lessen the probability of fragmentation occurring in the future.
> I have always loved a tech world with many programming languages, but I wonder if this might change in a few decades if software reliability becomes more of an issue.
I don't expect that to happen unless the liability landscape or some other financial incentive changes. Do you see force that would drive an increased focus on reliability in the industry?
Adding new things to the language (e.g. closures, conditions, a borrow checker, first-class functions, immutable variables, multiple return values) results in the user having to keep more things in their head at once to use the language.
Adding new things to the standard library increases the size of the language distribution, but doesn't appear to increase cognitive load that much. The fact that CL has a `round` function doesn't really make programming in it any harder.
Adding syntactic sugar to the language can make it harder to read, and usually breaks backward-compatibility.
My impression is that most language "bloat" is in the standard library and syntactic sugar, not the language "core" - although I have little experience with C++. Any counter-impressions?
A bloated core language means that programmers now have a lot of baggage to deal with--particularly features that reduce performance. In my opinion, I do not think features that reduce performance should ever be added to a core language. The only features that should be added to a core language are ones that increase code readability and safety and reduce the amount of code you need to write to accomplish your goal.
> Bjarne was resolute in preserving his fundamental goals … People were telling Bjarne that you should turn C++ into a fully object oriented language… make it automatically garbage collectable… make all the member functions virtual… eliminate global functions… in other words they wanted him to design Java. He listened politely to them and kept maintaining the fundamental integrity of the language. [0]
[0] CppCon 2014: Bjarne Stroustrup "Make Simple Tasks Simple!" (https://youtu.be/nesCaocNjtQ?t=296)
I'm a big fan of JavaScript as being a simple functional language with tiny bit of prototype based OOP. I do like some of the newer features of the language - arrow functions, block scoped variables, destructuring...
But I'm not sure stuff we're adding now are really giving us any value: the ugly async/await that is leaking abstractions and needs a lot of fixes to support iteration, classes, static members, constructors, annotations, etc.
For me personally it adds nothing valuable to the language. It simply is there to make language attractive to C# and Java programmers. It clashes with current libraries and way we used to work with JS. Thanks to that, we'll probably see explosion of crazy complex tools from these languages being introduced into JS - ORMs, DI containers, code generators and dynamic proxies everywhere...
Most of our problems could be easily solved using functions, objects and closures. But now we'll have Spring Data JS and in few years we'll also get our SimpleBeanFactoryAwareAspectInstanceFactory.