I wonder about how Common Lisp can be this fast however. I would like comparisons with languages like Python, JS and Lua.
I think that's too bold a claim. Your code might be too slow in any of these languages because you made poor algorithmic or data structure choices but once you have made the best possible algorithmic and data structure choices you may reach a point where you can't express the program with the choices you need and get the desired performance in some languages.
At that point, in the immediate present, you have to choose the tool that actually gets the job done. It doesn't matter if C++ could totally hit your numbers "once that LLVM optimiser bug gets fixed" or if Java could do it "when they port the new JIT to our platform", you need the thing that works now.
Over the medium term, language evolution makes an impact here. Once upon a time C++ didn't have move semantics. Obviously in some cases your compiler can move as an optimisation, but in other cases if you can't ask for one explicitly you're often out of luck. However C++ has a specific performance goal ("Leave no room for a lower-level language") that guides them to find a way to get great performance, and they changed the C++ language to fix this. If your ideal solution is slow in C++ then the C++ committee ostensibly wants to fix that. Just maybe not this year. In contrast for Java performance is not a primary goal and sometimes Java is OK with worse performance. In this particular essay it happens for RAM consumption. This problem shouldn't run out of RAM on your mid-1990s server at the scale demonstrated, but the Java solution would either run out of RAM or spin up the GC mid-solution (and thus ruin speed).
No argument that "I'll rewrite it in Rust/C++" is just as much the wrong first instinct as "I'll hand roll the assembler". But if you're under-performing with the right algorithm you need to either spend money on more (faster/ bigger) hardware or on rewriting in the language that might buy you that last drop of performance.
Python, JS and Lua are not in the same league of AOT and JIT compiler technology available to Lisp, even with all the money spent on stuff like V8, JS JIT cannot compensate for the fact that they lack the ability to provide the required semantics to AOT code with good enough performance to write a Xerox PARC graphical workstation (see Interlisp-D).
It is no wonder that Julia, a Lisp like language with Algol syntax, is the only modern dynamic language on the HPC club.
https://www.hpcwire.com/off-the-wire/julia-joins-petaflop-cl...
I didn't use the language but the overwhelming community consensus I get is that performance is out of the window once you truly embrace dynamic (type-unstable in julia parlance) code. Julia performance depends heavily on the compiler being able to infer a single type for variables in the hot loop so that it can specialize the used functions to it then get out of the way. Julia is not a Fortran-speed python, it's a language that can be _either_ Fortran _or_ python, depending on how you write it.
Lisp influences on julia are more for semantic power than performance.
https://www.researchgate.net/publication/267008293_Fast_floa...
The point being what dynamic homoiconic languages are able to achieve when the community actually embraces the push for performance instead of hand waving and rewriting everything in C.
If the JavaScript community had the same attitude as Python one, surely we would still be doing HTML 4 without pesky SPAs.
My experience is that small programs like this are usually at least an order of magnitude faster in SBCL compared to python. At least if the python code spends some time in actual python instead of offloading to C.
Out of curiosity, what things are you referring to?
Things like proper static modules so that you know what is being called so you can elide lookups at call-time. That is a pretty big omission, TBH.
For a long time (still?) python had a horribly inefficient value representation in memory. Now, integer tagging is probably no fun to add as an afterthought, but it shouldn't have been an afterthought.
Exception handling is sloooow. And iterators use exceptions to signal that younhave exhausted the collection. Because iterating over a complete collection is exceptional.
I understand the need to keep cpython simple, but some of these things ARE simple and have been well understood since a decade before python 1.0.
StopIteration is special-cased and has negligible overhead. See here (https://www.python.org/dev/peps/pep-0234/) for why exceptions are used.
That rationale for using an exception is only valid because CPython procedure calls are expensive. A proper optimizing compiler will have a hard time rewriting that implicit try/catch into a goto.
I think Dart nailed iterators in this regard: no special values, and a protocol that is transparent to the compiler, which has the benefit of requiring no special cases at all. In the current state of cpython, of course, that falls into the "2 expensive procedure calls that can't be inlined because everything can be mutated at any time" type of a problem.
I think the whole point of python is to make everything dynamic. It seems like this trade-off is voluntary, i.e. it is a feature, not a bug.
Stop-the-world GC doesn't stop you from using Java for high-speed apps, unpredictable stalls do. It doesn't matter if your GC takes 2 seconds to run fully as long as it doesn't kick off during your hot loop, and the opposite is true; there will never be a low enough overhead "magic" GC that can be run in the background that may cause a microsecond stutter on your finely tuned trading algotirhm, it has to be explicit.
A fair few trading system job ads seem to be for Java. I also wrote code that faced a Java server at various banks. Seems to be fast enough for certain market related applications.
Secondly, the figures that are often quoted for how long the program must be stopped are for the "happy path" where enough memory can be freed that the program can continue without issue.
In practice, what often happens with large heap sizes is that the GC starts lagging behind because there's not enough memory head-room. In these cases, the GC stops the world for much longer, because the program hangs on a memory allocation that cannot be satisfied until the GC completes.
This primarily affects programs that have high rates of allocation. You can avoid this in Java with effort, but you end up with very non-idiomatic code.
I guess Oracle, IBM, Azul, PTC, Microsoft can provide the necessary data about their pauseless GC implementantions to any relevant customer.
Unfortunately, CL has started to show its age (case insensitive, different namespaces for variables and functions, weirdly named stuff, etc.) and Chez Scheme doesn't have a lot of libraries.
In my experience, Chez Scheme is very small, blazing fast, and VERY high level. It is a great platform to use as a target language for compilers / interpreters. Racket has chosen to go this route, for example.
Enforcing the "strictly one meaning per symbol" paradigm seems pretty self-defeating to me, since even in a Lisp-1 users can - and eventually will - create their own namespaces, implementing a Lisp-N themselves.