Learning Common Lisp to beat Java and Rust on a phone encoding problem
renato.athaydes.com
renato.athaydes.com
[1] http://lush.sourceforge.net/ [2] https://common-lisp.net/project/ecl/ [3] http://www.ulisp.com/
There's this on the J wiki: https://code.jsoftware.com/wiki/User:Brian_Schott/code/feedf...
And there's this YouTube series for APL: https://youtube.com/playlist?list=PLgTqamKi1MS3p-O0QAgjv5vt4...
There is a series of videos of learning neural networks in APL cited by others here on this thread.
Pandas author, Wes McKinney, cited J as an influence in his work on Pandas.
Extreme Learning Machine in J (code and PDF are here too):
https://github.com/peportier/jelm
Convolutional neural networks in APL (PDF and video on page):
https://dl.acm.org/doi/10.1145/3315454.3329960
A DSL to implement MENACE (Matchbox Educable Noughts And Crosses Engine) in APL (Noughts and Crosses or Tic-tac-toe):
Which implementations are you comparing J to? To my knowledge, all of APLs, K, J are interpreted. BQN is compiled, but still very new. I also know that Dyalog was experimenting on a byte code compiler. I don't think there exists convincing benchmarks comparing those languages.
Python was already well known as a "scripting" and server-side web development language in the early 2000s, but it's commonly mentioned that it really exploded in the 2010s, where it was the implementation language for several scientific packages, most notably the machine learning eco system.
It seems that the language really found a local optimum that adheres to many different people across disciplines.
Preparing for Y10K? That's exceptionally long-termist.
- static binding
- closures (true)
- tail recursion
- garbage collector
s-expression or typing is a matter of choice, but, IMHO, if you lack one of the four previous items, it is not really a lisp.
I'm definitely not an expert in that area, but this list seems kind of arbitrary to me. Especially with s-expressions being optional, which are probably the widest-known feature of the language. According to that definition, Haskell is a "true" Lisp but at least 2 Lisps are not. That makes no sense to me.
I have encountered many functionnal languages when I was student (caml-light (the ancestor of ocaml), lelisp, gofer (a cousin of haskell), miranda, graal, FP systems, yafool).
The typing may be dynamic or static. The evaluation may be strict or lazy. They may have homoiconicity or a more suggared syntax. All theses choices are valid. These languages have in common the list of fundamental properties. IMHO, this list of 4 items encompass many aspects of SICP. When I evaluate a language, this list helps me understand the qualities and limitations of a language. For example, Perl5 does not have a true garbage collector. javascript does not have tail recursion. Knowing these limitations, I will not code the same way. In Perl5, I will take care of breaking unused circular data. In javascript, I will reorganise highly recursive algorithms.
Edit: And to be frank, while dynamic binding may be a horrible mistake in bigger projects, it sometime gives you exactly the easy way out that you may appreciate under time pressure. It's a classical case of "it seemed to be a good idea at the time".
Emacs Lisp will, I expect, never ever drop 'defvar' dynamic binding. It would break the entire world -- it's relied on far too widely to revoke. Having lexical binding alongside as we do now is probably sufficient.
I agree that it's useful to be able to locally override variables like deactivate-mark and case-fold-search. (This kind of thing makes tail-call elimination more difficult: any dynamically scoped variables must be restored when the "tail-called" function returns.) But there are some other such things in Emacs that can be similarly locally overridden and then restored, but aren't variables: (current-buffer), (point), and (mark), for example, which can be restored with (save-excursion ...). And it's common to have such locally-override-and-restore facilities without using linguistic dynamic scoping for it; PostScript has gsave/grestore, for example, which were copied by Win32 GDI SaveDC and RestoreDC, but that doesn't give C dynamic scoping.
I don't think SaveDC and RestoreDC are known to give rise to problems when "writing a prgoram for other end users that multiple people are working on".
I agree that elisp will never remove dynamically-scoped variables; it would break compatibility with all existing code. Even Common Lisp has "special variables" that behave this way.
It happens that in ordinary Lisps†, the function called by invoking a symbol does depend on the run-time value of a symbol (its function binding in a Lisp-2), and that's the sense in which an ordinary Scheme or (non-generic) Common Lisp function call can be said to be "dynamically bound", but that isn't the case in general. So even in that sense it doesn't correspond to the static/dynamic scoping distinction that they seem to be trying to discuss.
______
† I think this may be one of the points where Lush is atypical; I think its interpreter supports runtime rebinding of the function bindings of symbols, but its compiler doesn't. I'm not sure, though. I may not have used Lush this millennium.
When we "declare dynamic-extent" an object, the compiler may stack-allocate it.
The C language redefined "dynamic" from "stack" to "heap". If you look into the BCPL manual (one predecessor language that inspired Ken Thompson's B), it uses "dynamic extent" to refer to the stack, which C renamed to "automatic storage":
"[T]he extent of a dynamic data item starts when its declaration is executed and continues until execution leaves the scope of the declaration." (1967 BCPL Manual, 7.2)
https://www.gnu.org/software/emacs/manual/html_node/elisp/Dy...
https://www.gnu.org/software/emacs/manual/html_node/elisp/Le...
______
† Older versions of the Emacs Lisp reference manual mostly did not do this, except for one occurrence of "dynamic binding" in the "implementation of dynamic scoping" section.
It was definitely dynamically scoped, though.
______
* I'm assuming by "static binding" you mean static scoping; if you actually mean that the association between callsites and functions is statically computable, then it's not even true of Scheme.
Lush is uncontroversially a Lisp.
However, the part of this that's relevant to the point I was actually making is that Lush uses S-expression syntax, which is less readable than, for example, Python syntax.
This doesn't really scale well, but momentum has a way of making things scale. Especially when most of the popular libs of python are pseudo ports of other language libraries.
I think it caught on because the syntax is clear, the execution model has relatively few pieces of foot gun trivia to memorize, a really nice repl, and most importantly, almost no one doing exploratory work needs anything faster.
That is, your complaint is what I meant about it not scaling. It is terrible. But the momentum behind it is keeping it going, despite being laughably bad in that area.
That's...somewhat misleading. Python was well known for its scientific stack as well as server-side web development and scripting from the early 2000s (or earlier; NumPy, under its original name of Numeric, was released in 1996, BioPython in 2000, matplotlib on 2003, etc.).
In the 2010s, it became known for its machine learning stack, which was built on top of the existing, already solidly established, scientific stack.
It might seem stupid, but operator overloading and metaprogramming features make it fairly simple to emulate the syntax of other languages scientific users would have already been familiar with. Specifically, NumPy, SciPy, and matplotlib quite obviously tried to look almost exactly like MATLAB, and later pandas very closely emulated R. It's a lot easier to target users coming out of university programs in statistics and applied math who have been using R and MATLAB and teach them equivalent Python libraries. Trying to teach people who aren't primarily programmers to use Lisp is going to have a much steeper learning curve.
It really didn't explode in the 2010s, either. You're thinking of Facebook with pytorch and Google with TensorFlow making it dominant in deep learning, but the core scientific computing stack goes back way further than that. As for why Google and Facebook chose Python rather than Lisp, I think it was just already one of their officially supported languages they allowed internal product teams to use. Lisp was not. Maybe that's a mistake, maybe it isn't, but it's a decision both companies made before they even got into deep learning.
Clojure is definitely fast enough for everything I've done professionally for six years. But Common Lisp, while having plenty of rough edges, intrigues on the basis of performance alone. (This is on SBCL -- I have yet to play with a commercial implementation.)
The application was receiving binary messages from the exchange over multicast, rebuilding state of the market, running various (simple) algorithms and responding with orders within 5 microseconds of the original message, at up to 10k messages per second.
With SBCL you can write a DSL and have ability to fully control the resulting instructions (through vops). It is just the question how dedicated you are to writing macros upon macros upon macros.
I used this to the fullest extent and the result was as good as any hand optimized C/assembly.
For example, the binary message parser would receive a stack of complicated specifications for the messages in the form of XML files (see here if you are curious: https://www.gpw.pl/pub/GPW/files/WSE_CDE_XDP_Message_Specifi...), converted XML to DSL and then, through magic of macros, the DSL was converted to accessors that allowed optimal access to the fields. Optimal here mans the assembly could not be improved upon any further.
Large parts of the application (especially any communication and device control as it was done with full kernel bypass) was written in ANSI C but the Foreign Function Interface makes integrating it into the rest of application a breeze.
I write all of this, because I frequently meet complete disbelief from people that Lisp can be used in production.
I personally think it is exactly the opposite. Lisps offer fantastic development environment. The problem are developers who are mostly unable to use Lisp for work effectively, I guess due to too much power, freedom and complete lack of direction on how to structure your application.
The largest influence were definitely LMAX articles and Disruptor pattern.
SBCL was comparatively easy. Basically, if anything caused problems I just moved it to C and called using FFI. Think in terms of writing a C program but using pieces of assembly when C is not enough for some reason.
Even though a large part of small pieces was moved to C, the application still felt like Lisp. It just orchestrated a large library of utilities to do various things. I still had fully functional REPL, for example.
The hard part of the project was full kernel bypass. After the initial setup, the application stopped talking to Linux kernel except for one CPU core that was devoted to running OS threads, some necessary applications and some non-performance-critical threads of the algotrading framework (like REPL, persistence, etc).
All except for one core were completely owned each by a single thread of the application and never did any context switch after initial setup.
I hadn't appreciated that level of thread isolation was possible in SBCL, but of course it makes sense. Presumably you had some kind of instrumentation to let you know if a stray syscall had slipped in?
Don't know if you can call it instrumentation... I wrote an extremely hacky patch for the kernel to detect when any piece of kernel is running on anything than core 0 after certain flag was set.
Remember, it is not just syscalls. Even something as simple as accessing memory can cause switch to kernel to resolve TLB entry if you don't set up your memory correctly to prevent this from happening.
Yeah... I know. Now a bunch of people will come and explain how this could be done the right way. I just didn't care at the time to invest more time than necessary to get this right.
I'm very interested in the kernel bypass technique to talk to the hardware, I'm assuming it was the NICs ring buffer.. how was this achieved in userspace?
And that is how plenty of polyglot devs can enjoy their favourite language.
If some big shot corporation with weight in the industry doesn't shove it down the throat of devs, then it isn't possible.
And regarding Lisp, most tutorials keep ignoring that arrays, structures, stack allocation, deterministic allocation,... are also part of Common Lisp.
Well... don't give my application as an example.
I have worked around the GC problem by using CL as kind of compiler for the application -- I built high level DSL that was then compiled to binary supplemented with small blocks written in ANSI C.
If you think about it, SBCL already does this and so does JIT in JVM.
I will give you an example:
Before the order goes to the exchange, it needs to be validated against a set of rules. Like "do we have enough money to run this order?" or "How much of allotted limit this algorithm still has?" or "Is volatility on this market within bounds for this algorithm to be used?"
The rules were written in Common Lisp DSL, then Common Lisp code converted it to a decision tree, then optimized decision tree, then compiled optimized version of the decision tree to a binary function. That function itself had no longer anything to do with Common Lisp, it could have just as well been written in ANSI C.
Then Common Lisp code wired these functions to form the application.
While most of the application code was actually Common Lisp (and some 20% of ANSI C), if you were market order and observed what instructions are handling you, none of them would actually be Common Lisp.
After the initial setup, the Common Lisp application was running on only one core, and the constructed binary took over all over cores and was running without garbage collection on memory regions allocated outside of Common Lisp system.
I hope this description makes sense... I have never before or after met an application that was built this way. As far as I know it is only one of its kind.
So efficiency was an important consideration right from the start.
Other languages that most people use today grew at later times (borrowing from Lisp) when the resources wasn't such a big deal.
I consider languages like PHP, Ruby and Python peak examples of mainstream languages that were created with barely any consideration for efficiency. This coincides with 1990s and early 2000s where we saw dramatic increase of system resources and especially memory without much need to use it efficiently.
Nowadays it improved a little bit as we learn to run bigger loads so newer mainstream languages (like Rust) and some older (like Java, C# or JavaScript) are putting more effort on efficiency.
Working in the field, that seems overconfidently good, considering its faster than most wire to wire SLA of world class FPGAs doing market data translation.
5us is actually quite slow.
This is from 2014, around the time I worked on this project:
https://stackoverflow.com/questions/17256040/how-fast-is-sta...
They cite "750 to 800 nanoseconds" wire to wire latency and that their next platform is going to be even faster.
They were using FPGA and yes, FPGA is faster, but not as much faster as many people would think.
First, market events come in isolation. There are no concurrent messages. WSE guarantees 1/10000th of a secend between each message. So you have your entire machine dedicated to executing the order.
FPGAs are usually used to run multiple copies of same net to speed up a simple problem but that is not the case here.
Second, FPGAs are usually used as a shortcut to optimize the execution of the problem. With FPGA you say "rather than trying to solve this problem with generic instructions that add a lot of delays I will just design dedicated net that will not be bothered by the generic baggage".
So in generic assembly you may want to write a branch and the branch predictor may go the wrong way and that costs. On FPGA you design your net and so you just go straight to the point.
But it doesn't mean you can't design normal code to be fast. You just need to be aware of actual cost of every single instruction of the critical path.
And third, FPGAs are clocked slower. What this means you have to do a lot per clock cycle on FPGA just to be on par with x86 core.
That application I worked on it was not intended as HFT. 5 us was an arbitrary goal we wanted to reach knowing full well that it is way behind HFT-ers.
I also found a small bug, that you'll want to use `(elt s (+ x 7))`, not `(+ x 8)`. `elt` is 0-indexed, so first and last of an 8 element list will be 0 and 7.
Relevant Stackoverflow which contains a lot of simple suggestions to hopefully shore up the deficit compared to SBCL: https://stackoverflow.com/questions/14115980/clojure-perform...
To compare language compiler and runtime performance you should at least use similar data-structures and algorithms.
> but, you can generate very fast (i.e., more energy efficient)
I'm actually not sure this is true, I was surprised the other day to find a study on this and to find that the JVM was one of the most energy efficient runtime.
This was the link: https://thenewstack.io/which-programming-languages-use-the-l...
And what you can see is that execution time doesn't always mean more energy efficient. For example you can look at Go and see that it beats a lot of things in execution time, but loses to those in energy efficiency. Like how Go was faster than Lisp yet less energy efficient than Lisp.
The research showed that the Go version of the programs were in average faster in terms of execution time and consumed less memory than the alternate Lisp versions, yet the Lisp versions consumed less electricity and were thus more energy efficient.
So the interesting bit here is that better performance and lower memory consumption doesn't always mean more energy efficient.
Now, there is definitely a link between performance and energy use, the research did show that for the most part, faster execution often reflected in lower energy spent, but there were surprising variation within that range where it wasn't always clear cut.
Two things in the implementation are performance killers: Laziness and boxed maths.
- baseline: Taking your original implementation and running it on a sequence or 1e6 elements I generated, I start off at 1.2 seconds.
- Naive transducer: Needs a transducer of a sliding window which doesn't exist in the core yet[0], 470ms
- Throw away function composition, use peek and nth to access the vector: 54ms
- Map & filter -> keep: 49ms
- Vectors -> arrays: 29ms
I'd argue only the last step makes the code slightly less idiomatic. Might even say that aggressively using juxt, partial and apply is less idiomatic than simple lambdas
You can see the implementation here
[0] https://gist.github.com/nornagon/03b85fbc22b3613087f6
[1] https://gist.github.com/bsless/0d9863424a00abbd325327cff1ea0...
Edit: formatting
The main reason I stopped using lisp was because of the community. There were some amazing people that helped out with a lot of the stuff I was building, but not a critical mass of them and things just kind of stagnated. Then it seemed like for every project I poured my soul into, someone would write a one-off copy of it "just because" instead of contributing. It's definitely a culture of lone-wolf programmers.
CL is a decent language with some really good implementations, and everywhere I go I miss the macros. I definitely want to pick it up again sometime, but probably will just let the async stuff I used to work on rest in peace.
I’ve read through some of your async work in the past and from an initial glance, it seemed like you had the right idea by wrapping existing event libs and exposing familiar event loop idioms. At the very least, it seemed uncontroversial so I’m interested to see why others would choose not to build upon it.
Wookie (http://wookie.lyonbros.com/) was the main one, or at least the most obnoxious to me. I was trying to create a general-purpose HTTP application server on top of cl-async. Without naming any specific projects, it was duplicated because it (and/or cl-async) wasn't fast enough.
> At the very least, it seemed uncontroversial so I’m interested to see why others would choose not to build upon it.
A superficial need for raw performance seemed to be the biggest reason. The thing is, I wasn't opposed at all to discussions and contributions regarding performance. I was all about it.
Oh well.
This is probably an unpopular opinion in this thread, but despite having worked with it for years, I still don’t much like it, mostly because it’s far too terse and the syntax is so far removed from that of C-based languages. The other day I wrote a Java-based DTO and it was refreshing how clear and verbose everything was, despite the almost comical volume of code it took in comparison to what similar functionality would look look like in Clojure. Plus, the state of tooling in Clojure is not the best.
I would also add that while you might initially do well with something like Clojure, it may be difficult to maintain, especially if you plan to make it a legacy product with O&M support in the future.
You could easily use CLOS or Structs in Common Lisp to do DTO. If you were using CLOS for DTOs you'd also have generic functions to dispatch on said DTOs.
The difficult to maintain line really rubs me the wrong way. I've seen messes in all languages I've worked with and see no reason to think a lisp would be worse. Just conjecture from all around.
Microbenchmarking requires rather deep knowledge how the JIT works, how the hardware - CPU/cache/memory operates, the costs of calls certain system calls and what not.
EDIT: did you read the article? It is not a microbenchmark.
But I do agree that the post is not a benchmark.
Of course I read the article, the benchmarks are at the end. The rest is about LoC (which is yet another benchmark, albeit even more useless). I even read all the code on github, there is a room for improvement there as well.
That being said, I think CL is a fantastic language and there is a lot more libraries out there to do useful things than in scheme. My C programming is really weak so I find it challenging whenever I come across a library in c that isn’t already wrapped
That said, I'd rather use CL by far than any other language except Scheme, and there are cases where CL is the right thing and Scheme is not. The most brilliant, beautiful, joy to maintain enterprise system I've ever seen was written in CL.
Racket has a nice package manager and module system that kind of works for me, and the documentation is honestly some of the best I've ever used, if not my favorite. Comparatively, I've tried using GNU Guile and found the online documentation to be horrendous, and trying to find online documentation for what's considered to be the "standard library" in Common Lisp still confuses me.
I love seeing people use CL and other Lisp-likes in the wild, and Norvig was a big inspiration for me.
Undoubtedly there are some issues I haven't thought through, and I'm too lazy to actually try to implement it, but I've always thought one should be able to make "CL1" (or some such) that's basically common lisp but with a single namespace for functions and variables.
That was the moment I started my path to liking a separate variable and function namespace.
I also occasionally get bitten by this in Python, isn't "file" a perfect variable name for holding a handle to an opened file?
All letter combos for the number "6397":
#!/usr/bin/bash
m[0]=0;m[1]=1;m[2]="{a,b,c}";m[3]="{d,e,f}";m[4]="{g,h,i}";m[5]="{j,k,l}";
m[6]="{m,n,o}";m[7]="{p,q,r,s}";m[8]="{t,u,v}";m[9]="{w,x,y,z}"
var=$(echo ${m[6]}${m[3]}${m[9]}${m[7]})
eval echo $varFor parallel computing, we use: https://lparallel.org/ Its been great at handling massive loads accross all processors elegantly. And then for locking against overwrites on highly parallel database transactions we use mutex locks that are built into the http://sbcl.org/ compiler with very handy macros.
The only gaps we've had with our production code and lisp is PDF (we use the java pdfbox), translating between RDF formats (also a java lib) and encrypting JWP tokens for PKCE dPop authentication (also java)
The complete conceputal AI system and space/time causal systems digital twin technology is all in common lisp (sbcl)
Also fantastic is the sb-profile library in sbcl that lets you profile any number of functions and see number of iterations and time used as well as consing all ordered by slowest cummulative time. That feature has been key on finding those functions that are slow and optimizing leading to orders of magnitude speed improvements.
The conceptual AI models operational concepts based on an understanding on how human concepts work, inference and automatic classification using those concepts, and then learning new concepts. The operational side of the digital twin uses functional specifications held elsewhere, which is also true of the operational concepts which use specifications in the form of conceptual definitions.
And the technology takes in RDF graph as data for input, builds the digital twin model from that data with extensive infererence, then expresses itself back out with RDF graph data. (Making https://solidproject.org/ the ideal protocol for us where each pod is a digital-twin of something)
We are working toward commercial launch in the coming weeks. (We are adding Project Pods, Business Pods, Site Pods with harvesting the sematic parse we do of PDFs into the pod, so we handle very big data)
EDIT: It's kind of ironic for me to make this claim since I use Emacs as my editor...
I've been around the block for long enough to see how far the pendulum swings on this one. I'm guessing that it starts going the other way soon.
Modern static type systems are a totally different beast: inference and duck-typing cut down on noise/boilerplate, and the number of bugs that can be caught is dramatically higher thanks to maybe-types, tagged unions/exhaustiveness checking, etc. I think we've circled around to a happy best-of-both-worlds and the pendulum is settling there.
For mechanical things where you the programmer are building the abstractions (compilers, operating systems, drivers) this is a non-issue, but for dealing with the ugly real world dynamic is still the way to go.
tl;dr when used properly, static type systems are an enormous advantage when dealing with data from the real world because you can write a total function that accepts unstructured data like a string or byte stream and returns either a successful result with a parsed data structure, a partial result, or a value that indicates failure, without having to do a separate validation step at all -- and the type system will check that all your intermediate results are correct, type-wise.
For example: a technique I've used to work with arbitrary, unknown JSON values, is to type them as a union of primitives + arrays of json values + objects of json values. And then I can pick these values apart in a way that's totally safe while making no dangerous assumptions about their contents.
Of course this opens the door for lots of potential mistakes (though runtime errors at least are impossible), but it's 100% compatible with any statically-typed language that has unions.
What leads you to believe that static typing turns a task that essencially boils down to input validation "a nightmare"?
From my perspective, with static typing that task is a treat and all headaches that come with dynamic typing simply vanish.
Take for example Typescript. Between type assertion functions, type guards, optional types and union types, inferring types from any object is a trivial task with clean code enforced by the compiler itself.
There is no such thing as external data that is not checkable or inferable by typescript. That's what type assertion functions and type guards are for.
With typescript, you can take in an instance of type any, pass it to a type assertion function or a type guard, and depending on the outcome either narrow it to a specific type or throw an error.
> inferring types from any object is a trivial task
This is true for values defined in code, but TypeScript cannot directly see data that comes in from eg. an API, and so can't infer types from it. You can give the data types yourself, and you can even give it types based on validation logic that happens at runtime, and I think this is usually worth doing and not a huge burden if you use a library. But it's disingenuous to suggest that it's free.
The closest thing to "free" would be blindly asserting the data's type, which is very dangerous and IMO usually worse than not having static types at all, because it gives you a false sense of security:
const someApiData: any = { foo: 'bar' }
function doSomethingWith(x: ApiData) {
return x.bar + 12
}
type ApiData = {
foo: string,
bar: number
}
// no typescript errors!
doSomethingWith(someApiData as ApiData)
No, that's not right at all. TypeScript allows you to determine the exact type of an object in any code path through type assertions and type guards.
With TypeScript you can get an any instance from wherever, apply your checks, and from thereon either throw an error or narrow your any object into whatever type you're interested in.
I really do not know what leads you to believe that TypeScrip cannot handle static typing or input validation.
GP's claim was that Java was too verbose. But verbosity isn't really the problem. There are tools for dealing with it. The problem is a proliferation of concepts.
A lot of business applications goes like this: Take a myriad of input through a complicated UI, transform it a bit and send it to somewhere else. With very accurate typing and a very messy setting (say, there's a basic model with a few concepts in it, and then 27 exceptions), you may end up modeling snowflakes with your types instead of thinking about how to actually solve the problem.
I've mostly come to the conclusion that dynamic languages work well wherever business requirements change frequently and codepaths are wide but shallow (e.g. many different codepaths but none of them are particularly involved). Static languages work better for codepaths that are narrow but deep, where careful parsing at API edges and effective type-level modelling of data can create high-confidence software; in these situations the logic is often complicated enough where requirements just can't change that frequently. I wish we had a "best of both world" style to help where you have wide and deep codepaths, but alas that'll have to wait for more PLT (and probably a time when we aren't forming silly wars over dynamic vs static typing as if one was wholly superior than the other.)
The type declaration syntax definitely could use some love, I think it's a shame a more convenient syntax was never standardized. And sum types etc would be nice of course. It's all perfectly possible.
However, you have to consider that Common Lisp itself is quite different from other dynamically typed languages.
I find that, after the initial adjustment period with the language (which is significant, I admit), it's surprisingly hard to write messy code in CL, certainly harder than in Python or Ruby. At the very least, the temptation to do so is lower, because there are fewer obstacles to expressing sophisticated ideas succinctly.
And no, I am not talking about the ability to define your own macros and create DSLs. I think it has to do with the extensive selection of tools for creating short-lived local bindings, the huge selection of tools for flow control, and the strict distinction between dynamic and lexical variables.
There's just something about it that sets it apart from other dynamically-typed languages, even without the gradual typing aspect and even without the speed difference. Navigating a source codebase in Python without strict type annotations is like navigating in the dark in a swamp. I don't have the same issues in Common Lisp for the most part.
Maybe this has more to do with undisciplined programmers self-selecting out of CL than it has to do with any aspect of CL itself.
And on top of the excellent and unique language design, you have:
* A powerful CFFI
* An official specification
* The "REPL-driven" development style (if you want it)
* Several well-maintained implementations that generate high-performance machine code
* The unique condition system
* Literally decades of backward compatibility
* A core of stable, well-designed packages, including bindings to a lot of "foundational" libraries
* Macros if you really do want to invent your own syntax or DSL
Probably the only big downside is that the developer ecosystem is still focused around Emacs. That too is changing gradually but steadily, with Slyblime (SLY/SLYNK ported to Sublime Text), Slimv and Vlime (Vim ports of SLIME/SWANK), the free version of LispWorks for light-duty stuff, and at least one Jupyter kernel.
Also, Roswell (like Rbenv or Pyenv) and Qlot or CLPM (like Bundler or Pipenv) help create a "project-local" dev experience that's similar to how things are done in other language ecosystems.
And of course there is Quicklisp itself, which is a rock solid piece of software, and fast too!
Python and Ruby have their own merits, for sure, and there are plenty of things I have in Python that I wish I had in CL. But it really doesn't seem right to compare them, CL seems like a totally different category of language.
JS was, because browsers. Python was starting to be toward the end of the 90s. Ruby (as I understand) was in Japan though it wasn't until Rails took off that it became popular elsewhere. Perl (not on the list but similar to those on the list) definitely was.
All this however is rather orthogonal to the strengths of type systems. CL type system, for instance, is stronger than one of Java or C.
It probably won't reduce the intensity of the way a small minority of the community treats the language-level difference in holy wars, though.
Like Github or WordPress?
It does improve your quality of life as an engineer, I can promise you that.
I mean, like in most other fields no ? Most successful movies, books, foods, artworks, furnitures, ... are fairly different from the best ones.
Regarding type-checking, common lisp is expressive enough to support an ML dialect (see coalton), and is easily extended across paradigms [2]
That’s a quite absolute statement. At least Erlang/Elixir users would tend to disagree. “Dynamically typed” can still represent a huge variety of approaches, and doesn’t have to always look like writing vanilla JavaScript for example.
I'm aware that there exist dynamically typed languages in which large projects are written, I'm saying that they would be better off with type safety.
It is optionally as statically typed as you want, depending on what compiler you use, I am mostly familiar with SBCL, which does a fair bit of type inference and will tell you at length where your lack of static typic means it will produce slower code.
You can add type declarations and a good compiler will check against them at compile time: https://medium.com/@MartinCracauer/static-type-checking-in-t...
For lisps, I think Racket and Clojure feel the most modern.
But I’m starting to come around to the idea that there are enough modern, hyper modern, post-modern, languages and programmers out there.
Some of the value in learning Common Lisp might be in its value for living software archaeology. You can find actual code written at the time people were figuring out Big Ideas that we take for granted today. And usually the process of synthesis means that Big Ideas lose a lot of associated commentary and thinking that fed in.
There’s a place for modern languages that conform to the same general set of sensibilities. But there’s also joy in gettin’ weird with some of the old stuff. It’s smugly satisfying to beat the new kid on the block with old tricks on occasion.
> Even though Clojure might be a more obvious choice for someone, like me, who is used to working with the JVM, I actually wanted to try using something else with a lighter runtime and great performance.
I'm curious how far it's possible to push performance by generating Lisp code in SBCL compared to the classical C interpreter goto loop.
Obviously doing this from your dev environment is a bad idea, so most people write a script that loads the system and dumps the image. Or you can use one that someone else has already written.
Not all lisps work this way (most notably ECL does not), and you can use ASDF's program-op to create an executable directly from a package definition, which should work on any implemntation supported by ASDF.
It's even alluded to in the ruby bare-words example in this famous talk[1]
For any sizable Lisp project you would have an ASDF-file to load the system, with that you can create an easy build-script or even a make-file, which loads your program via ASDF and then builds the executable, it is described well here: https://stackoverflow.com/questions/14171849/compiling-commo...
For consistancy, I would recommend to build the executable from the freshly loaded source, not a lisp image which has been used to develop in.
Its slightly tricky in that the whole make process can only happen on a single thread, so you have to turn off all parallel threads during make - so we have a key on many functions :make that turns off parallel for using make. Ultimatel the make file uses (asdf:make :package-name) to compile to an exe. The other tricky part is to get a web server in the exe to work, and thats handled like this:
(handler-case (bt:join-thread (find-if (lambda (th)
#+os-unix (search "clack-handler-woo" (bt:thread-name th))
#+os-windows (search "hunchentoot" (bt:thread-name th))
)
(bt:all-threads)))
;; Catch a user's C-c
(#+sbcl sb-sys:interactive-interrupt
#+ccl ccl:interrupt-signal-condition
#+clisp system::simple-interrupt-condition
#+ecl ext:interactive-interrupt
#+allegro excl:interrupt-signal
() (progn
(format *error-output* "Aborting.~&")
(stop)
(uiop:quit)))
(error (c) (progn (format t "Woops, an unknown error occured:~&~a~& - restarting service..." c)
#+os-unix (trivial-shell:shell-command "service trinity restart"))
))This is true of all niche languages that are supposedly quicker to use than regular mainstream languages.
EDIT: To amend the preceding sentence, it's almost never faster. There are probably languages which people know that are sufficiently in conflict with the problem domain that they could be faster in some other new-to-them language versus the one they know for specific problems.
though, the fact that it wants its own keyboard is a bit of a hurdle :)
I found it odd, but I’ve seen weirder. Then I tried the simple example of writing a function to compute the nth fibonacci number (it’s almost the first example). After testing some numbers, I thought it seemed slow, so I quickly implemented the same function in Rust (with recursion too).
Rust took less time in compiling, and computing fib(n) for n 1..50 than Lisp to compute fib(50). Maybe I did something very wrong, but for now I’d rather learn Rust better.
https://lisp-lang.org/learn/functions
I know the reason for the time taken is because it’s not tail recursive, and that making it tail recursive would make it almost immediate, but for me that was not the point.
The thing is, I don’t really know how to write performant rust code, and yet my naive implementation in rust works better than the example in learn-lisp.
To me that means performant Lisp is non trivial (meaning you need deep understanding to achieve it). If you show a slow but easier to understand example, it’s because fast examples are way harder to understand.
Assuming your rust version is not tail-recursive, this is not true. Of course a tail-recursive (or iterative for that matter) version would be way faster, but that is independent of the language used[1]. The naive algorithm in both Rust and Lisp should be relatively similar in runtimes, and for me (after I properly enabled optimiations) they were.
Also, tail recursion is a red-herring here for another reason. For both Rust and Lisp, I would use an iterative approach, not a tail-recursive approach, like the example in the Rust num_bigint docs[2].
1: A theoretical language that automatically memoized pure functions would make this false, but that doesn't apply to the current discussion.
2: https://docs.rs/num-bigint/0.4.2/num_bigint/
The equivalent Lisp would be:
CL-USER> (defun fib (n)
(let ((f0 0) (f1 1))
(loop repeat n
do (shiftf f0 f1 (+ f0 f1)))
f0))
FIB
CL-USER> (time (fib 1000))
Evaluation took:
0.000 seconds of real time
0.000171 seconds of total run time (0.000136 user, 0.000035 system)
100.00% CPU
471,103 processor cycles
65,456 bytes consed
43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875
CL-USER>
here's the Rust code: fn fib(n: usize) -> u64 { match n { 0 => 1, 1 => 1, _ => fib(n - 1) + fib(n - 2), } }
fn main() {
println!("{}", fib(50));
}
(defun fib (n)
"Return the nth Fibonacci number."
(if (< n 2)
n
(+ (fib (- n 1))
(fib (- n 2)))))
I also know the Rust snippet will crash at fib(~100), and that I could write the thing in a loop and get there under 1ms. The same is surely true about Lisp.
In general, Common Lisp doesn't favor recursive functions (as opposed to Scheme), and as far as I know, most implementations don't put huge amounts of effort into optimizing it.
I'd be interested to see whether the same written using the loop (or iterate:iter) macros would still be as slow.
(defun nth-fibonacci (n &optional (a 0) (b 1))
(if (= n 0)
a
(nth-fibonacci (- n 1) b (+ a b))))20793608237133498072112648988642836825087036094015903119682945866528501423455686648927456034305226515591757343297190158010624794267250973176133810179902738038231789748346235556483191431591924532394420028067810320408724414693462849062668387083308048250920654493340878733226377580847446324873797603734794648258113858631550404081017260381202919943892370942852601647398213554479081823593715429566945149312993664846779090437799284773675379284270660175134664833266377698642012106891355791141872776934080803504956794094648292880566056364718187662668970758537383352677420835574155945658542003634765324541006121012446785689171494803262408602693091211601973938229446636049901531963286159699077880427720289235539329671877182915643419079186525118678856821600897520171070499437657067342400871083908811800976259727431820539554256869460815355918458253398234382360435762759823179896116748424269545924633204614137992850814352018738480923581553988990897151469406131695614497783720743461373756218685106856826090696339815490921253714537241866911604250597353747823733268178182198509240226955826416016690084749816072843582488613184829905383150180047844353751554201573833105521980998123833253261228689824051777846588461079790807828367132384798451794011076569057522158680378961532160858387223882974380483931929541222100800313580688585002598879566463221427820448492565073106595808837401648996423563386109782045634122467872921845606409174360635618216883812562321664442822952537577492715365321134204530686742435454505103269768144370118494906390254934942358904031509877369722437053383165360388595116980245927935225901537634925654872380877183008301074569444002426436414756905094535072804764684492105680024739914490555904391369218696387092918189246157103450387050229300603241611410707453960080170928277951834763216705242485820801423866526633816082921442883095463259080471819329201710147828025221385656340207489796317663278872207607791034431700112753558813478888727503825389066823098683355695718137867882982111710796422706778536913192342733364556727928018953989153106047379741280794091639429908796650294603536651238230626 WEB>
If I recall you have to set the optimization level prior to compiling the function using `(declaim (optimize xxx))` - where xxx is something I've forgotten. Perhaps someone can come along and point out what xxx should be?
; disassembly for NTH-FIBONACCI
; Size: 94 bytes. Origin: #x2264E531 ; NTH-FIBONACCI
; 31: 498B4510 MOV RAX, [R13+16] ; thread.binding-stack-pointer
; 35: 488945F8 MOV [RBP-8], RAX
; 39: 488B55F0 MOV RDX, [RBP-16]
; 3D: 31FF XOR EDI, EDI
; 3F: E8AC343BFF CALL #x21A019F0 ; GENERIC-=
; 44: 750A JNE L0
; 46: 488B55E8 MOV RDX, [RBP-24]
; 4A: 488BE5 MOV RSP, RBP
; 4D: F8 CLC
; 4E: 5D POP RBP
; 4F: C3 RET
; 50: L0: 488B55F0 MOV RDX, [RBP-16]
; 54: BF02000000 MOV EDI, 2
; 59: E8C2323BFF CALL #x21A01820 ; GENERIC--
; 5E: 488BC2 MOV RAX, RDX
; 61: 488945D8 MOV [RBP-40], RAX
; 65: 488B55E8 MOV RDX, [RBP-24]
; 69: 488B7DE0 MOV RDI, [RBP-32]
; 6D: E84E323BFF CALL #x21A017C0 ; GENERIC-+
; 72: 488BF2 MOV RSI, RDX
; 75: 488B45D8 MOV RAX, [RBP-40]
; 79: 488BD0 MOV RDX, RAX
; 7C: 488B7DE0 MOV RDI, [RBP-32]
; 80: B906000000 MOV ECX, 6
; 85: FF7508 PUSH QWORD PTR [RBP+8]
; 88: E99507DBFD JMP #x203FED22 ; #<FDEFN NTH-FIBONACCI>
; 8D: CC10 INT3 16 ; Invalid argument count trap
I wonder if different installs (or perhaps it's Slime) sets this value to different levels.
Something to bear in mind anyway..
WEB> (declaim (optimize (debug 0) (safety 0) (speed 3)))
NIL
WEB> (defun nth-fibonacci (n &optional (a 0) (b 1))
(if (= n 0)
a
(nth-fibonacci (- n 1) b (+ a b))))
WEB> (time (nth-fibonacci 9999))
Evaluation took: 0.002 seconds of real time 0.001887 seconds of total run time (0.001887 user, 0.000000 system) 100.00% CPU 5,476,446 processor cycles 4,715,760 bytes consed
The purpose of all these loop constructs is to place constraints on some essential aspect of the loop up front, thus narrowing the possible range of effects of that wicked goto, by encoding common patterns. "For" loops guarantee the number of iterations. "While" loops guarantee the exit condition. And tail recursion guarantees what state gets modified in the loop.
It is strange the way you say "better off doing it in a loop". As you say, it is a loop. What construct would you prefer? GOTO?
(defun tail-fib (n &optional (v0 0) (v1 1)
(cond ((zerop n) v0)
(t (tail-fib (1- n) v1 (+ v0 v1)))))Factorial and Fibonacci as a pair are good introductions to the issue of recursion--and lisp had recursion back in the dark ages before other languages did--but they're aren't what lisp is about.
so, were some other newbie to read your comment, it might really put them off learning lisp for entirely the wrong reasons.
cheers :)
It's going to make a lot of memory allocations and make a lot of uninlinable function calls, both of which are places that I would expect rust to have an advantage. That being said, I'd be curious to see your rust version, as just the number of bignum operations done for fib(50) is going to take a long time.
I wasn’t benchmarking CL vs Rust, but my CL vs my Rust. What I was trying to see is, can I do something useful with this? Or do I need to invest a good amount of time (something I don’t really have now) before being productive.
As for the rust version, I did the “trivial” (and terrible) translation. Match on input, recursive call on the _ arm.
Lisp (SBCL):
(defun fib (n)
(case n
(0 0)
(1 1)
(otherwise (+ (fib (- n 1)) (fib (- n 2))))))
(time (fib 40))
use num_bigint::BigUint;
use num_traits::{Zero, One};
// Calculate large fibonacci numbers.
fn fib(n: usize) -> BigUint {
match n {
0 => Zero::zero(),
1 => One::one(),
_ => fib(n-1) + fib(n-2),
}
}
fn main() {
println!("fib(40) = {}", fib(40));
}The original poster said:
> To me that means performant Lisp is non trivial (meaning you need deep understanding to achieve it). If you show a slow but easier to understand example, it’s because fast examples are way harder to understand.
And
> What I was trying to see is, can I do something useful with this? Or do I need to invest a good amount of time (something I don’t really have now) before being productive.
I could have equally said that my example demonstrates that "To me that means that performant Rust is non trivial" or that I would "need to invest a good amount of time ... before being productive"
Both of which are clearly not true. Posting my code let other people find the extremely trivial change to fix the huge performance difference.
I did not mean to say that I believe it inferior or anything, just that I felt writing performant code in Lisp is non trivial to learn, while rust feels pretty natural (to me).
I’ll try again at some point, but I don’t think it’s a wise investment of time at this point, I’d rather be “fluent” in rust first, then maybe try lisp.