I checked Lisp and what stands out most for me is the syntax and the powerful REPL but aside from that, why is it so praised compared to modern languages? I don't see many projects using it either aside from certain exceptions.
Apologies for my ignorance _in advance_
With something like SBCL, you can literally write assembly in lisp notation (called VOP in SBCL), then layer the Common Lisp language on top seamlessly, then layer your domain-specific code on top with your code looking and acting just like it is part of the native language.
When you're done, you've built language primitives up to deal with your specific problem domain then you can solve the problem with a language custom-made for your needs.
IMHO the biggest exception is macros. Lisp isn’t the only language that has them, but it’s still the easiest by far if you ever want a domain-specific language to concisely express some problem. The grammar of its special forms is simple enough that we can easily use the symbol and linked-list builtins to generate boilerplate expressions or functions for the compiler without fighting with in-memory parse tree nodes or generating entire files of syntactically valid source.
That is pretty profound, and I still have yet to find a language that just makes it so simple and elegant as with LISP.
P.S. If you're looking for Lisp enlightenment experiences, actually read through the code snippets in the book -- spending a few minutes staring at 5 lines of macro code and then having understanding dawn on you will give you some of those experiences (at least in the small) :-)
Mixing pure text and code became obvious. This made me want to write a text editor with a programming language and its current state is in this wiki - http://lambdaway.free.fr/lambdawalks
For me - I'm just an "amateur", it was that the Lisp "Enlightment", just to give me a way to implement from scratch a small engine to go down to the foundations of code, discover the lambda-calculus, and have a chance to understand and enjoy coding.
If you are just starting, maybe you would be better off using Racket Scheme - it has a modern and rich ecosystem around it.
If you do go with Common Lisp, you can grab a free copy of my book by setting the price to $0.00 https://leanpub.com/lovinglisp There are better introductory Common Lisp books, but mine has many fun example programs that you might enjoy if your technical interests overlap mine.
1) one can generate code and evaluate it
2) one can write simple interpreters, see McCarthy's first Lisp interpreter in Lisp -> your own eval
3) one can write procedural source code transformations, where the code is Lisp data -> transforming Lisp programs by Lisp programs -> apply the same development tools to the meta-level -> integrate it directly into the evaluator
The englightment is understanding how a minimal Lisp machinery for this works and that things fit together quite well.
4) how to apply source code to various software problems, for example in declarative programming or how to provide simpler interfaces to more complex, say, object-oriented mechanisms (-> example CLOS)
then there are a large number of consequences / further directions
* addressing the deficiencies of that model
* compilation
* language building (embedded DSLs)
* editor interfaces
* interactive programming
* UI representations as data
* machine models
List and atoms can be defined in terms of lambdas as well (see e.g. [1]).
You don't need any special forms at all. Perhaps we should talk about Lambda Enlightenment?
In lispalikes, code has the same "shape" as any other data.
The compiler / language environment and the runtime environment are "blended" in a much more natural way than is usual.
JSON being a native part of the language makes it inexpressibly easier to work with than in languages where you've got to define serialization and logic
In JS, you can toss around objects which can have functions or primitive values as properties in much the same way you can work with IE maps in Clojure
AIUI (not a lisper; just dabbled in a past life) in Lisp, the closest practice is idiomatic.
Learning Lisp certainly improved the way I write code in other languages and moved me to a more functional style of programming. So while I don't regularly code in Lisp, I do use a lot of concepts that can be traced back to Lisp.
The quality varies a lot: someone as new as I was would write a 30-line function using (for), then a few answers down I’d find a 3-line answer that used (frequencies), (juxt), or some other useful standard library function that I hadn’t discovered yet.
Clojure differs from other lisps (from what I can see; haven’t spent much time with the others) with its emphasis on immutability-first, and by building a ton of functionality around the seq abstraction. Judging by the comments here, other lisps place more emphasis on meta programming. While “code is data” remains true and powerful in Clojure, I have not yet had to write a macro for myself. The homoiconicity is still quite nice.
The chunk of lisp enlightenment I have a more sound claim to after learning Clojure is the power of the REPL. The ability to access any function, at any level of a running system, feed it any data to see how it responds, then change the function and see it immediately reflected without restarting anything, results in a very tight feedback loop that you won’t realize you missed elsewhere.
And yet, it is dramatically simpler.
You have not yet reached enlightenment.
A koan is incomplete.
File "koans/asserts.lisp"
Koan "FILL-IN-THE-BLANKS"
Current koan assert status is "(INCOMPLETE INCOMPLETE INCOMPLETE)"
At the risk of explaining the joke and ruining it: the author is a transcendental being doing a fizz-buzz coding interview. The interviewer doesn't like the fact that the author copied and paste in a solution....so the author then goes off and writes an interpreter in Clojure (Clojure is a Lisp) that actually interprets the original code snippet.
---
What I actually love and enjoy about Lisp (Clojure in particular) is how malleable it is. Functional Programming is the paradigm you hear about the most, but you can write object-oriented code (lambdas simulate objects very easily, see SICP) or Logical programming (clojure/core.logic) or, ultimately, imperative programming.
Adding new "features" to the language is something every Lisp intro book should include. Don't have an `until` operator? (the opposite of a `while`) bam: macro. It's trivial.
Want to have goroutines? There's a library: core.async. Languages like Go get popularity because of their features like goroutines, Lisps can just import a library and get the same (or similar).
I'll fully admit, I bounced off Clojure for the first three times I tried learning it. But when it clicks is when you assemble a handful of map/reduce/apply and transform your data with ease, instead of shoving it around. Lisp makes your functions and data sing. Lisps allow you to find higher levels of composing functions, e.g. Transducers (https://clojure.org/reference/transducers)
So basically lisp is clay that can be whatever you want. Other languages are concrete, set in stone.
Combine this with incredible live environment and interactive debugging and you have a truly "clay" programming system that you can mold on the fly.
I feel this a lot working on a C# website that takes 2+ minutes to build. C# hot reload is so limited and depending on what you modify you have to rebuild anyway.
1. The realization that code and data are the same kind of entity. From this realization springs the realization that you can use data structures for managing code and that leads you to Lisp macros - which is one of the superpowers of Lisp. Lisp stands for LISt Processor and Lists are the primary data structure (okay, technically CONSes are, but let's not confuse things) for storing both data and code. Why all the funny parenthesis in Lisp? Because that's the syntax for a list data structure! '(1 2 3) is a list of the first three positive integers and (add 1 2 3) is a list whose first member is the symbol 'add' - which is a function. When evaluated it will yield the value 6. There - now you know Lisp syntax!
2. The realization that many algorithms can be easily implemented using a list data structure having two simple operators: first and rest (CAR and CDR in Lisp parlance) and recursion. This is why you see so many articles about implementing Lisp in <insert popular language here>. It doesn't take a lot to get a Lisp interpreter working, in fact it used to be a routine exercise for undergraduate CS students. Evaluating (car '(1 2 3)) yields 1, and (cdr '(1 2 3)) yields '(2 3). A recursive function in Lisp then typically has two parameters, a list and an accumulator. If the list is NIL then you return the accumulator, otherwise you update the accumulator by applying some operation to the CAR of the list and the input accumulator and recursively invoke the function with the CDR of the list and the updated accumulator. Simple. Tail-call optimization also makes this highly-performant.
3. The difference between symbols in Lisp and variables in other programming languages. In many commonly-used programming languages variables refer to memory locations. That's not strictly true in Lisp. Symbols are an item in a symbol table and the current symbol table is context-sensitive. That means symbols have properties, including user-defined properties, and the same code may be executed with different symbol tables in effect - which is another part of how macros can be so powerful.
4. Functions are first-class objects. This is also different from many other commonly-used programming languages, though that has been changing. A symbol can evaluate to a function rather than a value, functions can be created on the fly (lambdas) and passed as parameters to other functions. You can have a list containing functions (such as (add 1 2 3) from above) and evaluate those functions and so forth. A function is simply another data type. This was more of a big deal 10 or more years ago, now many languages have been adding this capability.
5. The REPL. It's because code and data are the same construct that the REPL as you know it can exist. Even better is when you realize not all functions in the REPL have to be interpreted - you can compile them! You can freely intermix interpreted and compiled code together. You can load compiled code into your session, add new code, modify existing code - whatever you need to do. It's an extremely creative and productive development environment.
That's it! That's the Lisp enlightenment.
In my opinion, most modern dynamically-typed languages are actually better than Common Lisp in almost all regards that matter for social, collective programming:
- they are usually much more functional and declarative: more immutability, more value-oriented approach (Erlang, Clojure), less unpredictable mutation, more useful abstractions built into the language. They provide modern data structures in standard libraries in a useful and uniform way.
- they provide a builtin class/object/prototype common language for their ecosystems (CLOS in Common Lisp and Moose in Perl 5 are examples of too flexible opt-in OOP systems that fragment ecosystems instead of unifying them); this also gives decent namespaces, which are crucial for cooperation between teams.
- they usually have a good async story (iterators/generators/coroutines, promises/futures, async/await), again, built into the language and enabling async ecosystems on top of them.
- they have good(-ish) story about package management. Lispers seem to like to write everything from scratch.
- they have good story about testing infrastructure. Lispers are usually too proud of keeping the world in their heads and of their REPL-driven bug-fixing dopamine cycle to actually think about having computer test their code. The ability to change a running program from the inside is actually a detriment to abstract thinking about what code is meant to do and describing that in more declarative invariants like type systems and tests.
- tooling has finally caught up: with LSP and treesitter, programmers do not have to bear the pain of S-expressions to have structural editing anymore. A sufficiently educated programmer recognizes syntax trees as a valid data structure in any language. If really needed, modern languages do provide abstractions to manipulate their AST (`import ast, dis` in Python, `erl_syntax` in Erlang, `ast` in Go, declarative and procedural macros in Rust, Template Haskell, etc).
The only thing which I think Common Lisp still does better is conditions/restarts, which seems a better way of error handling than just catching exceptions.