Thoughts On Lisp And Racket (2019)
macadie.net
macadie.net
> It looks to me like the Lisp community knows something the rest of the world does not. They are not searching for the next hot thing. They seem to understand things on a deeper level than other communities. Perhaps “enlightenment” is not an overwrought term to use.
For those that haven't read this before. For me, any article making claims like this simply brings to mind the Lisp Curse:
http://web.archive.org/web/20210617140457/http://www.winesto...
(Not currently available... Last snapshot is from June, so unsure whether this is permanent or just temporary.)
Lisp is extremely flexible as a language. The core is very small, especially if you look at something like Scheme. But it lets you easily create and extend the language into what practically becomes a domain-specific language (DSL) for whatever problem you're working on.
Except what this really does is push things from being technical issues to being social issues. Sure, maybe I know Lisp. But how long is it going to take me to learn what your ultra-customized, macro-laden particular flavor of Lisp? And one has to do that for every single new project requiring coordination...
Lisp is not one language, but a wide variety of personal languages that happen to share a common core and syntax.
1.) If I needed to do some really crazy stuff (like I imagine what grammerly does) then lisp might be the right tool. I know it is far more performant than the scripting languages (really closer to the C end of the spectrum), but I seldom need raw performance (your mileage may vary widely).
2.) 99.99% of my own work or hobby projects were better served by more mainstream tools like Python, Bash, PowerShell, C++...etc as all those tools either had built-in data structures that were easier to use, or better integrated with the OS, or they had massive ecosystems of scientific, plotting, GUI...etc, far in advance of anything that the lisp ecosystem has. You can talk about blub languages all you want, but when I can write a short little script with Python to remotely control an application using the only API they provide (Python) and then start doing something else, there is a clear winner from a software tool choice even if the lisp language itself is a better design in a multitude of ways.
The tool I have where I feel like I have an insane amount of power that isn't widely known is Wolfram Mathematica. It shares a bit with lisp in that all computing has a REPL, you can see the AST & intermediate calculations, all the command names use a pretty consistent notation like GraphPlot[]. Documentation is really good and there are primitives for pretty much everything from graph networks to matrices to neural networks to GUIs and 3D plotting ...audio, Blockchain....the list goes on and on. It's a commercial tool which sucks, but doing R&D with it is like using secret alien tech. I could literally copy paste an image of certain things I wanted to use in my network (node diagram) as vertices, build that graph, start doing computations on it like max flow or traveling salesman. I could then just grab the graphs output picture and feed it as an input into some other command and get the AdjacencyMatrix and then do other computations on that. I still love Python for code I need to share with coworkers who don't have a Mathematica license, but it is truly some amazing technology.
I’m pretty sure I’ve heard that the Wolfram Language is in fact considered a Lisp, albeit one using M-expressions rather than S-expressions [0]. But I can’t seem to find a source for this though.
Lisp was actually developed for research and development of symbolic computation systems. Early on Lisp was applied to mathematics problems like integration and other manipulation of mathematical expressions.
Thus term rewriting systems and similar (-> rule-based systems) have been early on implemented in Lisp, but Lisp itself is based on a functional&imperative expression evaluation.
I appreciate the history. It doesn't make much of a difference from an ignorant end user such as myself, but this information is at least pretty interesting to me.
I've normally categorized languages into imperative (, Fortran, C, Basic), functional (Haskell, Scheme, F#...etc), object oriented (Java, Python, Ruby, C#, JS...etc), concatenative (Forth, Factor, 8th), logic (Prolong, MiniKanren), array (APL, J, K). I guess TRS in entirely its own thing?
TRS looks like it is a different paradigm. The system takes a term and rewrites it based on a rule set. In some ways this is similar to rule-based systems, but those tend to manipulate not terms but facts in a database of facts.
I appreciate you drawing the distinction between TRS operating on rules and things like Prolog being a rules based system using facts. TBH, I only understand much of this at a very superficial level (as in I've played with Prolog several times and even read a short book, but I'd never choose it for a production application if I can get away with using SQL and some procedural code lol).
BTW the Wolfram language isn't 'a bit like lisp'. It _is_ a lisp with an alternate syntax. This becomes extremely apparent in any kind of symbolic manipulation.
Only a galaxybrain like Wolfram could devise such an elaborate implementation of Greenspun's Tenth.
Now I think Mathematica is a combination of C, C++, Java, and of course Mathematica code. C and C/C++ libraries are used for the symbolic stuff and some libraries, Java is used for database connections and so forth, and I think they do eat their own dog food by implementing new functionality using existing Mathematica code. That's a little bit yucky, but no different than many ecosystems today. At least everything seems pretty seamless to me.
Source: https://web.archive.org/web/20081121205217/http://ymeme.com/...
Even during the 80s memory was very expensive (workstations) and limited (PCs). Thus it could have made sense to implement a language such that it runs in smaller space to save a lot of money on the user side. Still, there were Lisp-based computer algebra systems on small (Derive, MuMath/MuSimp, ...) and large computers (REDUCE, Macsyma, Scratchpad)...
Also depending on the availability of a certain Lisp system was problematic, when a C or later a C++ compiler was usually provided... -> but then one was responsible for the maintenance of the implemented.
Derive even ran on some TI calculators. It was written in muLISP.
I think that the big thing to recognize about Racket is that its approach is not really to do that. It's to implement complete, well-specified domain-specific languages that live on top of Racket.
What's unique about it is that they've put a lot of thought into how to set you up for success in doing this, so that your DSL retains a high level of compatibility with the core language and other DSLs, without any particularly special effort on your part.
The other thing to keep in mind is that Lisp, particularly the Racket community, tends to favor stratified design (https://dspace.mit.edu/handle/1721.1/6064). So when you think about the level of coordination involved, don't think in terms of a tangle of DSLs and macros all vying for control of the same conceptual space. Think of it in terms of layers. The goal is usually something more like a lower-friction version of the relationship between Lua and C, or Python and C++, or even TypeScript and JavaScript.
There's an online book, Beautiful Racket (https://beautifulracket.com/) that serves as a really great illustration of how it all comes together.
This happens in every language though. You just get to this point more efficiently with lisp.
A rigid language doesn’t protect you from having to understand the local terms e.g. ConcurrencyGeneratorObserverFactory.
Once people are off the ground with an MVP that is delivering value, they are willing to invest virtually unlimited effort into scaling up their pre-existing solution, even as the limitations and absurdities of their chosen framework become increasingly obvious.
Any language is preferable to a language you don't feel confident developing in. So whatever gets people off the ground is what wins, regardless of what would be better in the long run.
On the other hand something like Racket seems very easy to get going with.
> Check out the note for the first entry: Come on, everyone! Let's beat the dead horse one more time!
And indeed, it does say that at https://hn.algolia.com/?q=the+lisp+curse below the entry for https://news.ycombinator.com/item?id=2450973
But what's weird is it doesn't appear on the page https://news.ycombinator.com/item?id=2450973 itself nor anywhere in the page source that I can find.
But going back to https://hn.algolia.com/?q=the+lisp+curse, and looking at the source it says that it's the story comment.
So if I understand correctly, even though when you submit a story with a link it does not include the text if you write one, it seems that it's still stored on the HN server and returned from the HN API that Algolia gets its data from.
Just found this interesting.
And I wonder how many other stories have a "hidden text" like this.
If the Google BigQuery or what it's called dataset includes this same data it should be a simple thing to find all such stories.
https://console.cloud.google.com/marketplace/details/y-combi...
First, number of stories where url begins with "http".
SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE INSTR(url, "http") = 1;
1729544
SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE CHAR_LENGTH(text) > 0;
218726
SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;
113550
SELECT * FROM `bigquery-public-data.hacker_news.stories` WHERE INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0 LIMIT 3;
Anyways, let's require score > 0.
SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE score > 0 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;
113322
SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE score > 25 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;
1677
And somewhere between? Say, with score > 8.
SELECT COUNT(*) FROM `bigquery-public-data.hacker_news.stories` WHERE score > 8 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0;
3458
SELECT * FROM `bigquery-public-data.hacker_news.stories` WHERE score > 8 AND INSTR(url, "http") = 1 AND CHAR_LENGTH(text) > 0 LIMIT 5;
Title: Calling the NSA: "I accidentally deleted an e-mail, can you help me recover it?".
Text: A nice thought and practically the same thing that Google does. You hand in your privacy, but you get services back in return.
Id: 6306219. Discussion: https://news.ycombinator.com/item?id=6306219
Seems pretty clear that this is an example of a comment to go along with the submission. And it was preserved, although not shown in the thread as we know.
And with Algolia we indeed get the story comment shown as well. https://hn.algolia.com/?q=%09Calling+the+NSA%3A+%22I+acciden...
Well, there you have it!
So this also makes me realize that third-party HN clients could use this to offer users the ability to write and to see story texts that don't display on HN itself due to there being an URL attached to the story, but which are still stored on the HN servers and returned by the HN API. That's pretty neat :)
Try to make something like MediKanren in a mainstream language, and the advantage of lisp languages will become immediately apparent.
Lisp gets a single programmer leverage over large, complicated problems, and puts even complex projects within the reach of small teams. In the 80s, Symbolics had the stated goal of making very large projects feasible by a small team, and anything smaller within the reach of a single developer.
Ambitious, to be sure, but given the CF that is enterprise development, it shouldn't be ruled out as something to reach for.
When you're trying to do something that hasn't been already done programming-wise. For most other cases you would be better served by a language with a healthy and large ecosystem.
Still, it's not a bad thing to try to extend the user base as it translates to more libs and better implementations. Many new things are mostly same-old same-old with a sprinkle of 'new' glitter on top!
I personally would've rather them use Lisp than tack on their own conditionals and loops and such.
Understanding someone elses code is always a challenge. Outlawing custom abstractions in favour of a few very shallow ones doesnt seem like a convincing thechnical argument.
The problem is that LISP macros are extremely powerful. The DSLs you can design don't always neatly fit into a box and integrate seamlessly. They're leaky abstractions. You don't necessarily know what's a function or a macro. You don't necessarily know that a macro isn't doing lazy evaluation or redefining some operators you're familiar with.
Programming languages aren't just about coding, they're also about having a shared language that other people can understand. If you introduce 500 new syntactic forms that you designed yourself into your code, and newcomers have to look them all up to really understand what's going on, that makes it much harder for new people to onboard.
LISP loops minimalistic on the surface, but it's kind of a lie. In practice it's easy for it to become exactly the opposite of that.
In most lisps, few stray from the basic data structures, such that the data you can get is usable basically everywhere.
Contrasted with many Java libraries, where coding a library locks you in to said library.
Yes, a few have tried to keep interface compatibility with the core collections. I'd hope most do, nowadays. But that really just highlights that we are both talking of previous year's mistakes.
My hypothesis is it comes from novice attitudes in building everything as a masterpiece. As the community matures, things tend to stabilize.
The entire point of the Lisp Curse is that this is a social issue! Not a technical one. And, furthermore, what is a technical issue in another language can be pushed into being a social issue in a Lisp. And social issues are a lot more difficult to solve.
Compare this to something like Java + Spring Boot or Ruby on Rails or any other framework favoring configuration and convention over code. Where the explicit point is a social agreement on what a program looks like, which in turn allows everyone to hopefully focus on the actual business logic.
(Also: Sibling commenter already addressed the differences in abstraction power better than I would have.)
For instance, suppose that cheap and 100% effective sound-proofing were available and installed into every apartment by default (an example of (practically nonexistent) technology). Then unwanted noise from neighboring units wouldn't be a social problem.
This is also one of the two major problems that Forth has. The other is the lack of types and non-stack data structures.
https://groups.google.com/g/racket-users/c/hP1hZDWnj7Y
It is about "social contracts" and seems to kind of address some of what you are saying here. Basically, the community comes to an agreement on what some good common contracts should be. I wonder if there are any other social problems in the lisp languages that we could think of that can be developed by consensus this way.
can you point to any instances of someone actually doing this?
(bonus points if you're willing to say "paul graham and arc")
See https://beautifulracket.com/appendix/thoughts-on-rhombus.htm...
The major reason given for that was easier maintenance in the future so now that's completed, I wonder whether the effort will now will be re-directed back to Rhombus.
I keep a zoo of exotic languages, many of which must be build from source. I took a plunge buying an M1 Mac mini, imagining that emulation would hide the transition. Not so for the zoo. At least I have other machines.
For actually using a language, one wants there to be a critical mass of sophisticated support. Racket has long had best-of-class documentation, but this technical feat of native M1 support is a serious feather in their cap.
It is possible to use to write some basic things and most existing Racket libraries are usable with the prototype.
They are still in the brainstorming phase and you can look at the issues in this GitHub link https://github.com/racket/rhombus-brainstorming to see what is being discussed
Who told you Lisp, and especially Common Lisp and Scheme weren't made to "appeal to people"?
I'm not sure if this counts as "trying to appeal to new people" (or "an established language", for that matter) but the Rust community has made a big push towards creating more teaching materials that make the language more broadly accessible for the past ~5 years, and I think that's been effective.
I think the stronger the standard library, the more likely your language will be adopted. If anyone out there is building a language and has strong funding, invest in database drivers and a strong web framework that compliments your language and showcases its capabilities. I would love for D's Vibe.d to be more polished, seems they got as far as they wanted to and stopped, I havent seen much new in half a decade compared to other web frameworks.
- extremely simple, easy to learn.
- progressive learning (you can start with very little knowledge and add later)
- strongly encouraging a single way to do things, promoting readability
- minimal scalfolding - write once, run everywhere (in theory)
- many included libraries + a large ecosystem
I used to write in Perl and had to constantly go back to the book to get the syntax right, and one day tried Python, and cut my time to write stuff in half, was able to reread and understand my code, and so were others.
This doesn't seem to be so true nowadays. Python gives you lots of ways to do many things.
Python's rise in popularity is probably because it was a much nicer alternative to Perl. The percentage of non-programmers I would encounter using Python was significantly higher than those using Perl.
And then there was its "batteries included" philosophy.
And, of course, it came preinstalled with most Linux distributions.
That is, it was not uncommon to have a requirement that you cannot install new languages. But you could pip install dependencies, as that did not necessarily change the full system.
So, since Python was already installed, it had a toe in the door that the likes of Ruby did not.
Granted... Dependency management in Python has revealed itself to be a minefield...
Their communities adopted explicit "this language is for DOING shit understandably, not golfing, not squeezing cpu cycles, not researching type systems, not timtowtdi, etc." That helped tremendously.
In pythons case, it was very easy to wrap up C and C++ code and lift it into a nice environment with a repl and everything else you need to experiment. This put it in an excellent position to grow along with the scientific community. Numpy was a huge asset here.
The rise of rails, Django, heroku, MacBooks as the dominant coding environment etc all fostered an environment where coding went from being nerd shit to being fashionable. This ushered in the era of "people who code for reasons other than being huge fucking computer nerds," in which the benefits of Python and Ruby mentioned above became even more relevant.
Most of the claim of what the community in successful languages did, is so steeped in confirmation and survivor bias that it is hard to see it as more than just popular marketing.
So. No. I am not ignoring these claims. But nor will I assume their truth. If you have actual studies showing they are easier, I'm interested. Enthusiastic claims from members of the community don't count, sadly.
I don’t think that the DrRacket UI deserves the criticism in this article. I usually use Emacs for lisp languages (Common Lisp, various Schemes, and Haskell) but DrRacket is good enough.
The beautiful thing is that we all get to decide what programming languages we want to use. It is true that sometimes our jobs force a language on us, but we have freedom to peruse jobs using tools that we prefer. For a personal project, I recently used Swift and SwiftUI. Swift is a great modern language as is Rust.
Racket is amazing, great community around it. I started writing a Racket AI book a year ago, but shelved the project because of other commitments. I hope to get back to that someday.
Anyway, try to use the language(s) you prefer.
I think this is the most important thing. Say I'm writing a web app -- I want a language that can run on both the server and browser and that has a web framework.
Or say I have a database -- I want a language that can talk to it.
Or say I have to interface with some data format, such as .png -- I want a language that has libraries that can read and write it.
> I think this is the most important thing. Say I'm writing a web app -- I want a language that can run on both the server and browser and that has a web framework.
One of the things that bugged me initially about Racket (and still kind of does) is the implicit typing of some of the functions. Take `length` for example:
$ racket
Welcome to Racket v7.2.
> (length "asdf")
; length: contract violation
; expected: list?
; given: "asdf"
; [,bt for context]
I personally favor default abstract protocols over explicit type:
$ sbcl
* (length "asdf")
4
Maybe this is a silly peccadillo, but as I wrestled with Racket over time I found myself frustrated by this in a way that Python, say, didn't. It's these sort of bits that I'd like to see sanded away, and not the parentheses, to make for cleaner, easier-to-read, easier-to-write code. More concise syntax for indexing and other such things could be nice, too! Standard means for accessing nested hash tables (here's looking at you, deeply nested JSON! Who wants to `(hash-ref (hash-ref some-hash key1) key2)`? Yes, the language can accommodate more concise idioms through macros, but it seems reasonable to have standards around this access.
I really, really want to use Racket: I'm attracted to DSLs (maybe moth to flame) and boilerplate reduction, but it's been a challenge to get over these humps.
The Racket module system does provide ways to use the names you want without having to change the core language. The `collections` package (https://pkgs.racket-lang.org/package/collections) also smooths over some of these issues.
I knew there were historical reasons behind some of these things (just like the old complaints about car and cdr not being called first and rest). Sometimes it feels like this hewing to history comes with some baggage that makes adoption a little more challenging.
I find myself writing the following in some of my projects, for example:
(define-syntax (hash-ref* stx)
(syntax-case stx ()
[(_ hsh k1)
#'(hash-ref hsh k1)]
[(_ hsh k1 k2 ...)
#'(hash-ref (hash-ref* hsh k1) k2 ...)]))
> (define hsh '#hash((a . #hash((b . c)))))
> (hash-ref* hsh 'a 'b)
'c
It's first and foremost an academic project. It's not really the job of the core team - mostly academics - to implement a all those utilitarian bits on top of the language. Their job is to work on ideas in the design of the language itself.
Ideally, then, that work enables whatever wider community of non-academics it can attract to build web frameworks and database bindings and whatnot.
That said, I do get the impression that Racket hasn't done anywhere near as good a job of carrying things through to that second step as comparable communities such as Haskell and Pharo have. The rumor mill would seem to suggest that the project's leadership engenders an ivory tower culture; and perhaps there's something to that? By contrast, even as someone who still hasn't finished learning Pharo and isn't really active in its community, I've still managed to have some direct (if small) interactions with Stéphane Ducasse, and that does, in some subtle way, leave me feeling more personally motivated to contribute to the project.
Considering the whole article is about how to get more users for Racket, maybe they're not doing so well in the dating pool? The author also seems to have forgotten static typing. Isn't Typed Racket precisely the Lisp crowd seeing a need to be like other languages?
You could argue the same point about language "becoming more than Lisp" though, they add features because they are good features, not to be "mainstream". That doesn't change the fact that Lisp is far from a pioneer in static typing compared to other languages.
Emacs is too involved for a casual hobbyist like me, while Dr Racket feels a bit toyish.
what brought me to emacs first was intrigue - why do people keep saying that this little shitty-looking weird editor is the most powerful program on my computer :) they weren't wrong
There's supposed a one click dev system installer, but it hasn't been updated for a while. There's a many click recommended collection of tools, but it looks like half a day of work to get it all installed and then you're supposed to use Emacs.
You can add a REPL to something like Sublime Text. Or install Atom and go through another convoluted process which launches a Lisp server and etc etc etc.
I gave up and got back to work at that point. It might work smoothly if I worked through it all. Then again... it might not.
Which is a shame, because I like the look of Lisp very much and I really would like to play with it.
Also, I have shared https://github.com/susam/emacs4cl which provides a quick starter file to set up Emacs for Common Lisp development pretty quickly. Additionally, it explains every line of the quick starter file, in case one is willing to learn how it is setup and customise it for their needs.
Emacs is my regular editor so I do use it when coding lisp but I don't use any extensions other than syntax highlighting (so no REPL or SLIME).
I sort of get the desire to use the "one click installers" like portacle but in reality they don't add very much and now you're dependent on a third-party system with less support than the language itself. It does not make sense to say Common Lisp on Mac is broken because a third-party installer isn't great.
brew install sbcl.
For the price of an email address, you can download one of the proprietary CL IDEs https://franz.com/downloads/clp/survey and try it out, maybe by following their instructional videos/slides https://franz.com/services/classes/ Then after that you can be better informed if you want to pay for their IDE (or check out the other proprietary competition) or invest effort in emacs/vim/the atom plugin/something else.
I'm curious what languages do you use, or have used, that have shaped your expectations of setup? Do you remember what the first time setup was like for them? For me, I first started learning PHP in 2004, and over the next few years learned or played with JavaScript/Java/Python/Perl/Ruby/C/C++/Scheme (Java and "C with classes" C++ mostly in highschool). My "setup" for most of them was the same, just EditPad Lite, which was better than Notepad only in two ways: you could have multiple files open in tabs, and you could just save-as blah.ext without having to first select "all file types" so you didn't end up with blah.ext.txt. No syntax highlighting or indentation, I didn't appreciate those until after I had made the switch to vim sometime in 2007, though even my vim usage was pretty bare-bones for a while.
But still, for the longest time I just edited files, saved (and maybe FTP'd or compiled), and executed either in the command line or by pointing my browser at the files. When I played with Scheme (by following some of SICP), I changed things up a bit to also match what I had started doing with Python, which was I'd still edit/save/run most things, but occasionally I'd copy-paste from the editor to the REPL and play in the REPL directly for a while, copying stuff back out if it was useful. This was fine, and still is fine! You can have this bare-bones experience with CL, too. If you've installed SBCL, you just have to run `sbcl --script file.lisp`. If you launch SBCL by itself (it's even better with rlwrap), you can just play in the REPL. If you were doing an intro-to-programming course for highschoolers, would you have them setup much beyond any random text editor and telling them how to execute stuff?
On the other hand if you want a more "professional" environment, is it really reasonable to expect to not have to spend some time setting things up? Especially with CL where if you want more than the bare-bones experience, i.e. developing with the REPL, you.. need to have something that can talk to a REPL, and not just a pure text editor. One of these days I should try out https://microsoft.github.io/language-server-protocol/ and maybe I can just recommend it, I know someone made a server for CL that just wraps the classic one, which should let any editor with an LSP client have a good experience. But taking from my own experience again, when I took my second job writing professional Java, Eclipse was basically mandatory, which I hadn't needed really before, plus there were some company-specific extensions you had to install to make it work better, and anyway it took me a while to get used to Eclipse itself and learn the shortcuts. I never thought this was unreasonable...
I wrote way more than I planned... all I really wanted to say after the Allegro recommend was I think if you really want to try CL, just go for it, the bare-bones experience can give you a taste just fine, and maybe encourage you to take the pains of setup for a more professional environment (for which emacs, while popular, is not the only choice). If you want to follow along a book like PCL (https://gigamonkeys.com/book/) or Land of Lisp (http://landoflisp.com/), you don't even need an editor, you can just type things in.
(Disclaimer: I am the maintainer of Magic Racket)
Simply not true. Scores and scores of IT people came up on Microsoft. For many it was the door to a technology career.
As one's career evolves it becomes easier to pick what you want to work on.
https://github.com/jeapostrophe/racket-langserver/issues/45 https://github.com/Eugleo/magic-racket/issues/29