Lisp and the Web: Creating Web Apps Through Lisp and GCE
medium.math.dev
medium.math.dev
So I saved these notes in guide form, hopefully its useful for others. The trick is to extend a lot of this, all the hard parts are here - the rest is just basic Lisp, HTML, JavaScript, building upon this platform.
Highly recommend Lisp for Web Development, its *much, much better* than any of the other frameworks out there! So easy to use. Do learn HTML, CSS & JavaScript though. Esp. JavaScript as you would want to offload some of the computation to client-side to reduce your server costs.
Big Thanks to u/linnilbobo on Reddit for his code on user authentication in Hunchentoot - that was one of the missing pieces for me.
Could you elaborate a bit on this? I've never written a single line of Lisp so I get a bit curious.
- A fantastic and easy to use open source web server in Hunchentoot
- Very easy integration into databases like PostgreSQL. Just looking at my guide shows how easy all this is (assuming you know lisp, but if you look at the number of lines of code - it’s not that much)
- Live debugging - I.e. I can recompile certain functions and don’t need to restart my web server! That’s a huge win in my opinion
- It’s hard to explain, but I find Lisp very transparent. With knowledge of JavaScript, HTML, CSS and Lisp, I am very comfortable building any web app. Whereas when I use various web frameworks like react, angular, etc, I have to spend so much learning it’s idiosyncrasies (which to me is a waste of time since these frameworks keep changing), and then usually I hit a brick wall somewhere. Other frameworks in my opinion obfuscate too much - they solve for easy wins (which they do), but anything more complex and you are fighting against the language, spending hours on Stack Overflow for solutions to what should be simple problems. In my opinion
I spent 1-2 weeks setting up a CRUD app with Angular and still don’t know the best way to use it. With Lisp, I did the same in one day. It was that experience that pushed to me to learning and using Lisp (I was a complete beginner at the time). For me that was enough proof
- Stuff like NPM I don’t like because there are so many dependencies (which break from time to time) and I’m worried somebody could be putting in bugs / trackers in these packages. I don’t like having many dependencies
- Ability to write DSLs (Domain Specific Language) where I can easily write macros to generate JavaScript and HTML for me from Lisp, increasing the cohesiveness of the whole program
All that said, Lisp by itself is not enough. HTML/CSS/JavaScript are much more important - Lisp is the magic server side plug for all of this, and in my opinion, better than server side JavaScript because Lisp is a very well thought and well featured language (JavaScript is as well, but I think Lisp, at least to me, is slightly more enjoyable to write in).
I find the issue with Lisp is that it’s not used much, so outside of some very good packages, there are many parts which are easier to do in other languages / frameworks with much bigger ecosystems. For me, web development is the exception, because all you need is HTML/CSS/JS and a good server side language.
Other languages are also very good, so this is just a personal opinion. If you like to invest your time in one core language, and not have to keep learning new web frameworks, I recommend Lisp.
It’s very good for solo programmers or small teams, but I do think it’s not suitable in larger teams as I do find lisp code can be a bit harder to read and also most people don’t use lisp.
So in my uneducated view - if you have spare time, if you like working from first principles and have a scientific / mathematical mindset, and if you are working alone or in small teams, I highly recommend Lisp for web dev.
If you are more business minded and want to scale your project quickly and hire more developers, then using an off the shelf package makes more sense.
If you are time poor, stick to what you know - the differences aren’t THAT great, and it’s better to build something now than to spend most of your time learning a new language (unless you have time to devote to it - in which I highly recommend learning Lisp :) It really makes me smile every time I code in Lisp :)
In my experience, Lisp has this sort of brain amplification power. Whatever weird idea I have about how to think about a problem, I can kinda smush lisp into shape so it solves that problem. with language X, I have to think about how to I say what I'm thinking in X. It's kinda like spending some time working in C then jumping to python. It's far less hassle to get the idea out there. It's not exactly like that, but it has that flavor.
Two sort of related points.
Lisp is pretty transparent. If you need to think about pointers and stuff, you can do that and it'll mostly stay out of your way. But if you don't, you don't.
Lisp will let you represent whatever crazy notion you have in your head. You may understand why the code works, but explaining to other people can be tough, because your internal abstractions are probably not what other people intuitively think about when looking at the same problem.
Not saying any of the above is good or bad. It is different. With java or go or whatever, I tend to think a lot about how do I represent this in java or go or whatever. With lisp it's the closest I've ever gotten tot my idea out there, naked. That can be powerful, or scary, or both.
You might take a look at "Paradigms of AI Programming: Case Studies in Common Lisp". That shows a lot of tricky problems tackled in a pretty breezy way. Want to build a computer algebra system? Not that tough to get something working. Work through Norvig's approach and you can get something pretty good working pretty quick.
Once you flesh out the idea in lisp, it’s just a matter of implementing it in the language of your platform. It’s like doing ‘architecture’.. Lisp is about aesthetics and beauty
C programs don't understand other C programs. Of course the C compiler itself is an exception but in general you don't expect to be able to feed your compiled C .exe another C source file and do anything useful with it. That's crazy talk. Likewise with Java or C++ or Fortran or...
But Lisp programs understand Lisp programs. The compiler is always available even at runtime. And not just the compiler but the lexical analyzer, the parser, and the evaluator too. Output is also easy: You can write a Lisp program that builds Lisp data structures dynamically in memory and then prints them to a text file as Lisp source code. Which you can later read back in and execute as a program.
This kind of thing is so natural in Lisp that it opens up whole new ways to think about code. So why is this useful?
One example: In many programs you need configuration files. So you write configuration files for C programs in XML or yaml or JSON and then use a library function to parse them. Why don't we write configuration files for C programs in C syntax? Well, because C syntax is rather inconvenient for this purpose and because there aren't many C libraries that parse C syntax. Besides, how do you handle includes, compiled libraries, macros, linking, etc? Again, crazy talk.
Not so in Lisp. Lisp syntax is very convenient for configuration files (because it's s-expressions) and the parser is built-in. So you write configuration files in Lisp and parse them at runtime! If you change a setting at runtime, it takes not much more than one line of code to automatically print a new configuration file. Which you can then tweak with a text editor if you so desire.
Another example: Macros. Macros use this same principle but there's no external file involved. Macros change Lisp's syntax so you can write more advanced programming constructs and translate them to basic Lisp at compile time. It's the same idea as Typescript-- Typescript is a Javascript program that converts a type-safe version of Javascript into basic Javascript. But Typescript was a one-off major project--like building a compiler. In Lisp the toolkit for source-to-source translation is built-in and Lisp programmers use it every day.
Formatting queries like this is not a good idea at all. See: https://en.wikipedia.org/wiki/SQL_injection
Best way around it is to put in checks to prevent the code from being anything nefarious. I’ll have a think and figure out what’s the best way of doing this, but I assume not constructing the sql strings from scratch but rather using postmodern’s API (postmodern is Common Lisp’s excellent interface to PostgreSQL) makes sense, or the alternate solution recommended in the other reply to your post.
Thanks also to kagevf on #lisp irc channel for his help here
Very nice catch! Thank you very much.
edit: i adjusted my answer because i thought i was answering to parent
Also so cool that you worked in AI! That’s my dream field :)
You have a function (defun my-val () "a string")
Referentially transparent, easy to reason about right? Wrong. Lisp strings are mutable, and everything is a reference. So if you do something like this
(replace (my-val) "b" :start1 0 :end1 1)
(my-val) => "b string"
I appreciate that CL doesn't force everything to be immutable, but it seems like at least strings would benefit from being treated as immutable values.
I thought I could do (string "a string") to prevent it, but that didn't work.
(concatenate 'string "a string") worked, though ...
[0] http://www.lispworks.com/documentation/HyperSpec/Body/03_aba...
Sounds about like most large codebases. Languages can't save you from this, it seems, only discipline can.
You want at least one experienced Lisper who can mentor the less experienced and build guard rails where needed. The good news is that it's unusually easy to build guard rails with Lisp because of that same "bendable" quality you mentioned.
Another thing that is unusually easy with Lisp is building code-analysis tools. The source code is made of standard Lisp data structures that you can operate on with standard Lisp functions, and you can do it all interaactively in the live environment. That makes it much easier to build linters and other analysis tools than it is in most other languages.
You need someone on the team who is experienced enough to know how to do these things, and team-oriented enough to want to do it.
You also need to be aware that you probably don't know what guard rails and analysis tools you need when you're starting out. You'll discover them along the way. So you need project management with enough foresight to invest in prevention up front in order to save on cures down the road.
I'm not the only or primary author of all of these; only a few of them.
- Panels [Apple, circa 1988]: an experimental graphical UI framework with a built-in constraint solver and extensible event-handling system
- GATE [Apple, circa 1989]: a knowledge-based, distributed, automated testing system used to discover and report thousands of application-compatibility bugs in Mac system software before release
- bauhaus [Apple, circa 1991-1994]: an experimental Lisp OS for Newton. The Newton OS (except the microkernel) was written in Dylan back when it was basically Scheme+CLOS; the Dylan compiler and development tools were written in Common Lisp.
- SK8 [Apple, circa 1995-1997]: rapid-application-development system sometimes referred to as "HyperCard on Steroids".
- Reactivity Application Firewall [Reactivity, circa 1998-2004; later Cisco]: Reactivity's product was a complex application-in-a-box that provided rule-based application-level security and integrity for networks. Cisco bought our company and folded it into their product line. I used Common Lisp for several code-analysis and documentation tools used in development of the product.
- Clozure Common Lisp [Clozure Associates, circa 2006-present]: I wrote documentation and code for Clozure Common Lisp and some associated tools
- Secure Outcomes LS1100 [Secure Outcomes, circa 2010]: I wrote the bulk of the control system and GUI for the Secure Outcomes fingerprint livescanner product, using Lispworks.
- Delectus [my own product, circa 2010-present]: Delectus 1, a simple personal data manager, was written mostly in Scheme. Delectus 2, a more capable, flexible, and portable version, is still under development and is written entirely in Common Lisp.
- Confidential project [Confidential client; 2020-present]: a knowledge-based, reasoning-based neurosymbolic machine-control system. As I mentioned, it's confidential, but I can say that it's designed to monitor and orchestrate a variety of different kinds of robots with human supervision in real time.
I've omitted several free libraries and other open-source Lisp software that I've written over the years.
Do you find Lispworks superior to SBCL / Emacs?
Any tips for other programmers on what to do / what not to do when it comes to production level lisp programs?
I use both.
As for tips, there are probably a lot, but I have to take a break from HN right now to attend a work meeting. :-)
SBCL is better in the sense that almost everyone who writes an open-source library for Quicklisp distribution is going to test it with SBCL, so most of the time Quicklisp libraries just work with SBCL.
Testing with other Lisps, including Lispworks, is not quite so automatic, so there's a little greater likelihood that something might fail to work.
That being said, most libraries I've tried with Lispworks have worked without any trouble.
I also left out a bunch of library projects that live on Github.
(If you want me to see if I can help, send me mail at mikel@evins.net and we can talk it over.)
In most of the cases on my list of past projects, the people who organized the projects were Lispers themselves or had contacts in the Lisp community, and recruiting from that community was part of the early project setup.
The bad news is that the community of Lispers is small compared to those of more popular languages. The good news is that Lispers are generally eager to work in Lisp, which can be helpful in recruiting. For example, I prioritize Lisp opportunities over others, and I'll work for less money if I like the people and I get to work in Lisp.
I have been part of hiring and mentoring some programmers that were not experienced with Lisp. In all but one case, they were interested in learning it, and made quick progress.
I've worked on a few teams of mixed Lispers and non-Lispers, ranging in size from about three to about sixty. I've found that you want one or more experienced Lispers who are willing and able to take on a mentorship role. With mentors, good programmers can pick up Lisp and its style and idioms pretty quickly (assuming they want to). Without mentors it can be rough going.
Configuration is very simple (for example, reverse proxying is just three lines - for the entire config!), and SSL works out of the box!
Thanks for the kind words
(bt:make-thread (lambda () (swank:create-server :port port :dont-close t)) then connect with M-x slime-connect
I've debugged many tricky to reproduce issues like this. Also it's just damn cool.
So you are running a second server to test issues away from production? I thought this is possible in other languages / frameworks, so I must be missing something
I read the cook book but still didn’t get the main benefits of it. However the quote was very cool:
That’s how Ron Garret debugged the Deep Space 1 spacecraft from the earth in 1999:
We were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem.
Swank doesn't allow remote connections though, otherwise anyone could connect and mess with your server. You must connect through the loopback interface. Easiest way is to set up an ssh tunnel, like: ssh abc@example.com -L 4200:localhost:4200
Whether you can call the Swank server a second server or not is a somewhat philosophical question I guess. There's one application serving different kinds of requests on different ports.
The Swank connection runs on a different thread, not on a different server.
Besides that, nice and interesting write up!
I agree with you, it’s not perfect for readers and until I caved and signed up as a member, I also was annoyed at hitting my monthly limit of articles.
They have amazing SEO so inevitably many articles I need while researching a solution led me to Medium.com
It’s just a very hard market to serve (long form reading / journalism etc). That’s why I’m not that bullish on substack because it’s hard to make it good for readers and profitable for writers / the startup.
Medium paid me $10, and I have about 10,000 reads in the last year. I didn’t do it for the money, but that’s an indication for those who do want to write for a side income - it’s a hard business.
For writing, its very good - to the extent, I use it as one of my main word processors lol. Then just copy it to something else for formatting. Very distraction free and the font is nice to make you want to write more.
It would be great if a new competitor did what Medium did, but made it better for readers. IMO the business model should be like Wordpress but for writers, together with an overall platform with high quality SEO. Make the money selling the software and a bit with memberships on the main site.
I feel for Medium and it’s writers - there’s so little money in it, that to some extent they have to resort to click bait titles, bad UI for readers, and high amount of not amazing quality content.
Perhaps a donation model is best.