Clojuring the web application stack: Meditation One
evalapply.org
evalapply.org
The consummate Clojurist's default (and very normal-feeling way) to build a web application (or any application for that matter) is to roll their own web stack from production-grade libraries.
Of course, this state of affairs is a double-edged sword, just like is true for traditional web frameworks. In my post, I try to go into the whys and the wherefores, building upward from first principles.
[1] Biff – a batteries-included web framework for Clojure https://news.ycombinator.com/item?id=44037426
I'd say the nice thing with the Clojure way of building your own stack is it becomes quite easy to swap parts out. On a previous project we swapped out our web server three different times with minimal changes (jetty -> aleph -> httpkit) as for the most part they all shared the same interface.
After a while you get good at seeing where you want things to be configurable and where you don't. It also gives you the confidence to roll your own micro stack/framework which means you are not dependent on third party aggregates to adopt new features.
Yes; next to the sheer stability of parts, their fungibility is a business-critical feature of the Clojure ecosystem. Of course said fungibility does not magically manifest. However the effort to get there is "not much", I'd say. The use of "system" libraries, with some well-reasoned module design brings it pretty close to magic.
As in the post, a fungible, production-grade part can be just a multimethod (e.g. the router in the post). Why? Because "production" comes in all sizes. A small SaaS with a few hundred customers may chug along happily with a bunch of functions.
You could just as well say that PHP has “production grade” functions.
To analogise further...
- HackerNews is a "production" system, you would agree. Back in 2015, it was still true. Are flat files a "production grade" primary data storage choice? [1]
- Suppose your production service transacts a million requests an hour (say it is a short-link maker). Further, let's say it has only a handful of API endpoints. Do you really need a whole routing library for that, if a single multimethod does exactly what you need, correctly?
etc...
[1] https://news.ycombinator.com/item?id=9990630
(edit: add reference for flat file storage)
My personal site is statically generated — it’s just a bunch of HTML files and some CSS. Do you not think it would be a bit pretentious of me to describe that as “production grade”?
If “production grade” simply means fit for purpose, then given the GP commenter’s reason for their initial web server swap, wouldn’t you say by definition that jetty is not production grade, since it doesn’t (didn’t?) support SSE?
A static site that serves is most definitely a production system. Perhaps one that could scale in traffic almost without end.
evalapply.org gets (to my continued amazement) 20K+ unique visitors a month when it's business as usual. On a busy HN day, it's easily that much in hours. I don't have to think about "scaling problems". I don't have one.
I'm being the realest real with you.
Why is this desirable?
On all my projects over the past 10 years, I’ve swapped out the web server exactly zero times because the one I have works just fine. The parent comment describes these components as “production grade”, but then if that’s the case, what could be the reason for swapping them out other than self-indulgence?
In our case the first time we needed to, we needed SSE and at the time there was no ring-jetty async interface (it's a long time ago so I'm forgetting the details) so we moved to aleph. Much later, we wanted to try out http-kit (self-indulgence) as we were operating behind a proxy anyway for performance reasons and it made a significant difference.
If we'd just started with http-kit that would have been fine. I guess it comes down to what features you need.
Also I 100% agree it's something ideally you would want to avoid a in the case of databases for example so much performance is left on the table because for some reason we want to be able to swap between SQLite, postgres and mySQL. Which in practice you never want to do.
Generally, yes one would not want to swap out their web server willy-nilly...
Yet, this is one of those "YAGNI in 99% of your use cases", but when that 1% use case arises, a server swap would be far more desirable than a whole framework shift.
So while self-indulgence can certainly be a motive (and why not? as long as everyone's having a good time), may I offer a few more charitable reasons for this:
- programming API ergonomics
- performance
- application runtime model (servlets -> embedded server)
- security model
- application server features (websockets, comet?)
- binary size
- server configuration niceties
etc...
That said, a developer only has flexibility if it is built in from the get go.
A counterfactual would be to consider the set of developers who have had to put in ugly hacks because they can't just rip the web server out of the framework of choice they are locked into.
(edit: bullet list formatting)
Because I pay the people who work for me.
Because Brawndo's got electrolytes.
The part about building just the functionality you need, using the bare minimum libraries etc stuck out.
NASA's Common Metadata Repository is worth exploring too https://github.com/nasa/Common-Metadata-Repository
It is a neat example of how an org can structure and manage multiple projects and services in a single git repository. They've use Leiningen to achieve their objective.
> The Common Metadata Repository (CMR) is an earth science metadata repository for NASA EOSDIS data. The CMR Search API provides access to this metadata.
> Building and Running the CMR
> The CMR is a system consisting of many services. The services can run individually or in a single process. Running in a single process makes local development easier because it avoids having to start many different processes. The sections below contain instructions for running the CMR as a single process or as many processes.
(edit: add relevant context for quick reference)
https://flownet.com/gat/jpl-lisp.html
> Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem. The story of the Remote Agent bug is an interesting one in and of itself.
- Clojure: https://clojure.org/community/companies
- ClojureScript: https://clojurescript.org/community/companies
Also, a few case studies may interest you: https://clojure.org/community/success_stories and community stories: https://clojure.org/community/community_stories
We are also hiring Clojure devs: https://www.metabase.com/jobs
The web frontend is written in TypeScript/React.
I've now had it several times in the years-long lifespan of small clojure web projects that people have moved on and the thing (framework-ish) basically doesn't exist anymore and going by the issues it only had like 10 users in the first place.
It's not the end of the world, and fortunately there's not a lot of needless churn, but I guess I would prefer to have this "I am trusting project x and I only have to care about their releases (pre-testing all the moving parts) and then my 5 dependencies" and instead I have 20 dependencies/moving parts for my web app.
Yes, I'm lazy and I don't think it's a problem in an env where you have a proper dev workflow anyway.
How often do you have to do that though? I was under the impression that Clojure was a bit like CL in that old code will keep working basically forever. Unless there are new features you need, you just leave most libs alone to do their job.
that’s HTMX, Air, Red & Cro btw
that said … I am a true believer in HTMX for the right amount of UX dynamism and I don’t initially get solves that piece
Following writing advice or post structuring guidelines is not in my job description at evalapply.org Luckily, Michael Hamburger offered a legitimate excuse in his classic (so I'm told) essay, "An Essay About Essays": https://substack.com/@bombaylitmag/p-162583447