I recommend CGI instead of web frameworks
halestrom.net
halestrom.net
The reason we stopped using it 15+ years ago was performance: forking a new process for every incoming web request just didn't make sense on ~2000 era hardware.
I wonder how true that is today, given that our machines have vastly more RAM and CPU?
If you squint at them the right way AWS Lambda functions are pretty similar to the CGI model.
I see that starting up a Python interpreter and running a Python program (or Ruby or whatever…) might be slow, but how far can I get with a golang, C or ocaml binary?
I don't know, but I'd expect the Linux people to have optimized that to death.
2000 era CGI tended to be in Perl or PHP. Even then Java made an appearance, with a long-lived host process, e.g. Websphere.
One advantage which the author gets right is that CGI consumes no resources while not serving requests, so you can have a lot of different CGI programs on the same machine. It also plays nicely with multi-user systems, so an ISP could host a lot of different customers on one box. We could serve CGIs from several hundred users off a 128MB Pentium system, for significantly less resources than one Slack instance.
- modern frameworks have a much higher startup time (see Python imports, Java VM and other). CGI was fine to run a perl script with no dependencies.
- it prevents any form of caching. caching is very important for many use cases.
- it requires to open a fresh connection with every request, to the database and elsewhere (too bad if you thought you could use redis for caching).
- SSL is everywhere and it has a notable overhead on initialization, meaning you really wish you could reuse connections. (Not just the databases but to other API services)
Speaking from experience, I've inherited the CGI platform at JP Morgan (the largest US bank), that went back to 2006 and approached a thousand running applications at some point. It works and it was still the easiest way to deploy any application in 5 minutes 2 decades later but the drawback are real. It's only for short scripts that can tolerate a 5 second startup time and zero caching.
https://thehftguy.com/2020/07/09/the-most-remarkable-legacy-...
> it requires to open a fresh connection with every request, to the database and elsewhere (too bad if you thought you could use redis for caching).
nutcracker, twemproxy?
Maybe current web servers are written in such a way that they do cut connections between CGI requests, but I’d be surprised if they really have to.
With CGI, any connection you create in your script is going to close at the end of the request when your process shuts down.
That's fine, they can keep. TFA suggests using CGI in lieu of modern frameworks, not WITH modern frameworks.
>- it prevents any form of caching. caching is very important for many use cases.
No, it really doesn't. You can cache whole responses with a reverse proxy on top, or you can cache complex results in whatever (Redis, disk, etc) and query it from the CGI program.
>- it requires to open a fresh connection with every request, to the database and elsewhere
Which might be fine. MySQL for example is notoriously cheap to connect to for each request. Redis as well. (And you can always bump CGI to FastCGI).
>- SSL is everywhere and it has a notable overhead on initialization, meaning you really wish you could reuse connections
You can terminate SSL on your proxy, so that's not a issue.
There are alternatives; even back in 2000 you'd keep the stuff you wanted cached in the filesystem, where the webserver would handle the cache headers for you, and there are ways to hook a cache miss by the webserver to generate content on demand.
It doesn't scale to what we do on the web today, of course.
It's a good place to start. Not a good place to stay.
It's very common even to do this: just use Apache `mod_proxy` (or something more specialised) to forward requests to a backend server such as Apache Tomcat.
Need a site to order XYZ? Unless you are a mega chain, small biz doesn't need to do anything more complex than serve its 100 customers reliably. Most 'modern' stuff is wasted abstraction for complexity which does not exist.
Once you've switched to hard-compiled Go, vs. some interpreted language, you've obtained most of the possible speedups.
That's really all you need. I have a moderately busy server that's been up for over a thousand days without a reboot or restarting the FCGI service running under Apache.
You have to have a huge load before FCGI, multiple servers, a load balancer, and back-end database machines are not enough. Wikipedia used to run on something like that, until they hired too many people and had to keep them busy.
So the remaining advantage to use FCGI with Go is that memory leaks would never be an issue?
That's overly strong. I will create a CGI app every now and then. I can use basically any language I want. Not a lot of thought has to go into it. As the article says, it's a simple approach that makes sense for those of us that aren't so familiar with web development (we're usually making those apps for ourselves). In particular, the "you can never have too many dependencies" philosophy of modern web developers is strange to me.
It isn’t loading, just sits there spinning. Pile of shit. I wish they’d chosen to write the service as a reliable cgi rather than some web framework where errors are eaten and hidden in Ajax calls.
Uber eats let’s me place the order then vanishes when it comes to paying.
Nandos yesterday told me error UK03 when I tried ordering in the restaurant, slightly better, but perhaps those more concerned with scaling to a million concurrent users could deal with 1000 reliably first.
[1]: an oxymoron, I know
I think it is also worth to mention there are plenty of "slim/light" frameworks that do very little (i.e. take care of threading, provide html request/response as writable streams, have the http headers parsed for you into a Map-like data structure etc.). Those lightweight frameworks exist i.e. express without middleware (nodeJS), Nancy (.NET), Flask (Python) etc. The "full-blown" counterparts would be ASP.NET or Django which bring far more to the table than you might need.
I understand that they can solve the problem of horizontal scale as they are spawning a container rather than just a process, but surely if you started with CGI scripts it would be easier to move if you needed to at a later date.
Even for a web-facing tool, as long as it doesn't receive more than a couple requests per second CGI is good enough, and that probably represents most web apps in existence. YAGNI principle applies, too. Most people don't work at Google of Facebook or hyper-scaling startups. A fantastic number of real-world development consists in building boring web front-ends for in-house use of random companies; and 99% of accounting forms or support tickets won't require huge performance, even running with CGI.
And on the plus side it scales to bigger things thanks to FastCGI with for instance nginx.
At work, every new team member get the task to add their profile to our team rooster. Works great thanks to the low barriers of changing a single PHP file and the low risk of breaking the whole web page is great especially for juniors.
As a strategy for dealing with xss, its fallen out of favour, but static taint analysis, which is the same thing but not at runtime and less accurate is still super popular in big shops as a CI step.
As an approach though its more a way to make sure you dont screw up as opposed to a way to solve the problem in general.
a) static documents - basically HTML+CSS and no scripts on the back-end
There is not much to discuss here a lot of stuff could be just that but people don't want to write blog posts directly in html :) We have static site generators that are doing great so it seems to be working well.
b) dynamic documents - you get data from DB based on query, like list of phone numbers in Texas and want to find specific city
Static page generators would not be that useful if one wants to reflect changes from db. Queries are also nicer in db than having insane long list on page with CTRL-F. I would say a CGI only thing would work great for such use case. You probably want to thing about SQL injections but as it is browse only then might not be an issue.
c) web applications - here people want all bells and whistles
Security is important as you probably need authentication and preventing XSS is quite important here. I would never build web application with only CGI - security headers are not that hard to add. But authentication and authorization + XSS prevention is really hard. Then you have lots of requests that send/filter data. You can have problems with SQL injection as you have to store some users and their passwords and their data, framework+orm helps preventing a lot of troubles. One probably should not use a framework for making his blog/static-page. Unfortunately nowadays most people build web apps.
This is what rubs me with posts about "you don't need X framework, it all should be static documents", well yes you don't need big framework if you build personal website. You probably need one if you build a web app. Downside is we have HTML+CSS as interface that was designed as document framework and not as application interface building framework. That is why we need a back end and front end frameworks.
You still care about xss (and other vulns, although the technology the grandparent mentioned is specificly for xss) in case b just like case c.
- you might host other things on that domain (or even subdomain, which is a lot trickier but not impossible to attack), and this could be a launching point of an attack.
- attackers might rewrite your page to mislead people, e.g. as part of a phishing attack, to harm your reputation or just to redirect to advertisers.
The impact of any security vulnerability is going to depend on what you are doing and what you have to lose. It seems less significant in case B, but its a mistake to extrapolate as its impossible to tell without business-specific context. Maybe the simple list page is listing out life-saving information.
I think it's fine to play with CGI to learn, but I wouldn't push that to production.
Sadly I was at an agency which (I suspect like many other places) just threw stuff like establishing database connections straight into the templates, and had ludicrously complex stuff being done in VBScript, a language designed for light automation.
The programming model does have its advantages though. Persistent servers risk leaking information across requests (seems to be a particular risk in async programming systems where requests execute concurrently in the same thread) so there's a definite advantage to isolation between requests. I'd love to see a server that used per-request v8 isolates (with snapshots for fast startup time.)
It's much less likely you'll lookup the wrong user's session than accidentally store some user data in a module or closure variable shared between requests which may come from different users.
Do not store sessions in signed cookies, because those can be stolen, and they bloat your requests and responses shipping all the data up and down each time. Store sessions in a database, tie them to an IP, and time them out. Transmit only an opaque ID. Ideally, re-create a different ID with each new request. If you are using a web framework, it should have tried and tested mechanisms to do all this for you. This is the kind of thing you lose cooking your own CGI.
Most websites are either primarily static, with no server side code, or they’re fully dynamic, where every page is generated. CGI shines when sites are a mix.
One other alternative to CGI that the author doesn’t mention is reverse-proxying a single page. I find that to have more practical use than CGI.
There is nothing else.
Sure, if you want to learn html just write a page and reload in a browser. But that doesn't teach the simple bits of talking with a server from a browser.
I def agree that there is no modern framework that has an easy mode like this - they are all abstractions which insist you learn a bunch of local, mostly throwaway, tribal knowledge.
Do you want to build a multi user scheduling system to serve tens of thousands of users or robots, probably not. But its perfect for hooking up the output of a couple variables to a webpage.
These web servers are battle-tested and feature-rich. If you add another layer like an application server (Puma, Gunicorn etc) that sits between the server and your programming language does the application server end up duplicating some (most?) of the functionality of the web server?
Yes we do. All. The. Freaking. Time.
It’s availability bias: most of the web sites we visit are popular and big and complex and have had to solve serious scalability issues. Most of the web sites we make have few visitors and are small and simple and do not have any scalability problem beyond the occasional aggregator hug.
Likewise, most of the software we use is big and complex and used by many. Most of the software we make have less than 10 users. Google, Facebook, Microsoft, Amazon… are everywhere, but a stupidly small proportion of companies in the world are as big as they are.
Mike Acton urges us all to "understand the data". That includes how much data we’ll be processing. How many request per days are we expecting? Are they evenly spaced, or will there be spikes? What’s considered acceptable latency? Stuff like that. Remember, the Pirate Bay at its most powerful only needed 4 rack servers, on top of each other. Very few of us will exceed the capacity of even a single server.
It’s not always easy to see, because the front page of HN (and the front page of pretty much anything for that matter) doesn’t feature the ordinary. So we only see the extraordinary, and get the impression that we have to measure up to that.
When you're saying it's preferable to serverside web frameworks you probably ought to point out that it does far less for you. I started out in web dev working on Perl CGI scripts and they were great (once you got your FTP app to use the right CRLF encoding), but you really had to do everything yourself. Some devs might see that as a benefit but I don't.
Most of the complaints here are about performance. I don't think the decreased performance would be much of an issue until you get to pretty large scale. I do think the real issue is the huge complexity in doing things correctly and securely in such a manual environment. Oh, you're going to do cookies by just printing the Set-Cookie header manually? Well now you have to handle everything about your cookies manually and do it correctly. What are the odds you're doing that? Just let Rails etc handle it the right way for you. Going to do CSRF protection manually too, and do anti-XSS escaping correctly everywhere, and mitigate a ton of obscure security issues that most people have never heard of? No way, unless you're a world-class expert. Just use one of the major proven frameworks that already take care of all of that stuff for you.
The newest version of my app uses service workers to store almost all the app code in the user's browser. Almost everything the user does with the app is done on the client side so for the most part they only hit the server to get or put data in the server side database (CouchDB) and for the most part those are very small gets and puts.
Compared to earlier versions going back to 2002 my server barely has any load on it. If your goal is to track every click a user makes then CGI isn't a great server side option, but if your goal is to make a fast and reliable app than an offline-first/local-first side benefit is you don't need a huge box or tons of bandwidth to run it and CGI scripts are a fine way to handle small server side chores, which is pretty much all that's left.
Ruby, python, PHP every server side tech use FCGI at the backend, only exception being stacks that have built in webserver [Java].
FCGI works mostly like cgi, but the program doesn't terminate after execution, rather continues to listen for next request and serves it. Runs like a daemon.
Web frameworks are libraries/helpers on the application side to help with business logic for serving requests.
You can use CGI with web frameworks (look at the ton of useful PHP/Perl/Ruby frameworks out there).
You can also build a fully competent website without CGI OR web frameworks. Modern languages now all have built-in web servers which perform a lot better so Apache/nginx etc. need to function at most as reverse proxies.
In fact even if teaching is the goal I'd argue that Apache/CGI introduce more opaque abstractions, not less. You can create a web server and request loop in any language of choice in like 10 lines and take it from there.
Deployed: https://k.malhotra.cc/go/
Code: https://hn.malhotra.cc/git/cgi_k-malhotra-cc/tree/script.py?...
https://sqlite.org/althttpd/doc/trunk/althttpd.md
Heard about it in the Changelog podcast the other day.
Plus, just look at this dudes website. Sure it works just fine but it's nearly impossible to read on my large screen. It feels like it was designed for an older CRT screen and has never been updated since.
I wonder how people like this can survive in todays world with the requirements people have on web software. You will sooner or later be out competed by a guy using a web framework.
Using the browser's Reader View mode I can read the website beautifully just fine. In fact, this is my preferred way of reading articles on the web, because I get consistent and controlled styles.
> I wonder how people like this can survive in todays world with the requirements people have on web software. You will sooner or later be out competed by a guy using a web framework.
Not everyone doing webdev is competing to get a job or gig as a webdev. There are people who makes their own sites for various purposes, i.e. they are their own webdev customers. And if you're suggesting these people to "require more", it looks to me you start to meddle with how they run their business (which might benefit from your suggestions, but also maybe not).
I digress, learning CGI might be useful in a historical context but as others pointed out its pitfalls are large and there's a reason we don't use it anymore. If you've ever seen a mess of a perl and bash to try to serialize some JSON you'll know why. It encourages bad behavior that doesn't scale well.
If your students aren't understanding template generation and mapping routes to functions they probably lack a background in a lot of fundamentals. At best they'll simply copy and paste best practices without understanding why. I think taking a step back and looking at interops, FFI and APIs will explain why web servers and the web itself became popular. To understand that you begin to need to now the basics of operating systems and by default compilation and some other undergrad theories. These aren't taught just for fun.
That said modern web frameworks hide a lot and that's not necessarily a bad thing. In attempt to make things isomorphic and routing client-side the delineation between client and server is blurred. Even I had a had hard time figuring out whether things were rendered on the browser or the client-side and WASM blurs this further.
The weird late 90s derail into "just use Windows" is a huge red herring too. I use OS X, and I've used Windows. I spend most my day in a terminal or a Linux VM and still won't have Linux as my primary desktop. On the same hand if you've ever had to do something low-level on Windows or a Mac it is painful. See Linux's cpufreq vs whatever Windows or Mac has to deal with big-little CPU architectures. The trade off is that the desktop experience is brittle at best. That's okay there's a hundred of ways to develop for Linux on Windows and Mac, that's a solved problem.
I'm seeing a lot of React jockeys come out of bootcamps like these with a cursory understanding of computing. Similarly I see people come out of top colleges thinking they'll be perfecting algorithms in Rust. Neither is right, but the React jockeys are dangerous as they have a "works on my machine" mentality that a lot of us learned the hard way doesn't work. Now it seems you can sort of throw cloud resources to make it go away. I'm not saying everyone needs a CS but a fundamental understanding of what you don't know can go a long way.