1993: CGI Scripts and Early Server-Side Web Programming | Hacker News Reader

1993: CGI Scripts and Early Server-Side Web Programming

webdevelopmenthistory.com

146 pointstmfi96 comments

I feel like CGI Perl scripts back in the late 90s was definitely a critical reason for me picking up on programming, web dev (as a high school kid), and eventually becoming a professional web developer, and software engineer.

The whole web dev paradigm back then starting from basic HTML files, to client JS, to server-side Perl CGI scripts, was just really easy to wrap my head around, and actually create usable products out of. It maps well to a Windows OS file system -- you have these HTML files sitting in folders, you can write them locally and open them locally to see them. Upload to a server via FTP in the exact same structure, and see it work on the internet. If some files are CGI scripts, they run some code on server before output. Input from browser comes in as stdin in the script, and output to browser is just stdout. It was simple enough that as a high school kid, I was able to start one of those "webmaster services" (providing guestbooks, counters, etc.). I have nothing but fond memories building scrappy stuff back in those days.

With that perspective, what are your thoughts on the modern ecosystem?

Not OP but things got way to complicated and not much better IMO. Usability has gone down and the number of ecosystems and methodologies has grown and continues to grow at a pace that makes few people want keep up with the latest trend. And things are only accelerating now with wasm. Featurewise the web technology has grown enormously though and one can live inside the browser and get all their needs met but I wonder if this is the direction we really want to take. Living inside the browser is also more distraction prone, not very compatible with the ones who want to do deep work

I started in the early 90s with CGI, I think modern day stuff is really good despite all the complaints. I often think people look back with rose tinted glasses, or remember specific things that were really good while forgetting the general trend of everything at the time. We ( or at least I ) were a lot more forgiving as the new new things enabled stuff we never had before, now expectations are a lot higher.

The only real objective thing I can say about it is that much of the modern ecosystem can have a pretty steep learning curve. Frontend alone, React is quite hard to grasp for a beginner. I'm a Svelte fan and I think it's a lot easier than React, but even then it's no competition to just plain old HTML files and some light JS.

On the backend, now you have this fragmentation in Node, Python, PHP, Ruby, Go, Elixir, etc. In the Matt's Script Archive days, CGI scripts were pretty much only Perl. There were other languages used by some, but they were rare. If you wanted to learn Perl CGI scripts, you pretty much downloaded MSA as well as tons of other free scripts and read them, tweak them, play with them and figured things out. (Not unlike how you simply View Source on frontend to learn HTML and JS)

Bigger sites would use c/c++ and actual shell scripts weren't unheard of either, but that was presuming you actually had an full unix account, which with the arrival of mod_PHP usually werent the case.

Todays deployment pipeline however have gone completely off track with all of the "large team, big cluster" overhead enforced on even small agile part time projects that don't need even 1 full server.

In the early 2000s, Sun's patching support website, sunsolve, was a massive Apache/CGI empire. Perf. problems were rife and by early 2002 the whole thing was grinding to a halt under traffic.

It just happened to be intern season. In walks some fresh-faced kid and installs mod_perl. Like magic, the entire website stands up, dusts itself off and starts working again. CPU load drops by two thirds. I think the senior engineers were kind of pissed but couldn't really say anything. I think the kid got a beer or two out of the whole thing. (And of course got an offer to stay.)

My first code for CGI was all done in C. When I found Perl, there was no looking back, never used C for CGI again.

While I haven't written any significant Perl in a long time, I will always appreciate how much it improved things for me back then.

Same here. cgic was a useful library for C, but Perl was a revelation.

Same. Wrote a complete database-driven web application in C, an online bookstore. Switching to Perl 4 was a vast improvement in terms of development productivity.

Almost same here, I went with Tcl instead.

C-based CGI programs that implemented a web app for doing drop shipment of phones to customers. It was supposed to be used for just a few months while they waited to roll out a more official solution. The users hated the official one, and kept using my version for another 4 years.

After that, it was Java all the way.

Just thinking about C's story for strings, and trying to use that for webpage rendering/input handling/validation... [shudder]

Same. My first C cgi program was a small Tic Tac Toe game that kept the current state in the request string. It had a little logic to play against the user. It did not, however, support number of players 0.

Mine played Othello using the same mechanism. It was cool because every square that was a valid move would change the cursor when hovered over. The CGI would make the players move, plan the computers counter move, return the board showing the players move with an automatic reload after X seconds to the URL encoding the game state plus the pre-planned machine move. Used a html table ang gifs. So fun.

CGI was a wonderful paradigm for when I was first learning web development, a few years after this. A shell account somewhere, some scripts that just had to deal with stdin and stdout, and you could create amazing web experiences. Previously I only knew BASIC, but CGI let me start to share my weird little creations with the world.

There should be more simple ways for young people to write and share programs with each other. Maybe that’s what Roblox is, to an extent? I haven’t used it myself.

Am I the only one here who remembers Philip and Alex's Guide to Web Publishing? https://philip.greenspun.com/panda/

OK, looks like my memory isn't that good. panda came out in 1998. At the time I was working for a company that paid Y2K bonuses to dissuade us from quitting and moving to the Bay Area. I often wonder how different my life would be if I had done that.

Who am I kidding? I was having way too much fun writing motion control software to do any of that Web crap ;-)

No, you're not. That site and Greenspun's book about database-backed websites were a big inspiration to me.

I used Lincoln Stein's CGI.pm module (https://metacpan.org/pod/distribution/CGI/lib/CGI.pod#CGI.pm...) all the time back then. I wrote a Perl module for SMIL which I called SMIL.pm and emailed him to tell him and was thrilled when he responded. I'm strangely saddened to hear it has been removed from Perl.

It’s still available[1] (and maintained) for those that need it, but there are much better tools available for web work these days, like Mojolicious and Dancer.

[1] https://metacpan.org/release/CGI

You must be a latecomer to the scene... "back then", people used cgi-lib.pl:

https://cgi-lib.berkeley.edu/

CGI is in my list of what I consider "core Web technologies", and still include and use in my projects.

I haven't written it up yet, but basically it's things which are supported by almost every browser and/or almost every web server.

Other things on the list are parts of HTML 3.2, like <p> tags, the standard format of access.log, HTTP/1.1, etc.

I try to not use anything outside of that list in my projects, some JS things being an exception (which I feature-check before using)

When I got to college and got online in 1995, I pretty quickly learned about CGI scripts and the cgi-bin directory, which of course my school did not have enabled on campus-wide accounts (security risks). I really wanted cool stuff like a web hit counter on my page. A lot of web hosts at the time would advertise something like "5MB of space and your own cgi-bin!" and I eventually got to one of these, got into Perl, and start making stuff using Berkeley DB and other on-disk databases. Eventually I outgrew this, found PHP & MySQL, found a web host who supported it, went to work at that web host, ... All said, CGI's were totally the reason I got into internet programming.

FCGI is still quite useful. You can use FCGI, Apache, and Go to get quite good performance. FGCI will spin up more Go servers as needed, and the Go servers can process multiple transactions without reloading the program. Probably outperforms node.js.

Basic CGI reloads the worker program for each request, which is secure but slow. FCGI is an orchestration system. Like Kubernetes, but with lower labor costs. If one of the workers crashes, a new one will be started.

You can run stuff like this on US$10/month shared hosting accounts. To scale up, put a load balancer in front and a replicated database on the back.

> FCGI is an orchestration system. Like Kubernetes, but with lower labor costs

It does about 0.00001% of what Kubernetes can do.

Launching processes to handle web requests is pretty basic for any web server.

Why would you do this instead of just proxy directly to a Go server with sufficient GOMAXPROCS?

While CGI scripts written in perl and shell script were all the rage, I remember using C (and ODBC) quite fondly and successfully. It wasn't until I encountered the webscr.cgi at PayPal that was more than a GB compiled and spanned millions of lines of C language that I realized the error of my ways.

Yikes, surely you exaggerate the size of a C exe. I guess if enough text was baked into the binary?

No, that definitely happened. I wasn't there, but I'm pretty sure I've seen discussions from multiple companies about how they managed when their C based CGI/FCGI/HTTP+logic executable got too big to fit in 32-bit memory space; for some, amd64 came in time, for others, they had to split things out, or do terrible stuff with PAE. Adjusting the user/kernel address split was also common, IIRC.

CGI still work great if you know your constraints and for whom you're building stuff. It is underrated alongside its cousins FCGI and SCGI. A ton of little tools work great as CGIs.

Yes, I have many scripts that I throw an echo "Content-Type: text/html\n\n" or text/plain at the top and serve from apache. Job done, move on.

is there a great docu for fcgi? Tried to learn it.

Did you read the spec https://fast-cgi.github.io? Was that not helpful?

Shameless plug: I wrote this small CGI script in bash to serve stuff from my home server with auto-expiring links: https://github.com/tzahola/share-link

What about security? Is there a safe enough way to still use it today ?

Sure it can be used safely. Appropriately sanitize inputs, don't pass things through the shell, pretty typical stuff.

also, taint checking built into the language to enforce sanitization: https://en.wikipedia.org/wiki/Taint_checking

This feature is one of those things that proves that the technology industry is bullshit.

There's an entire industry, probably worth hundreds of billions of dollars, built around trying to detect insecure code and force it to be less insecure. Meanwhile, this one old language that nobody uses has a single option you enable to be [more] secure by default. And I'll bet you a billion dollars the reason other languages don't have it is "the design doesn't look very clean".

If anything no input interpretation should happen, everything should just go directly to the program, and the program should not be capable of file operations. stdin/stdout only.

Why wouldn’t it be safe? Many scripts I have are only available internally so don’t need to worry about people hacking them (unless they want to risk being fired to look at say syslog messages without having to ssh into the box).

Perl -T helps ensure there’s no tainting of variables - you take the parameter that’s passed and regex pull out the matching pattern you need to use. If your variable is a match of ([\d]+) then it’s only going to have numbers in, and you can use it fairly safely

yes, in fact the security model of Apache mod_cgi(especially with mod_userexec) will isolate scripts to an single user, which is also the level most modern reverse proxy based setups offer.

The problem was that the newbies writing cgi scripts in 1993 generally did not have much experience with input sanitation and had very few libraries to lean on to do it for them.

It wasn't easy, but everything was so much simpler back then.

Modern Web Development are needlessly complicated. Both Front End and Back End. Oh... and deployment.

Yep. I love Go, but I am certain it would not be half as popular as it is were it not for those simple statically linked executables that can be deployed anywhere quickly.

I had no idea what I was doing back then - copying scripts from MSA, no idea what chmod or ftp binary mode or even what the unix system I was telnetted into was (couple of years before). Managed to get things working though.

No mention of server-side includes, which were great -- you could include a nav bar on all your pages and just have one file to update, you could include a 'last updated' line at the bottom based on the file modification, you could include a visitor counter.

I did a website for a local art+book shop back in 1997, it was awful. Stopped updating it after 6 months, but it was kept online for some reason - even after the shop closed. archive.org shows the SSI counter was still incrementing in 2005. I assume the (copy-and-paste) code ran a script that opened a file, incremented the number, wrote it back to the file, then printed the number.

The non-static portions of the Tarsnap website still consists of CGI scripts written in C.

Sure, forking a process per request isn't fast... but I'd love to have enough traffic for that to matter.

There seems to be a general view that anything you write today has to scale to a million concurrent users.

There’s literally 20 people that have client certificates that can access my page. I don’t care if they fork a process each.

Well, AWS kinda has made it so that almost every millisecond of CPU time (or other resources) is considered billable. While in the olden days you could run some scripts in some unused corner of a server for "free", with clouds that is less likely.

Services like Lambda are basically single process per request. They just pre-fork instances under load, similar to Apache. Because of the not insignificant instrumentation and other overhead in cloud architectures (e.g. Lambda uses VMs for isolation, Cloudflare uses WASM), a small, natively compiled CGI app probably uses less CPU and I/O than these modern solutions as starting a new process in Linux from an already cached copy of the executable doesn't really have much overhead in comparison to the layers of software involved in some of the aforementioned options, despite their other optimizations.

A lightsail server can cope with 500 or so people that need to use my server every day for $10 a month. It’s not worth thinking about the price, just throw the script up and job done. Even if you could spend a couple of hours making it twice as performant it wouldn’t matter.

"CGI is still a useful web programming paradigm" is a hill I am prepared to die on.

I still have cgi scripts running, I assume, somewhere.

Sometimes you just need a single file that does a few things and mostly behaves like a webpage. It doesn't need to be deployed or managed or anything. I had this one that served approximately seven requests on average for over a decade. It's simple. It doesn't need any love. It's in the documentation for the overall site because I wrote the documentation, but it isn't "omg this needs an APP and a BACK END and we have to worry about SCALING."

I still have a book somewhere on CGI with Perl and C, from the 1990s. Probably due for a purge.

My go-to for that now is python -m http.server, with nginx if i need anything more complicated.

I may have to look into it since python is deprecating the cgi module, to my great annoyance.

Or maybe not. I seem to have gotten out of the web business at the right time. It's gone in a direction that makes me grind my teeth.

One of the best aspects of CGI is how easily tested a CGI program is. All you have to do is set a couple environment variables and connect your request entity to stdin, expecting things on stdout. What could be better?

The best FAAS framework

I remember creating a web query form for an employee search or something like that using CA Clipper and reading stdin an spitting our HTML on stdout. Worked like a charm, was fast and made that data available to whoever needed it.

Was also easy to "verify" contents of /etc/passwd and /etc/shadow. cough cgi-bin/phf cough

That's not the fault of CGI but rather of the people who wrote bad code. You can easily have the same problem in your NodeJS express app if you're careless!

Perl had an interesting invention - taint mode (although much later). All inputs from the network were marked as tainted and you got an error if you tried to use I/O with them. You had to untaint the data by passing to if/case statement or at least a regex.

IMO when programmer did not fight it, it was very good for sanitizing the input. Curious why this did not get more widely adopted?

I have recently been using something very similar to CGI scripts on my Jetforce Gemini server. It actually goes in cgi-bin although its not exactly CGI. I used it to make an interface to Zork.

You can try it out if you have a Gemini client: gemini://zork.club

For these types of things I recommend the diohsc client https://repo.or.cz/diohsc.git although you may need to build from source since he had an issue with Jetforce compatibility and not sure if it's fixed in cabal yet.

But any Gemini client will work such as av98 or Lagrange.

The guy who made diohsc set up something a little more sophisticated using SCGI and he has several interactive fiction on his Gemini server (if it's up). (You can search on gus.guru for gemrepl).

I recall writing CGI scripts in perl. I met some guys are work that used C and I was shocked. But not as shocked as my boss. It took them so long to deliver applications compared to cranking out perl.

Sure, that's also how PHP started. Thing is, with Perl and PHP, before mod_perl and mod_php, your script needed to be parsed for every single uncached request, easily dominating everything else, like URL parsing and process creation time. Hence natively compiled CGIs weren't such a bad idea. They're even used today a lot, with Apache/CGI uptimes measured in decades since neither Apache nor CGI binary updates require planned downtime on Unix. Its ultra-robustness is one of the reasons the concept of FaaS has become fashionable again.

Random question but I've often heard people refer to computer graphics in movies and television as "CGI" instead of "CG." I've always wondered: is this just a weird conflation of terminology from an era when Common Gateway Interface was a thing, around the same time fancy computer graphics technology was starting to take off in the 90s? Or is there there an alternative, legitimate reason why one would refer to Computer Graph*I"cs as "CGI?"

It means Computer-generated imagery.

Oh neat, I had no idea!

It's for Computer Generated Imagery, not Computer Graphics. And it was used in the 80s already, before the web-CGI.

Ah, those were the days... few people really cared about web security back then and Perl allowed you to get really creative :). https://seclists.org/bugtraq/1997/Jun/67 or my favorite https://insecure.org/sploits/glimps.http.badchars.html .

I started learning HTML back in early high school (~96). Very fond memories of doing our warm-up laps in gym class with a friend (who is my part-time LLC business partner to this day), and we tried to remember whether it was the 'a' or 'href' part that came first. We heard rumors that there were web developers who would make $100/hr programming. The combination of those dollar signs and the magic of being able to write something that would manipulate the screen (the latter being much more motivating, to be honest) is what kept me up until 3 or 4am many nights/mornings, only to wake up at 6am to hang out with my dad before he left for work. This resulted at least one time in in me falling asleep in my high school health class, waking up entirely alone after the period ended. Oops.

I felt so many times that I was _so close_ to being able to understand how to do this new, exciting CGI thing. Something about the form action? What should that be -- the filename I want to write? Do I put the counter + 1 value there? Oh, it's just GET or POST. Oops.

I bought a book, CGI How-TO, one day at Media Play[0] with some friends while on a brief break from our LAN party setup. (Coaxial networking was a huge PITA - as soon as someone came or left, our in-progress game was terminated.) It discussed C and Perl. That book sits on the bookshelf four feet away from me right now for the nostalgia and the love of learning. A few years later, I asked my parents for Learning Python Programming for a Christmas gift. I flipped through it trying to grok what a tuple was. That thick book sat on my shelf for a lot longer than it should have, but the joy of having such a tome of possibilities was unlike most others.

[0] https://en.wikipedia.org/wiki/Media_Play

The linksys WiFi router I was using until last year used cgi app for the admin interface.

My first web application was developed in 1999 in IISAPI (VC++) running obviously on IIS. I had to write a rudimentary session storage (in plain txt file!), because it was not provided by the framework.

We do a lot of assessments of embedded devices and a surprising amount of them still have compiled C programs and CGI for their web interfaces.

Made a lot of use of this site: https://www.scriptarchive.com/

I do have fond memories of the simplicity of CGI, but also remember the big speedup and scalability improvement when we swapped over to NSAPI (no more forking, pooled processes, etc).

Kind of interesting that "serverless" (AWS Lambda, etc) seems to be going through the same evolution now. That is, initially liking the simplicity then hitting a wall with performance and making it more complicated.

I still think Lambda was just CGI tarted up and sold as a whole new thing.

There was a transition period where people were trying to still use CGI but make it scale so you had things like CGI mod_perl in Apache.

Ah, yeah...mod_perl did make a nice bridge for Perl stuff. I had mentioned NSAPI because we had C++ CGI and NSAPI was available before things like FastCGI.

Been thinking this recently, lambda is a return to CGI, you provide a small program that runs and provides its results to something which sends them back to the client.

And you similarly worry about things like cold starts :)

Unless your lambda was written in Go.

Maybe. ~300ms floor isn't terrific for some use cases, and could be higher if you need an ENI, etc.

FastCGI was the standardized approach.

See also: https://news.ycombinator.com/item?id=24683304 ("FastCGI – The Forgotten Treasure (2002)")

Sure. Wikipedia mentions FastCGI initially being a reaction to NSAPI.

"Open Market originally developed FastCGI in part as a competitive response to Netscape's proprietary, in-process application programming interfaces (APIs) (Netscape Server Application Programming Interface (NSAPI)) for developing Web applications"

Oh, I'm showing my ignorance here - I didn't know NSAPI was in-process. The Netscape server software was expensive and kinda hard to find in Europe back then. I guess FastCGI was an improvement in terms of stability - I'm assuming the NSAPI was .so/.dll-based? So if you messed up, the entire server went down.

Back in the mid 90s I worked on the Spinner/Roxen webserver that was built on an interpreted C-like language modelled after the LPC language being used for MUDs. Modules were were run in-process, but because of the interpretation a failure resulted in an exception/error, rather than a full crash. This software didn't get very popular - we sucked at marketing.

Most people kept the in-process stuff down to something that worked like Fast-CGI, just proxying requests to some non-NSAPI persistent process over a socket.

Somehow this ended up at Oracle? That's just nasty.

https://docs.oracle.com/cd/E19146-01/821-1833/index.html

Yes. Sun/Netscape co-owned the iPlanet brand and products, Oracle bought Sun. AOL still has iPlanet rights though, so it's actually in two places.

Tricky to follow where all the stuff ended up: https://en.wikipedia.org/wiki/IPlanet

I loved spinner! It was one of my favorite web servers, even if I never did learn enough pike to do anything useful. But the graphical web server configuration interface was amazing for its time.

> In many ways then, it was CGI — not JavaScript — that was the start of web applications.

"Before we had these nifty LED bulbs, we burned candles."

I remember when Tripod offered free Perl scripting, I played around with CGI.pm and made what might have been the most terrible forum script ever made - luckily no one else ever used it.

Also I think I tried to get the forum from Matt's Script Archive (which is still around by the way) working but couldn't.

CGI is the future of lambdas.

Good read, thanks for sharing. It’s quite remarkable how we still use the same concepts today almost 30 years after it was invented. Modern web apps are ‘just’ text boxes over databases with logic in between to serialise it over the network in a request / response driven manner (using string-based key/value pairs). Today there’s a stack of JavaScript on top of it but it basically still boils down to the same principles.

Those were interesting times. I think I prefer having a lot of things be plug and play now.

I remember writing cgi scripts in quickbasic! What a fun time that was to be alive.

I got my start doing professional programming working on Perl CGI code.

I remember SSI. I haven't seen much of that albatross in ages.

Good riddance.

I did write an entire CMS in PERL, once.

I still wake up screaming...

Hmm, I don't have any negative associations with SSI, seems like similar ideas have existed in lots of more recent platforms (like partials in Rails, e.g.). Static site generators are doing something similar, just in advance?

The problem with SSI, was that it was half-baked. It didn’t have enough power to be truly useful. PHP (and browser JS) ended up obviating it.

Static site generators don’t generate “live,” like SSI did.

Actually writing CGI scripts was a bit before my time, but I have very fond memories of installing and using Newspro and Imagefolio on my old web pages.

Reply on news.ycombinator.com