Hacker News Reader

> I still mention maintaining this site on my CV and LinkedIn - disappointingly I've never been asked about it in an interview. I suspect most of the people doing the interviewing these days are too young to remember it.

This is astonishing to me. I check back to see if this site is still up once every year or two just to have a smile. If you were sitting across from me in an interview I am quite sure I'd lose all pretense of professionalism and ask you about nothing else for the hour.

>This is astonishing to me. I check back to see if this site is still up once every year or two just to have a smile.

The only reason I am sad about the death of flash is that it has all but killed my version of this: zombo.com.

wow, what a way to learn that zombo is dead. RIP zombocom

It isn't quite dead yet. It is just entirely Flash based so it likely won't make it to 2021. Safari has already blocked Flash completely while the other major browsers require you to manually enable it. Odds are those too will begin blocking Flash completely sometime in the next few months with Adobe set to end official support at the end of the year. So maybe add zombo.com to your Chrome whitelist and visit it one more time, because soon the infinite won't be possible.

https://html5zombo.com/ might assuage your grief a bit.

Thanks. This like that old trope of a child's pet fish dying and the parents buy a nearly identical fish to swap it in and pretend nothing happened.

"We were forbidden from running scans again by management"

Seems like management believes it is better to wait for real bad actor to purposefully destroy your site than have it done by your honest employees by accident.

I know security in 2001 was much more lax (I started in 2000) but this still shows ignorance of management.

The right way to handle this would be to ask your staff to ensure it is possible to restore services and to ensure you know what the tests are doing before you run them.

>Seems like management believes it is better to wait for real bad actor to purposefully destroy your site than have it done by your honest employees by accident.

From a politics standpoint that is completely true. Which would you rather tell your boss:

Q: Why is the website offline?

A: One of our sys admins accidentally deleted it.

OR

Q: Why is the website offline?

A: Some nation-state/teenager launched a sophisticated cyber attack, we need to increase the cybersecurity budget. It's the wildwest out there!

There is this saying that only a person that does absolutely nothing never makes any mistakes.

Mistakes are normal course of action at a corporation. Sane managers will understand that is not possible to not have people make any mistakes. Mistakes are part of the learning process.

Now, when somebody makes a mistake what I am looking for are:

Does this person show good judgment? Were precautions taken by the person reasonable?

Does the mistake show pattern of abnormality? Some people seem to attract failure, maybe there is some underlying cause?

Is the person learning from mistakes? Learning is expensive, if somebody made an expensive mistake I want as much learning as possible for the expense.

Is there some kind of external factor that made the mistake possible or more likely? Usually it is possible to improve the environment to reduce the number of mistakes.

As to preventing these guys from scanning ever again, that is bad decision because it is likely they would never make the same mistake again. What's done is done. The scan showed there are problems with the app, now we should want to know if there are more problems but without risking the application stability (too much).

---

-- Do you know what the Big Co. pays when they pay high salary for an experienced engineer?

-- They pay for all the mistakes he/she made at her previous place.

We have stopped all payroll payouts because one accountant mistyped a digit last month

Was the solution to bar accountants from typing?

I don't really agree with that decision, but maybe that phrasing is not 100% accurate. Maybe they just meant that they should run the scan on a local test environment and not in the production deployment. Obviously, there's value in running it against live, but at least for this particular issue, they could probably had caught it in a testing environment.

You are probably thinking in today's terms. Back then it was unlikely there was separate test environment. For application... probably, but not for what is a static website and HTTP server.

They may not have had a fully fledged staging environment which is an exact copy of production. But I'm sure they ran the http server locally in their desktop to test new configuration. I was a high school hobbyist "developer" (maybe a power user would be more appropriate) at the time and I certainly tested my pages locally before I'd upload them to Geocities. 2000 is not the bronze age :P. People were already talking about TDD then and the concept of testing things before shipping existed for much longer.

I think you're looking back at this from the modern perspective of cheap hardware and open source software. We didn't have Sun workstations as desktops - we were using FreeBSD on cheap Dells. Our production servers cost something like $200k each. And the http server cost money - we didn't have licenses to run copies locally. These days you can just run things in Docker and have a reproducible environment but it wasn't that easy back then.

I have been sysadmin at that time.

Usually, HTTP servers would be huge bloats of configuration running on huge machines. As a sysadmin, it was not typically feasible to replicate the configuration on the local machine.

With a static website nobody would figure you can accidentally actually do something harmful to the website. The worst thing that could happen was that you made a configuration error in which case you just rolled back the config.

Of course we were supposed to have backups in case the server failed. While the servers would typically use RAID arrays we knew it would not save against rm -fr so there was a backup but backups rarely could be restored instantly.

Interesting. I never thought it was so complicated to run the http server. What was the bottleneck that prevented them from running locally? The memory used by all the config?

The amount of hardcoded stuff that was specific to the machine. All the domain addresses, firewall rules, etc. There was no git or ci/CD to keep everything in sync (easily).

Server side JavaScript? That’ll never catch.

I love these examples of historical calls made too early...

But node js is alive now.

He was being sarcastic :)

Whooooosh

Read past sentence 1...

Having done it NES-style, I'm kind of glad that server-side JS didn't catch.

Node is an unlikely event. It hit the sweet spot precisely when async started being needed, there was a runtime environment that made it viable (v8, thanks to massive investment by Google on Chrome), and a language that made it kind of natural.

is there consensus on whether Node did or did not "catch"? like, sure, it was a launch language of AWS Lambda and Netflix uses it heavily. but every backend person I know uses Go or Python or Ruby, anything other than JS. pretty much only fullstack JS people consistently pick nodejs as first choice - but that's just my POV. I'd love some more definitive numbers.

Its components are part of every developer's toolbox (a lot of really essential command line tools use to be installed with npm). I don't see many backends powered by it, but I'm mostly a Python developer and that certainly biases my samples.

Node as a tool definitely did catch on: automated testing, package management, REPL integration, bundling etc.

Node as a server/backend language: I would say the say the ecosystem and usage are at least as large as Go/Ruby. This is hard to gauge. But I assume/expect a factor of 2 or more. If you look for web specific libraries it is unlikely you'd find something for either Go/Ruby but not for Node. Python is harder to compare because it is used much more broadly.

For me the biggest use-cases for Node are: server-side/static rendering (get all the power/expression of frontend libraries on your http/build server), web-socket ease of use and the fact that there are tons of programmers who are at least familiar with the language.

And even though it is steadily declining, the most popular web-backend language is still PHP by a long shot. And this won't change until the other languages get managed hosting that is as cheap and simple and similarly ubiquitous.

It's absolutely catching on with younger developers who don't want to bother using multiple languages. Anecdotally, like 100% of developers in their 20s at my company push hard for JS.

Paypal, Linkedin, Netflix, Uber

Haha

But I used a few servers that allowed for JS scripting way before Node.

I guess, most of them were proprietary, so it never caught on until Node.

At my first job out of college in 2008 we wrote JS for a desktop app on top of Mozilla's xulrunner platform. At the time I hadn't heard of using JS outside the browser and Node was still a few years off. It was a great experience but Xulrunner got killed by Mozilla and the company had to rewrite in C#.

Just an example of (sort of) non-proprietary, non-browser JS.

It used to be super slow. Like, you cannot imagine how slow it was.

The usual example I trot out is when I was writing a client-side pivot table creator in the mid-2000s, as far as I can remember, with just 100 items the js version took 20-30 seconds. I then tried it using XML/XSLT instead[1] and it was instant.

I haven't checked recently, but even a few years ago javascript is extremely slow with large datasets, I was mucking around with Mandelbrot generators and js compared to C# was like a bicycle vs a jet engine, they weren't even vaguely in the same league performance-wise. Just had a quick look at some js based versions looks like it's got a bit faster, but still slow.

[1] Awesome performance, super hard for others to understand the code. XSLT was great in some ways, but the learning curve was high.

In principle, Javascript on modern JIT can approach roughly 40-80% of the performance of C++, more or less across the board, not just in some toy benchmark. This benchmark is from 2014:

https://blogs.unity3d.com/2014/10/07/benchmarking-unity-perf...

Granted, this benchmark is for generated Javascript, not "idiomatic Javascript" (if there is such a thing). Of course you can write arbitrarily slow Javascript as well. There's a lot of stuff that can throw a wrench into the JIT.

That seems an optimistic reading of those graphs. As far as I can tell. a lot of them are 20-30% the speed of native, not 40-60%.

For the Firefox graph, none of them are below 35%, only three are below 40%. The average is well above 60%.

That said, there's a lot going on there. Javascript can not use SIMD or multiple threads. Some tests are heavier on C#, which is converted to C++ in the Javascript version and thus becomes "faster than native C#".

The point is not give an exact number on how fast Javascript is compared to any other language, there's wide variance across usage and implementations. The point is to show that Javascript to rival C# performance is feasible.

>> Server side JavaScript? That’ll never catch.

Server side javascript didn't catch on the first time around because you couldn't create threads. Because in the early 2000's, all technical interviews asked about Java style multi-threading. At some companies, the first round of technical interviews was a very, very detailed discussion about the Java threading model. If you didn't pass, you didn't get to round two. So everybody wanted to use threads.

I did my own "JavaScript Server Pages" using Rhino (JS for Java), which does compilation of JS to java bytecode. It worked great for me. I did two sites using it, then changed job.

Made my day too! Wow.

I like how it loads instanly! :-) I remember staring at this thing for about 2 mins watching it load over dial up in a third world country on an laptop dad somehow smuggled into the country from dubai without paying import duties. Yeah.

97 on mobile and 99 on desktop. [0]

For a site that never had pagespeed as a tool, that is super impressive and just goes to show how bloated today's websites really are.

[0] https://developers.google.com/speed/pagespeed/insights/?url=...

I like to think you can still make a fast site by sticking to the basics; write plain static HTML and CSS, avoid JS, craft your images specifically for the site, and put it behind a decent webserver with compression enabled. I'm sure Pagespeed rewards HTTPS and HTTP/2 support as well but that's webserver config.

Or make it even faster by not using CSS at all! (spacejam.com predates CSS1)

Even gzipped jquery is already larger than the entire payload of the homepage, and jquery is already considered an old technology.

Funny you mentioned matrix.com; back between Matrix 1 and Matrix Reloaded I knew someone who knew the guy who owned thematrix.com and apparently was not offered enough to sell it. I guess they must have agreed at some point because I also recall that the matrix.com at around the time of Reloaded then started to redirect to the canonical Matrix site (which it sort of does now). Wonder what the price turned out to be.

> matrix.com at around the time of Reloaded then started to redirect to the canonical Matrix site (which it sort of does now)

Actually matrix.com now appears to be a site relating to hairstyling and haircare products.

I think they meant thematrix.com which forwards to a WB site; maybe autocorrect added the space.

Yes, that is the case. Autocorrect strikes again.

:) This made my day. Were you based in LA at the time? I’m out in LA now and love hearing stories like this from colleagues who’ve done engineering work in the entertainment industry.

Yes, Burbank.

Here's another one - Solaris had a 2GB file size limit (this was before ZFS). Which isn't as crazy as it sounds now - hard drives were 9GB at the time. So ordinarily this wasn't a problem, but when the first Harry Potter movie came out, harrypotter.com (which was being served off the same server as spacejam.com) was the most popular website on the internet and the web server log would hit the 2GB limit every couple hours and we would have to frantically move it somewhere and restart the process.

And somewhere was /dev/null ?

Most likely the main Oracle server which had a disk array attached. These days it's easy to forget how tricky it could be to get basic things like this working. rsync was in its infancy and I doubt we were using it. As I recall most servers didn't have ssh installed; telnet was standard. Lots of tar piped over rsh to get files from A to B.

7-8 years ago I've encountered a full-on rsh internal network in a fairly big datacenter. No rsync, rcp. It turned out that the sysadmins were too lazy to set up ssh keys on all the servers, and their manager was skilled at deflecting issues.

Hello, I run a monthly magazine about Harry Potter, could I interview you about this? Any email address where I could write to you? Thanks!

Sure, username at termcap.net

Posts like this make HN amazing. Thanks.

> Another fun fact about NES - they were really pushing server side Javascript as the development language for the web

I started “web stuff” with Apache and Perl CGI, and I knew NES existed but never used or saw it myself. I had no idea “server side JavaScript” was a thing back then. That’s hilarious.

I entered around the time LAMP took over and the old-timers would always trash on server-side javascript.

> the old-timers would always trash on server-side javascript.

Plus ça change, plus c'est la même chose.

I submitted this and thought, "this is neat." Didn't expect to return to it being on the front-page nor the former webmaster to show up. The internet is a fun place.

That background story is fascinating. I wonder how many full-circles server side JavaScript has made up until now.

> the person who had just written a book on server side js for O'Reilly

I'd love to see this unpublished book, if possible!

It will be great!

Unfortunately I can't even remember the author's last name and LinkedIn isn't helping. Let me ask around.

It would sell now like hot cakes. An interesting artefact

OReilly's Javascript books from circa 2001: https://www.oreilly.com/search/?query=server%20side%20javasc...

Plus a few more from 2001:

https://www.amazon.com/Server-Side-JavaScript-Developing-Int...

https://www.amazon.com/Server-Side-Javascript-Developing-Int...

Netscape's 1998 Server Side Javascript guide: https://docs.oracle.com/cd/E19957-01/816-6411-10/816-6411-10...

> Fun fact, unlike Apache, NES enabled the HTTP DELETE method out of the box and it had to be disabled in your config. We found that out the hard way when one of the sysadmins ran a vulnerability scanner which deleted all the websites. We were forbidden from running scans again by management.

Oh man, the early days were so exciting. Like that time I told my boss not to use Alexa on our admin page out of general paranoia... and a few days later a bunch of content got deleted from our mainpage because they spidered a bunch of [delete] links. I learned my lesson; secured the admin site a lil better and upgraded to delete buttons. Boss kept on using Alexa on the admin page tho.

> they were really pushing server side Javascript as the development language for the web (and mostly losing to mod_perl).

Enterprise server-side JavaScript was the first stage of dynamic web servers that connected to relational databases. Netscape LiveWire, Sybase PowerDynamo, and Microsoft Active Server Pages (with interpreted LiveScript or VBScript) were early. The enterprise software industry switched to three tiered app server architectures with JVM/.net bytecode runtimes. Single-process, multi-threaded app/web servers were a novelty and none of the client drivers for the popular relational databases were thread safe initially.

It took some time for RESTful architectures to shake-out.

Apache (and mod_perl) was thread safe by being multi-process, single threaded. You were always limited by how many perl interpreters could fit in RAM simultaneously. Then came the era of the Java app server (Weblogic and WebSphere).

Everyone has mostly forgotten AOLserver (and TCL!). Almost everyone of that generation was influenced to use a RDBMS as a backend by Philip and Alex's Guide to Web Publishing which came out in 1998.[0] Although I never actually met anyone who used AOLserver itself! Everyone took the idea and implemented it in ASP or perl, until the world succumbed to EJBs.

[0] https://philip.greenspun.com/panda/

AOLserver was a joy. I was able to speak about it at the O'Reilly OSCON in 2000 or so.

I used AOLserver with OpenACS, https://openacs.org/ . AOLserver was apparently more optimized than Apache (at the time, at least).

My manager even had us (early 2000s) take a one-week(?) bootcamp at Ars Digita HQ in Boston. Though about the only thing I remember from it was the fancy aeron chairs.

That was also the era when DBMS locking was iffy, so competing inserts/updates would happen once sites got too busy, and everyone was encouraged to turn off transactions (or, on MySQL, just use MyISAM instead of InnoDB) and put it on XFS or even tempfs for speed.

So many websites I remember came back up with "the last two weeks of posts are gone, sorry" or just shut down for good because it was all too corrupted to fix, and so were the backups, if they had any.

> We were forbidden from running scans again by management.

A scan detects a severe vulnerability and their reaction is to never run scans again...

Well, the issue seems to be the scan not only detected but realized the risk from the vulnerability, which is exactly what is the point of scans to help you avoid.

It's hard to detect what HTTP DELETE does without, well...

did they turn the scan loose on PROD without scanning a dev first? granted, I wouldn't blame anyone for assuming a scan is read-only

A what first?

What dev?

Real software engineers ssh directly into PROD to write code.

It was a different era. A lot of best practices we take for granted now as being common sense were stuff we (and I say this as myself also being an old time sysadmin) had to first learn...and often the hard way.

Plus the web wasn't as important business stratergy back then as it obviously is now. I doubt Warner Brothers would have been willing to invest in replica dev infrastructure when "developers can write code on their desktops". I know dev infrastructure is for more than just developing code, but common concepts we take for granted like IaC, CI/CD, config management etc wasn't formalised or widely used back then and servers were pets that were held together with duct tape and sacred rituals.

In many ways, that's what made being a sysadmin in that era fun. There wasn't any shame in hacking together a solution.

"In many ways, that's what made being a sysadmin in that era fun. There wasn't any shame in hacking together a solution."

hah true, and management was absolutely amazed! In the late 90s/early 2000s i worked for an independent pharmacy chain. I wrote what was basically just a proxy sitting between our dispensing systems and the central clearing networks for rx drug pricing. All it did was double check the price on the prescription (our stores weren't applying price updates which was a manual process at the time) and reject prescriptions that were priced wrong with a message telling the pharmacist to apply their price update. The CEO invented an annual award to give to me for the "work" hah

Back in 2000, it was the wildwest. You actually did run tests on production and editing production code was not as insane of an idea as it is today.

I don't think anyone (at least no one I worked for at the time) ran staging or dev servers.

It was always stupid, yes. But back then we didn't have the tools and testing suites that we have today. CI/CD setups didn't exist. Git wasn't even built until 2005. The only version control solution was SVN at the time, which was released in 2001. But it was clunky and immature.

Back in 2000, launching a site update meant someone would log into the server via FTP, drag the files over, and try to "be careful" while they did it. Using passwords that were written on a sticky note, stuck to the CRT monitor's screen (password managers weren't a thing).

It is easy to forget how immature and primitive the world of web development was at this time.

Also, the internet was still so new, that every C-level executive had built their careers by running businesses in the 70s, 80s, and 90s. Back before you built websites or relied on them for any significant impact on your bottom line. So to tell an executive from that era that your website broke when you poked at it, their solution would be to stop poking at it. Security wasn't really a major concern like today, and having a website was still mostly a novelty in the eyes of most executives.

> Back in 2000, launching a site update meant someone would log into the server via FTP, drag the files over, and try to "be careful" while they did it.

Heck, I remember just compiling Java classes on the server machine itself, copying them to a production directory, and restarting the app server (tomcat IIRC). Source control was the sysadmin running a nightly backup of source directories

> The only version control solution was SVN at the time

Pre SVN there was CVS, and before that was RCS. Now these didn't work the way we think about git today, but they did allow you to roll back to a known good state with some futzing about.

Website is still alive and kicking more then 20 year later, guess it worked.

/s

Bit like testing for COVID-19, if you don't test for it how can you have it?

no politics, but this is like the "if you stop testing numbers go down" lol

They might give you a call now, as the site just died from traffic.

Which is weird - this site comes up on reddits front-page often and it has never gone down. Is HN higher traffic??

Obviously not.

Wow, every paragraph has a great story!

>the person who had just written a book on server side js for O'Reilly - he got his advance but they didn't publish it because by the time he had finished it they considered it a "dead technology".

Makes sense. This server-side Javascript thing will never take off. The Internet in general is really just a passing fad!

This is a wonderful post. Thanks for your insight.

What machine is this being hosted now? Is it the same Solaris machine?

Your comment made me remember netcraft.com which was the tool we always used to see which web server and OS was running a site. There was a running joke back in the day on slashdot about "BSD is dying" as Linux started to take over, based around Netcraft which used to publish a report showing which servers were most popular. I'm glad to see they haven't changed their logo in 20 years!

Their site report used to show an entire history so you could see every time they changed servers but it doesn't look like it does anymore. Now it's running in AWS so certainly not the same Solaris server. Although those E4500s were built like tanks so it could be plugged in somewhere...

https://sitereport.netcraft.com/?url=https://www.spacejam.co...

> here was a running joke back in the day on slashdot about "BSD is dying"

Lol. I used to read Slashdot when I was in college and that time the running joke was "Had Netcraft confirmed it?", right?

> Now it's running in AWS so certainly not the same Solaris server.

Hmm. Would they have maintained the site to run on modern software stack or would they have just used legacy emulator like this: https://aws.amazon.com/blogs/apn/re-hosting-sparc-alpha-or-o...

I used this site as teaching Material for young adults the concept of A Webpage that is not an app.

Also Space jam has « style » to be compared to « wikipedia » that is « pure html text »

Thanks for sharing this -- I love stories like these

Thanks for the story! Also, I wish 'webmaster' was still a title used in the industry.

thanks for sharing mate, great to know what was going on behind the scenes during this time.

Thanks for sharing. Is this your comment in html code?

1996 site.