Majority of web apps could just run on a single server

It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It

Sysadmins usually avoided complexity because their attitude toward it was more sensible: it’s expensive, fragile, and tends to actually multiply failure modes.

I also blame cloud marketing. This stuff is a gigantic money printer for cloud companies. It’s in their interest to encourage as much over engineering as possible, especially if it locks you into things like Kubernetes that are hard to run and thus usually used as services.

If you're hiring, you want to be able to add/replace people as easily as possible. If you're being hired, you want to charge as much as you can. And to satisfy those two demands, the current web stack is great. Almost like it's been built for it.

It's got very little to do with the tech itself, a lot more to do with market dynamics. That's the problem it's trying to solve.

I'd argue it's just as much the dogmatic nature of all things software-related, together with the attendant shiny new object syndrome.

See SPAs, NoSQL, micro-services, etc. There's generally a use case for all of these, but they tend to be too easily extrapolated into, "if you're not using these, you're not doing it right."

Was a devops engineer at my previous job.

We already had k8s clusters setup, pre-made CI templates and pre-tailored helm charts (along with monitoring and much more). All those things you (a developer) could mostly clone, slightly customize and ship both to a development k8s cluster and to a prod k8s cluster (with all the safety nets already in place).

Creating (and maintaining) a single vm for a pet project is way more work than using the pre-made and pre-customized and curated toolkit.

This was at a 100+ developers organisation.

If you think you could easily get off with a single vm then you've never seen devops done right, i'm fairly sure.

EDIT: I probably fell for the bait, but the post i'm replying to really made me remember why we went on a killing spree in order to eradicate everything that was not k8s at my previous job and removing as much developer access to prod as possible. Some idiot developers think they know better, usually end up re-inventing a square wheel that breaks as soon as it's not running on their laptop anymore.

The problem is that containers are excellent, and IMO there's a gap in the market between "I want to run one container" and "I want a fully managed k8s cluster"

I’ve lost startup jobs for basically this. As in “let’s focus on our mvp instead of adopting k8s and discussing “the definition of done” for literally six hours a week”. That one ran out of money and folded, sadly. They had tons of potential and a few bad hires.

But also on the other side, my time is expensive, engineering time is expensive, downtime is catastrophic for a new business. Some people want to spend 4 5 or 6 figures to save twenty dollars a month. Imagine my time is worth $500 an hour, if the effort to make something cheaper doesn’t pay for itself in a year then it’s probably a waste of resources.

And like, internet search should be more resource heavy than the sort of websites that regularly do keel over to HN. Every query is like up to 50 MB in disk reads.

[1] https://news.ycombinator.com/item?id=28550764 [2] https://news.ycombinator.com/item?id=35611923

Kubernetes is/was a way to fight off walled gardens from cloud providers. The other path would have been to learn the bespoke implementation of each cloud provider depending on what that employer ended up using.

Kubernetes was at the right place, at the right time just as AWS was trying to force feed people their own proprietary solution, as Azure was trying to wall off people into their own walled garden, as GCP was being Google just not giving a damn about any other usecase than what works great at a massive search company.

With Kubernetes, developers can learn one API to deploy their applications and hopefully it works on AWS, Azure, GCP, DO, OVH or a laptop at home.

So that way, developers can learn one thing and transfer their knowledge at an employer that hosts on AWS, and then another that hosts on Azure and so on.

This is in contrast to the experience of a Python developer who's mastered FastAPI/Flask/SQLAlchemy and feels absolutely lost in a Django project or an Angular developer who stares a Next.js project wondering what the heck is happening and how it all works. Neither a Next.js or an Angular developer would start off with an AWS Amplify solution if they could help it.

When I worked on teams that denominated egress in terrabits/s, TPS in millions or higher (sometime much higher), and daily warehouse ingest in petabytes it was just the default thing to spin up an instance and a hot standby (sometimes per region or something) if that’s all that was required, and containers and whatnot were used only in the context of bare-metal: you do usually want one level of indirection so containers are really useful if you’re racking metal from a small menu of SKUs.

But as for why it ever became conventional wisdom to wrap a venv inside a container on a hypervisor, often with multiple images composed on a dizzying array of low-friction SKUs?

I never got it and still don’t.

Ah yes, time for the annual debate on the complexities of Kubernetes versus the unparalleled genius of custom scripts that seem to work...sometimes. Because reinventing the wheel is always superior to something with a standardized API.

And let's not forget the sheer elegance of a homegrown scripts that rivals the structured approach of Kubernetes. A true testament to intuitive design.

Sure, Kubernetes might have a few minor benefits beyond 'scaling out'. But honestly, who needs the ability to manage complex applications with any semblance of ease?

The current DevOps culture comes from the FAANG guys who really are getting a bazillion requests per second. In the last decade I worked for Amazon, Avalara and Audible Magic. None of these could build an app around a Digital Ocean droplet.

But I think you're pointing out there are PLENTY of useful webapps that can run on a minimal system.

I'm just curious where the middle ground is. Cause I think we've all seen sites blow up after a reference on HN or SlashDot or whatever and by the time you get there the only thing you see is a stock error from the PHP engine saying "My MySQL Engine is Melting" or somesuch.

It would be very cool if one could write an app using {NODE|Ruby|Python|Whatever} and have the infrastructure around it notice when things spike and do some magic under the hood to spin up new containers in geographically distributed data centers and scale up a simple persistence tier.

That way you could move forward with a SIMPLE application instance and not freak out that you'll disappoint new users if there's a spike in demand. You know, sort of like what AWS Lambda was supposed to be.

Hmm... I think I might have come up with a plan for my next startup. Thank you for speaking your truth.

[Edit: To re-iterate, I think I'm saying it's just as wrong to think a small, simple app needs Amazon-level redundancy and immediate scalability as it is to say Amazon could run on a single machine. But... the "Slash-Dotted Website Goes Down" scenario is real and there should be SOMETHING the industry could do that's easier on people than to force them into a custom AWS ECS solution across multiple continents.]

In my 1st job we had autoscaling on AWS to handle peaks… except that our servers took about 30 minutes to do the upgrades, download gcc, compile all the needed python modules… We'd always reach the autoscaling limit because the new servers weren't doing anything at all. All of them would be downloading and compiling the same things.

I was very junior, but I told my boss if we shouldn't maybe use base images that already contained everything, instead of a default blank ubuntu for the servers.

The boss said no, because that wouldn't be agile.

The whole thing of using yaml files to configure servers already existed, it worked terribly.

It was basically meme development there. Including using mongodb for no reason at all, and using it badly so that every write was actually moving thousands of records around.

The irony was that the cloud was supposed to be the simplest bit. All the computation and cryptography occurred on the mobile clients, the cloud was really just to provide storage. The same team then rewrote the mobile clients to have a "beautiful" API that took 100s of times more resource than the original code.

I guess they just loved complexity for the sake of it.

Most pages can really just serve a jekyll/astro static-generated page and be fine. But if you shove a database and php in the middle, it’s gonna be multiple orders of magnitude slower.

I was firmly told not to suggest it again unless I wanted to put all of us out of a job.

That seemed ridiculous to me — we had a laundry list of ways we could help the business if we got basic platform stuff off of our plates. But I sure learned something about how incentives affect otherwise-good engineers.

I work on the Ops side partially, and our platforms are defined by the highest level of complexity we need to support. That is to say that your basic web app can run on k8s, but that auto-scaling, messaged-based behemoth that 85% of the business flows through will not.

So I can either support k8s, or I can support k8s _and_ old-school rsync deployments to VPS'

The complexity of running a basic web app on k8s is entirely too high, but the cost of keeping an entirely separate deployment/monitoring/oncall/permissions stack for VPS' is worse. Better hope your monitoring vendor has an agent that can run in a VPS' on the OS you want, or you're back to running Nagios yourself.

If at least 1 group is going to write an app that runs on k8s, you might as well run most of the company on k8s. Otherwise you're either going to manage 2 separate stacks, and/or try to write an abstraction layer that will probably itself resemble k8s.

I’m that kind of a person and the usual response I get is - hey, that’s too complex!

VPS services are pretty cheap these days. Hetzner, digital ocean, genesis, etc. Or you spend a couple grand to build a dedicated machine in raid6 and only pay utilities.

But let's be clear, your serving infrastructure is able to be that simple because you outsource donation management to https://fundraising.fracturedatlas.org, contribution management to https://github.com/standardebooks, collaboration, membership and communication to https://groups.google.com/g/standardebooks, and marketing signup management to... looks like postmark.

So sure, you can run your stuff on a single server, but you're relying on a bunch of other people running much more sophisticated services on a lot more infrastructure in order to do it.

[1] https://github.com/koreader/koreader

* everything running in docker containers, but you aren't supposed to run that locally, only local dev\test is through unit and component tests.

* all k8s is maintained by a separate group pulling in those images through a seemingly very over engineered solution

* when it is all combined in a deployment, things break, or more scary, they don't and you need to manually test and hope it's working.

Issues take too long to track down and resolve and the reasons are very obvious to everyone but technical leadership who are constantly high fiving each other over their amazing creation that is already collapsing under it's own weight.

I'm pretty sure it's possible to configure access to logs, not to the entire thing (whatever that means). He's probably lazy and does not want to bother.

Besides, you should have centralized logging using loki or something similar. Kubernetes logging is not enough for any reasonable use-case.

There's clearly a skill issue at play here and your intuition is correct

If you’re just serving static content then of course you can get away with one small box.

If you have user content that’s being constantly updated and inserted you need a lot more.

You need databases, caches, and in our case elastic search (with ~10 billion documents). The data needs to be indexed 16 ways to Sunday to make sure that a user hitting the page with this filter or that sort selects just the right records.

If you care about reliability you should have read replicas of your DBs as well.

Where are your logs going? I’d expect servers for that as well.

Now alternative to a bunch of this you could just use SaaS products but that costs an arm and a leg.

Anyone saying otherwise failed to do basic capacity planning. Granted at any given time the number of junior/untrained/nontraditional engineers outnumber senior engineers so it’s not surprising things are build out of proportion, and I suspect that’s what’s behind the majority of the anecdotes.

A simple proportional-integral-derivative controller equipped onto your server resource can help you see if future traffic spikes are occurring. The question is, what kind of person is able to have that kind of perception ability and is willing to integrate that into his daily life? Not the average software developer, we can definitely say with great confidence.

So many problems they were dealing with would magically disappear if all the services were running on a single high-end physical machine (the scale didn't mandate anything bigger than that) with a standby one sitting in a different DC for redundancy purposes with incremental DB snapshots shipped to it every 15 mins.

The cloud-native, "modern" infra became a liability and impediment to business but too many people would lose face to admit it.

But I also think that many applications are far more tolerant of small outages than you imply. And a more complex setup also adds more points that can lead to an outage even though it reduces the chance that hardware will cause one.

I think this is mostly a self imposed requirement. Banks regularly have overnight technical breaks. My country national rail has 30 minutes of downtime every night (!).

Unless your product is already global, most people won't have a problem with occasional overnight scheduled downtime.

Most people never even test number 2. If your a novel or niche product, you can afford a “lot” of downtime

For number 3, it really all depend on your load. I’ve run most early stage startups on an incredibly simple setup with a proxy placed in front of an app server.

Occasionally, our monitoring will start reporting high p99’s so we go in and bump the servers to the next tier or scale horizontally just a bit more. Eventually, that breaks down but many startups will be well into Series A or Series B point. At that point, you know your customer’s needs and can hire dedicated engineers to solve reliability and uptime.

(Regular maintenance meaning OS updates needing a reboot, which happens like what? Once per month for a minute?)

You don't even need to be at FAANG-scale for this. Once you have paying customers, you care.

at Grafana we're now decoupling a monolith primarily so teams working on different parts can manage their own deployments/rollbacks/incidents independently of "official" monolith versions. it's a very hard/expensive refactor once you're at a scale of millions of users and hundreds of engs. but really an impossible thing to predict at inception.

If you keep the same database schema this should generally work pretty well.

I migrated a terribly written web app this way, it worked pretty well.

Distributed database is usually painful.

Every company out there loudly claims they need 100% uptime, which grifters will be happy to sell them knowing nobody will actually put their solution to the test and if it does fail they'll have a myriad of moving parts to shift the blame to.

If you look at the very few things that do actually need 100% uptime (real-time stock market, etc) you'll note the absence of most "cloud-native" and supposedly "best practice" bullshit. The reality there is much more boring, serious and rigorous - that's what you get when you actually need 100% uptime.

Reliability through redundancy is simple, particularly when you aren't replicating multiple machines / machine images.