Hacker News Reader

It seems like such a waste of time and complexity for a knob that only a tiny, tiny fraction of startups will ever have to turn.

I think of cloud platforms as less complex than managing your own systems. It's trivial to go from zero to functional product on GCP or AWS, and it's honestly not very expensive to get started either. GCP and AWS very clearly subsidize costs for early-stage startups in an effort to lock them in. A tiny startup might have a cloud hosting cost of 3 figures a year until they really take off.

And when your product does take off, scaling is often trivial.

Yes, scaling is trivial on AWS if you rewrite from scratch 3 times and rearchitect 4 times, but at least you don’t have to enter your credit card number twice.

His assumption is that you're familiar with cloud design patterns. So if you know what you're doing, costs are manageable and scaling is a breeze, that's what the cloud was literally designed for.

I just don't see this. Building what I am describing can be done in days sometimes hours. Where exactly is the time and complexity? I would be willing to bet given a true greenfield(e.g. I can't use an ansible script that stole from my last job) I can build an autoscaling flask app with CI/CD and local environment on ECS twice as fast as accomplishing a lesser facsimile with a bare metal server or a VPS. If I built on a VPS I would still have concerns like backups, High Availability, DNS, Networking, etc. etc. to deal with.

I consider the latter to be far more complex.

I'm often perplexed when I hear people complain about cloud costs. I can't tell if people just have really different workloads than I've experienced or if they just haven't considered cost when they pick their compute, databases, etc. If you use object storage, some serverless database, and some serverless compute a small company (by which I'm thinking "normally very low volume with occasional, very important, often large bursts") can keep their cloud spend to tens of dollars per month and save considerably by not having to manage all of the stuff that goes along with running VMs, database instances, or physical hosts.

Like, I assume I'm the one missing something because people swear up and down that it's the other way around, but I haven't found the happy path for managing hosts (virtual or physical) that delivers on the promise of simpler-than-cloud. There's a disconnect somewhere because I'm sure there are a lot of people who can't find the happy cloud path that delivers on the simpler-than-diy approach.

The stuff you’re saving time on, managing hosts, just isn’t that much compared to dealing with aws/terraform/etc. I’m at a few orders of magnitude more spend than that, and in absolute dollars is very much noticeable how much more expensive aws is. But as a percentage of costs it’s so little next to wages, as is your example, that cost is really the wrong thing to get hung up on. Most engineers I’ve worked with don’t get that, they see a value they can objectively minimize, without realizing that even dropping it to zero his little business effect.

> The stuff you’re saving time on, managing hosts, just isn’t that much compared to dealing with aws/terraform/etc.

This is where I feel like I'm doing stuff wrong. If we're comparing Terraform, it means we probably care about reproducibility (rather than an "AWS console vs SSH into pet hosts" scenario), so the like-for-like comparison involves bringing in something like Ansible. On top of that, you need to pick, install, and configure logging exfiltrators, monitoring agents, process managers, etc and you need to operate systems that let you explore those logs and metrics. You also need to configure SSH access and manage keys. You may also need a custom base image, so maybe you're doing packer stuff as well? On top of that, you need to run some database which means managing backups and running replicas with failover (or maybe we/re a small business and we don't care that much about reliability?). And again, we care about reproducibility, so we need to encode all of this stuff in Ansible playbooks or similar. You probably also need something like security groups to restrict which things are allowed to talk to which other things, and encoding this in Ansible or similar is maybe impossible if you don't have software-defined-networks.

It seems like a lot to get to parity with what someone could throw together with API Gateway, Lambda, S3/DynamoDB in a couple reasonably-sized Terraform files in a few hours for a pretty marginal cloud spend (most small businesses would probably stay pretty close to the free tier--these services are extremely inexpensive).

Yea, I mostly agree. I feel like a lot of the negativity comes from hobbyists, where an extra hundred dollars does matter.

I think one place that I disagree is while you could do that in a few hours, your average developer couldn’t. You’re now talking about everyone to learn lambda and terraform and whatnot, whereas with a “standard” web server, that people are familiar with, a lot of that is more easily centralized. just throw some annotations and routes are done, vs the arcane api gateway config. The tools and frameworks for lambda just didn’t seem to be there yet.

Fwiw I’m all in on aws, cost was one of the easiest arguments to deflect. Ultimately we needed to show developer velocity increases as that’s the cost that mattered. And security isn’t compromised, which the bigger the company the more roadblocks I’ve seen to just give devs terraform.

Agreed that there is a learning curve, I guess I was assuming some competence with both stacks—not starting from scratch (although I think if you were starting from scratch it would still be easier to learn AWS than self-managed hosts).

Yeah, big companies don’t like giving devs raw Terraform. My company has sandbox environments where devs can do iteration with permissive Terraform access. That works pretty well for stuff like this.

Also, I use AWS for hobbyist stuff and you can easily use this serverless stack for <$5/month.

I would like to see some real numbers on this. I spend between $5 and $6 million a year on infrastructure. I have a physical datacenter in a Colo and two clouds AWS and Azure. I have detailed KPIs on my spend. One of the KPIs is total cost of ownership of an instance or server. My physical instances cost 3 times as much as the exact same compute in the cloud.

This is because I have to factor all the costs. This includes electricity, maintenance, incident response, networking, renting the cage, vendored software for backups, threat detection, fire suppression, equipment upgrades, licensing, alerting, and it goes on and on and on...

I'd challenge you to break down the full cost of owning a server as you see it. I bet you will miss 75% of the actual costs involved. I promise short of seizing a colo like its Nakatomi Plaza and running it at gunpoint you will never in a million years come close to the total ownership cost of cloud instance. You can't compete with the economies of scale and the caliber of the engineering.

37signals disagrees. They’re moving off the cloud and have published several blog entries about the process and the approx ~1mm/yr (fully loaded) they expect to save.

I don’t have a horse in this race, but it’s interesting that your experiences are so different. Would love to hear your take on their numbers.

The most recent article, with lots of hard numbers, is here:

https://dev.37signals.com/our-cloud-spend-in-2022/

There was a recent article that made the HN in front page that broke down the savings they expected, but I’m on mobile and can’t find it now. Something about “two datacenter racks.”

Where are the numbers and how do they calculate them? This article lists their cloud spend in detail then "waves hands" at the Datacenter costs simply saying "it will be far far less". Ok why not break it down?

I don't understand how they are going to achieve that. Does it include routers, switches, IPS's? What about the costs associated with having a physically wired network instead of a software defined network.

Also they state they are region redundant which is probably way overboard. Will they be protected if they lose their entire datacenter? Will they flop over to another geo? If not then you must consider not their current spend but their spend if they were single region. That would further eat into proposed savings.

Don't get me wrong, I do believe you can achieve cost parity in a Datacenter but you need a certain level of scale. I am skeptical that it can be done at $3 million in spend.

I’ve done the modeling down to pulling actual quotes in my area and with a decent footprint (like couple hundred xxlarge instances) you break even after a year (including colo, remote hands and networking). Cost is decisively not the reason most orgs choose public cloud

Here's their article. "We stand to save $7m over five years from cloud exit."

https://world.hey.com/dhh/we-stand-to-save-7m-over-five-year...

>But this isn't just about cost. It's also about what kind of internet we want to operate in the future. It strikes me as downright tragic that this decentralized wonder of the world is now largely operating on computers owned by a handful of mega corporations.

Yep read that guys blog history and the agenda just pops right out. It's not just about cost for Basecamp its ideological. I can't help but imagine this bias leaks into the financial and operational calculations.

We had a similar situation. We had a team did not want to move to the cloud, the business forced them, so they built the system in a way that fought the cloud. Then the self fulfilling prophecy kicked into high gear. "See we told you it was a bad idea, look at all the problems we have!". The problems were created through half baked attempts to be "agnostic" to the cloud. Once we removed those elements we were able to reduce the cost of the system by over 90%. It was far cheaper than when it was running in the Colo. These folks had no interest in optimizing for cloud native execution they were already planning their move back into the Colo.

These complaints mostly seem to come from folks who are spinning up lots of VMs and architecting solutions the same way they would build in a data center. If you're running VMs 24/7 of course it's going to be expensive.

This feels right to me. If you build with lambda, api gateway, and s3/dynamo a small business can stay pretty close to the free tier. Obviously this doesn’t work for every single application, but I’d bet there are a lot of VM applications that it would suit just fine.

> He instilled in me the lesson of knowing approximately where you are going to land and not do anything now that would jeopardize that landing.

Thanks for sharing that.

> object storage

Is this like S3 or more like DynamoDB, or something else?

S3

I have to disagree. There is an initial phase where a startup can simply use docker to provide their service. Having prepared that initial layer they can easily "scale up" and move parts to cloud services as they require.

You can get a VPS for like $5/month. So you can save near enough hundreds of dollars a month.

I don’t think cloud cost is anywhere near as much as the cost to build a semi-decent HA/DR system and the devops process around it. Every minute the site is down it’s real monetary loss for most startups. It also takes time and energy away from building features. I have done both bare metal and cloud in two different startups. (This assumes you can’t afford to lose data or be offline for extended periods of time).

Here’s what you’ll need to setup and run a single database :-

- Installation script and infra provisioning. You need a repeatable way to do this with as little manual intervention as possible and I can swear it never just works.

- Backup and recovery scripts and associated monitoring

- Replication setup, monitoring and debugging. It will break often.

- Monitoring for infra - CPU, memory, network. There’ll be unforeseen spikes and random slow downs and a whole bunch of slow queries you’ll debug on a regular basis.

- Security - users, permissions, auditing. Both machine and database. Regular password rotation and firewall rules. Ability to off-board users.

- If you survive all this, being able to scale up. And I mean just vertical scale up will take you far but you cross that limit, you now suddenly have to deal with sharding or partitioning.

So when I joined a new startup and we were building it all from scratch it was a very easy decision to go with the cloud. We spent $3500 a month. Not a lot but nothing compared to the salaries and marketing spend.

The hundreds of dollars a month was quoting the comment I was replying to.

The underlying point I made that is being ignored is that 99.9% of start ups never need any of the stuff you are talking about. And hundreds of dollars a month is a large amount of money for them.

So, say you're saving $2500 a year? That seems pretty cheap for an option to scale up without fundamentally changing your operational architecture.

The bigger question mark than the money, I think, is whether this scale up approach actually works, or whether you end up fundamentally changing everything anyway. In my experience it's a mixed bag. There are some advantages, but it also isn't a panacea, and certainly isn't as easy as turning that dynos knob on heroku was. But at the end of the day, I land on it being worth it for a business (though not for a hobby). But it isn't a no-brainer.

A couple hours of engineer time per month. Maybe 5 if we're being generous.

And even more importantly, dozens of hours per month.