Hacker News Reader

Comment by idunno246 | Hacker News Reader

The stuff you’re saving time on, managing hosts, just isn’t that much compared to dealing with aws/terraform/etc. I’m at a few orders of magnitude more spend than that, and in absolute dollars is very much noticeable how much more expensive aws is. But as a percentage of costs it’s so little next to wages, as is your example, that cost is really the wrong thing to get hung up on. Most engineers I’ve worked with don’t get that, they see a value they can objectively minimize, without realizing that even dropping it to zero his little business effect.

View on HN

> The stuff you’re saving time on, managing hosts, just isn’t that much compared to dealing with aws/terraform/etc.

This is where I feel like I'm doing stuff wrong. If we're comparing Terraform, it means we probably care about reproducibility (rather than an "AWS console vs SSH into pet hosts" scenario), so the like-for-like comparison involves bringing in something like Ansible. On top of that, you need to pick, install, and configure logging exfiltrators, monitoring agents, process managers, etc and you need to operate systems that let you explore those logs and metrics. You also need to configure SSH access and manage keys. You may also need a custom base image, so maybe you're doing packer stuff as well? On top of that, you need to run some database which means managing backups and running replicas with failover (or maybe we/re a small business and we don't care that much about reliability?). And again, we care about reproducibility, so we need to encode all of this stuff in Ansible playbooks or similar. You probably also need something like security groups to restrict which things are allowed to talk to which other things, and encoding this in Ansible or similar is maybe impossible if you don't have software-defined-networks.

It seems like a lot to get to parity with what someone could throw together with API Gateway, Lambda, S3/DynamoDB in a couple reasonably-sized Terraform files in a few hours for a pretty marginal cloud spend (most small businesses would probably stay pretty close to the free tier--these services are extremely inexpensive).

Yea, I mostly agree. I feel like a lot of the negativity comes from hobbyists, where an extra hundred dollars does matter.

I think one place that I disagree is while you could do that in a few hours, your average developer couldn’t. You’re now talking about everyone to learn lambda and terraform and whatnot, whereas with a “standard” web server, that people are familiar with, a lot of that is more easily centralized. just throw some annotations and routes are done, vs the arcane api gateway config. The tools and frameworks for lambda just didn’t seem to be there yet.

Fwiw I’m all in on aws, cost was one of the easiest arguments to deflect. Ultimately we needed to show developer velocity increases as that’s the cost that mattered. And security isn’t compromised, which the bigger the company the more roadblocks I’ve seen to just give devs terraform.

Agreed that there is a learning curve, I guess I was assuming some competence with both stacks—not starting from scratch (although I think if you were starting from scratch it would still be easier to learn AWS than self-managed hosts).

Yeah, big companies don’t like giving devs raw Terraform. My company has sandbox environments where devs can do iteration with permissive Terraform access. That works pretty well for stuff like this.

Also, I use AWS for hobbyist stuff and you can easily use this serverless stack for <$5/month.

I would like to see some real numbers on this. I spend between $5 and $6 million a year on infrastructure. I have a physical datacenter in a Colo and two clouds AWS and Azure. I have detailed KPIs on my spend. One of the KPIs is total cost of ownership of an instance or server. My physical instances cost 3 times as much as the exact same compute in the cloud.

This is because I have to factor all the costs. This includes electricity, maintenance, incident response, networking, renting the cage, vendored software for backups, threat detection, fire suppression, equipment upgrades, licensing, alerting, and it goes on and on and on...

I'd challenge you to break down the full cost of owning a server as you see it. I bet you will miss 75% of the actual costs involved. I promise short of seizing a colo like its Nakatomi Plaza and running it at gunpoint you will never in a million years come close to the total ownership cost of cloud instance. You can't compete with the economies of scale and the caliber of the engineering.

37signals disagrees. They’re moving off the cloud and have published several blog entries about the process and the approx ~1mm/yr (fully loaded) they expect to save.

I don’t have a horse in this race, but it’s interesting that your experiences are so different. Would love to hear your take on their numbers.

The most recent article, with lots of hard numbers, is here:

https://dev.37signals.com/our-cloud-spend-in-2022/

There was a recent article that made the HN in front page that broke down the savings they expected, but I’m on mobile and can’t find it now. Something about “two datacenter racks.”

Where are the numbers and how do they calculate them? This article lists their cloud spend in detail then "waves hands" at the Datacenter costs simply saying "it will be far far less". Ok why not break it down?

I don't understand how they are going to achieve that. Does it include routers, switches, IPS's? What about the costs associated with having a physically wired network instead of a software defined network.

Also they state they are region redundant which is probably way overboard. Will they be protected if they lose their entire datacenter? Will they flop over to another geo? If not then you must consider not their current spend but their spend if they were single region. That would further eat into proposed savings.

Don't get me wrong, I do believe you can achieve cost parity in a Datacenter but you need a certain level of scale. I am skeptical that it can be done at $3 million in spend.

I’ve done the modeling down to pulling actual quotes in my area and with a decent footprint (like couple hundred xxlarge instances) you break even after a year (including colo, remote hands and networking). Cost is decisively not the reason most orgs choose public cloud

Here's their article. "We stand to save $7m over five years from cloud exit."

https://world.hey.com/dhh/we-stand-to-save-7m-over-five-year...

>But this isn't just about cost. It's also about what kind of internet we want to operate in the future. It strikes me as downright tragic that this decentralized wonder of the world is now largely operating on computers owned by a handful of mega corporations.

Yep read that guys blog history and the agenda just pops right out. It's not just about cost for Basecamp its ideological. I can't help but imagine this bias leaks into the financial and operational calculations.

We had a similar situation. We had a team did not want to move to the cloud, the business forced them, so they built the system in a way that fought the cloud. Then the self fulfilling prophecy kicked into high gear. "See we told you it was a bad idea, look at all the problems we have!". The problems were created through half baked attempts to be "agnostic" to the cloud. Once we removed those elements we were able to reduce the cost of the system by over 90%. It was far cheaper than when it was running in the Colo. These folks had no interest in optimizing for cloud native execution they were already planning their move back into the Colo.

Reply on news.ycombinator.com