Hacker News Reader

Comment by Rantenki | Hacker News Reader

TIL: AWS is only expensive if your time has no value. Article neglected to add the cost of labour for setup (edit: aside of racking), maintenance (security updates on infrastructure, etc), and disaster response. It also missed the cost of downtime based on projected MTTR for a failure. The word "fail" only appears once in the page, and not in regards to recovery, only preventative replacement.

It's absolutely true that colo is cheaper than AWS for projects that can absorb the labour and failure related costs, but that usually is only true for hobby or very small business services.

View on HN

Do you actually use AWS or are you just parroting something you've read? Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response (US-East-1 outage?) If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice.

I have extensive infrastructure running across 3 separate org accounts for blast radius, with multiple separate VPCs per org, RDS databases across multiple AZs, and a large variety of instances.

Everything is built up and torn down via Terraform, and ZERO maintenance tasks are done via the console. Changes to infrastructure configuration are managed via CAB which approves PRs on Github which are automatically deployed via TF cloud.

We do monthly disaster simulations where we replace a _production_ environment.

I'll agree that AWS paints a very rosy picture of their own services, but there is substance underneath that PR.

We see just as much downtown with AWS with DR/HA/MTTR as with colo. You still have to set it all up and maintain it, its just different setup and maintenance than if you actually have a servers in DCs. Most people still have the learning curve with AWS and usually do it incorrectly, or have to wait for Amazon to admit degradation of a service is occurring.

It's really a matter of who within the company wants to know whats behind the curtain. I'd almost always vote for self hosting but its a hard argument internally when everyone thinks the cloud eliminates these issues magically.

> or have to wait for Amazon to admit degradation of a service is occurring.

Please compare apples to apples. If you are comparing to colo, I don't know of any colocation facilities which provide higher level services (say, Kinesis). They might provide some features like CDNs, but nothing very fancy.

You are left with a EC2 vs servers racked in some colocation facility comparison.

At that point, it's not even fair. I can run a terraform script and spin up a copy of the production environment in minutes. I don't have to create purchase orders and wait for hardware. I don't have to issue tickets. If any machine goes down it can be recovered in minutes, and it will spin up in another hypervisor, maybe even a different datacenter.

You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff.

You don't have to spend cycles diagnosing issues just to find out that there's a bad transceiver in a NIC somewhere. YOU DONT CARE. AWS cares and, even if they can't find an issue, stop/start, new hardware in minutes.

You cannot compare the two in good faith. Sure, it might make sense to host some workloads in a colo, but you are giving up a lot.

There is a lot there, and some of it is apples to apples and some is definitely apples to oranges. I'm just saying overall you have the same number of headaches, they are just different headaches. Just last week there was significant degradation that took quite some time to be admitted (admitted might be the wrong word but everyone seemed to know there was an issue with AWS before AWS said their was an issue)

  > I don't have to create purchase orders and wait for hardware.

Why would this need to be done if initial implementation was planned and done correctly? Baring catastrophic hardware failures you shouldn't need to do this? Im nitpicking your comment and I think is tangential to my original point (And dont want to get into a planning/design/implementation discussion either).

> You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff.

This isn't that hard if you still have the expertise in house. Yes it is work and knowhow, but so is AWS networking/wiring, service interop.

> AWS cares

I think this is arguable and YMMV. Sure it depends on who you are and who you talk to.

Im not saying AWS isn't viable tech, and there are 30 ways to skin a cat at the end of the day if you end up with a skinned cat you like you're good to go. I believe most people don't need AWS to accomplish what they are looking for (subjective for sure). I also find it not the savings or experience that is in the marketing materials or sales pitches. This isn't a binary right or wrong decision, all we need is a skinned cat at the end of the day.

>Why would this need to be done if initial implementation was planned and done correctly?

Anyone who thinks they planned everything correctly is just deluding themselves. Admitting you've fucked up, don't know how you fucked up yet but will one day need to fix the fuckup is very important in engineering.

It's possible but not to think enough through the process that you need to create purchases orders large enough to need approvals? However if you don't have a basic understanding of hardware you'll need you probably shouldn't be working/in charge of this proejct anyhow. If you fked up this bad you're in for a bad time regardless of where you've decided to host.

That's the thing about this argument- it gets sold as not having to worry about something (racking etc) except the reality is it's just a different worry.

Setting up and maintaining Terraform scripts to manage your AWS infrastructure takes time and effort; besides, there’s this nagging feeling that should something happens to Terraform you’d suddenly be left trying to keep track of hundreds of resources (which cost you money each hour, or millisecond in case of Lambda) and their access policies “by hand”.

On the other hand, it could be a very powerful combo.

What’s CAB?

> If you've run anything at any scale for any length of time you'll know that EC2 hosts frequently get rebooted with short notice.

They do not. We run many thousands of instances on AWS. Sure, given our scale, every week we get a couple of instance retirement emails. Usually they are issued many days in advance(sometimes, weeks). Per year, we may get a handful of instances that are suddenly unresponsive.

And we don't care. You know why? Because it's just a matter of issuing stop/start. Done! Server is back up, potentially even in a different datacenter, but it is none the wiser. It looks like a reboot. Even better, add an auto-recovery alert and AWS will do this for you, automatically. If part of an ASG, add health checks.

For the most part, we don't even notice when instances go down. Our workloads are engineered to be fault-tolerant. If a meteor destroys one AWS datacenter, it might temporarily take out some instances. So what? New ones will be back very shortly, all the while databases will fail-over, etc.

If these were physical instances, someone would have to do the maintenance work, purchase orders, wait for hardware to arrive, and so on and so forth. And, for most "co-location" scenarios, if your datacenter has issues, everything will go down. A single AZ in AWS has multiple datacenters, you might not even be affected if one goes up in flames.

But let's say you run a massive pet server farm and none of then can go down for any period of time. You have given names and everything, and you celebrate their birthdays. Cool. Run that on GCP then. They do auto-migration. I've never seen an instance go down.

> Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response

This is true. However, if you are running your own hardware, you have to do that IN ADDITION TO dealing with hardware and datacenter shenanigans, with either a specialized (and expensive) workforce, or a barely capable one that's shoehorned and doing double duty, with zero economies of scale, probably in a single data-center.

I have to echo everything you're saying here.

Anyone putting anything in "the cloud" and not designing for failure as a first principle is shooting themselves in the foot. I've encountered a lot of folks trying to do this and building systems that can't tolerate downtime and it starts long, difficult conversations always.

But I've never once needed to actually pay attention to those instance retirement emails or GCP auto-migrations. We've built to expect it and we let the provider handle that for us.

Time saved for everyone.

They do

If you're at the scale where a data center going down is a problem and thus you need to run in multiple coasts for disaster recovery, you're going to have to be able to handle the kinds of distributed computing issues that'll crop up in both cases.

The advantage of AWS is that in theory you can script your infrastructure. Deploy to multiple zones, add hosts, remove hosts, deploy artifact, etc. Makes all kinds of things very easy. Dedicated SQL / Cassandra / DynamoDB offerings make that a few clicks to get going.

This saves people costs, but just so happens to be more expensive in terms of hardware costs.

You can replicate all of this in your own DC or whatever. You do need to invest in tooling, hardware, etc. which is worth it at the highest scales. Or worth it if you don't need everything and thus don't need to pay for it.

Personally, I think many large organizations can optimize by running with a mix of on-prem and cloud services.

Demand may be elastic but not totally elastic, so your base load could be cheaper in house. Storage is cheaper on premise for huge datasets.

You can afford enough staff to rack servers, but you aren't dominated by the lack of servers.

You buy more EC2 instances to scale your business, letting the product team deliver the product, and deploy on-prem to squeeze out more efficiency behind them. Yes, you probably don't use the full offerings of a cloud provider, but that also means less lock in at a cost of a little lower convenience and more staff.

Idk why this take isn’t more popular.

Nothing is stopping you from spinning up ec2s for massive spikes in traffic just because your app is currently on-prem.

Nothing is stopping you from scaling first to AWS, realizing oh this is normal now, and then procuring hardware.

The cost of being down might be high and happens seldom enough that hiring people to sit idle is really expensive.

I have several million dollars in infrastructure that I'm responsible for in multiple cloud providers and on our own metal in multiple physical datacenters.

In terms of management burden, I will take the cloud infrastructure a thousand times out of ten.

Automation work is easily an order of magnitude of less effort in the cloud than on-prem. Labor dollars spent go exponentially further.

They don't get force-rebooted that often, I have EC2 instances that have been up for over 365 days with no issues.

Setting up instances, managing updates, adding IAM perms is orders of magnitude faster/easier than dealing with rack-and-stack data centers.

Full downtime isn't that frequent, either. The Kinesis outage on Wednesday didn't affect any of our USE1 functionality, with the exception of maybe Cloudfront propagation.

I don't remember them force rebooting for nearly 10 years.

A server is easily surviving over 1000 days and it's usually our side that needs to reboot.

I agree in general, though many were affected by the 2014 Xen security issue.

Everything you said still has to happen with colo, as well as everything the parent mentioned.

"The cloud is just someone else's computer" - I used to say that with contempt, and now I say it with understanding.

Things got rebooted a lot due to Specter and Meltdown mitigations a few years ago, but rarely now.

We’ve had more issues with the collocation data center, like removing one of the good drives in a degraded raid-array and extremely expensive bandwidth.

EC2 is hardly the main service that people decide over. EKS, ECS, Fargate and Lamba, Aurora and other managed DBs, S3 eliminates a lot of the setup skills and time required to setup something even remotely reliable and on par in features. Sure if you need a reliable VM there are cheaper options perhaps.

clearly you've never had to deal with physical servers. A dead mouse once caused an outage (fire) that took an entire dev team days to fix. I'd rather not deal with that in my lifetime.

Running my site on dedicated servers costs me $500 a month, running it on AWS would cost me about $20,000 a month (probably just for the egress traffic alone).

I could hire four system administrators for that money to look after each of my four dedicated servers full time.

On top of that you first need to learn AWS. That thing is complicated - but I can set up a dedicated server in minutes because it's indistinguishable from my development environment.

I joke that dedicated servers are only expensive if your time and money are worthless. Because AWS is going to eat both.

How much of that egress traffic would be saved by using a CDN?

Can you please explain how would this reduce cost? 1TB of egress on EC2 is $90 and on CloudFront is $85. Or are you suggesting to use something like BunnyCDN with $5/TB instead of AWS services?

He did,

Quote :

We are going to say we used four hours of labor. This includes drive time to the primary data center. Since it is far away (18-20 minute drive), we actually did not go there for several quarters. So over 32 months, we had budgeted $640 for remote hands. We effectively either paid $160/ hr or paid less than that even rounding up to four hours.

Read those figures again. They're such an underestimate as to be laughable, and apply only to the physical racking efforts.

There is no way in the world that the only effort taken related to infrastructure is entirely encapsulated in ~ $200 per year.

100%. We had 11 months where nobody touched the racks as an example and the visit 11 months later was the one to physically get a tool I had left there. I mentioned I even track drive time.

How much time did you spend deciding which components to upgrade and purchasing parts to later install? How much would you charge a client for that time?

Very little, but that is probably because we cover the server industry in-depth with reviews and such. I mentioned a bit that that knowledge is key. This is our cost analysis and not going to be the same for everyone.

For a business with growing needs (most people at the point of making the decision), won't they need to go to install more hardware frequently, if nothing else?

Stackoverflow was run on a few servers (don’t know about how many there are now) http://highscalability.com/blog/2014/7/21/stackoverflow-upda...

Often you don’t need that much hardware.

That may be true. We grow 20-30% Y/Y which is not as big as many other places (but is more than most others in our space.) 20-30% growth we handle with refresh cycles since we have extra capacity.

A lot of businesses aren't growing at breakneck speeds, and with modern cpu/ram setups, are often massively underprovisioned. A very big car rental company you'd know the name of was being run on four servers, and they were only upgraded at EOL (if something broke, Dell had same day or next day service)

> They're such an underestimate as to be laughable

If I am reading the post correctly, those are actuals, not estimates.

They are perfect world actual hours that don't include any unexpected downtime, or any ancillary costs outside of racking.

> They are perfect world actual hours that don't include any unexpected downtime

It isn't clear from this particular post, but considering Patrick refers to "overprovisioning" and having a "hot spare" I get the impression that they have designed their DC setup to minimize unexpected downtime. Sure, it can still happen, but with proper up-front planning what you call "perfect world" can become typical.

> or any ancillary costs outside of racking

As I see it from Patrick's post and my own colo experience, there are three groups of costs for colo that are baked in to the cost of AWS compute and storage: capital cost of hardware (Patrick has a specific line item for this); facility cost; and labor for maintaining the hardware (which is the cost under discussion. What do you think is significant, missing, and unique to colo?

This is SO myopic.

You think bad things don't happen on AWS? You think that an experienced admin isn't going to spend significantly more time trying to communicate with an actual human at Amazon than fixing a problem on a physically controlled server?

This is incredibly naive.

We moved to AWS because bad things happen there less often than they were happening to us with various DC providers we used to use.

I've never had AWS tell us that they can't find the server we're renting that suffered HDD failure. AWS has never cut us off from the Internet because a sysop typed a command in the wrong session.

And we find AWS support to be far quicker to respond, and far more effective, than our previous providers.

So, YMMV, but AWS is pretty damn good for our requirements.

I agree with you, but I can't find a good way to communicate this to the other camp. I wonder if you can't clearly see this without having been a "sysadmin" long before "devops" was a title.

It's just that they have a vested interest in not understanding or even acknowledge your arguments.

Scalability is the thing and AWS its Viagra. Whoever has the biggest scalability get to pat itself in the back.

It's both myopic and naive to assume there's a binary set of circumstances that lend an upper hand to self-hosting or cloud hosting.

It's a sliding scale. What makes you money? What's your expected growth? What are your security needs? What experience does your team have?

In this thread, people at each end of the scale are 1. conflating different use cases (wordpress is not the same as a massive, scaled data ingest pipeline) and 2. are ignoring the hidden costs of both.

> TIL: AWS is only expensive if your time has no value.

Is it only expensive if developer time is free or if it is, let's say, a quarter of a typical Bay Area senior engineer?

Salary levels are vary quite a lot across the world, even looking only at developed countries. At what $/hr level does managing your infrastructure become more cost-effective?

The labor was added in this calculation with data from the last 3 years. It's a lot less then one might think.

> very small business services

Which accounts for almost all hosting and startups. Most startups will never see more than a handful of users and yet microservices, lamdba, nosql etc must be deployed because ‘scaling and failover’, wasting their own and investers’ money for nothing. Only a small % will ever need any scaling and failover.

If aws were truly cheaper, we would be using it for everything; currently we mix which is far cheaper. Anything not fail critical that benefits from fixed price cpu/gpu, bandwidth and storage we host traditionally. We and our clients saved fortunes over the years. And for clients or projects that do not need such heavy lifting as aws, we simply host traditionally completely.

It also depends on how computationally intensive your application is, AWS premium can be higher than labor cost if you need large scale data crunching.

Or if you need to egress a lot of data to the Internet. Or send a lot of data between different AZs. Or between Regions. Or anywhere really. Plus now you have to get a PhD in AWS Billing first: https://twitter.com/QuinnyPig/status/1155838435065655296/pho... and https://twitter.com/QuinnyPig/status/1172239124251709449/pho...

That's definitely true. It still makes sense to run your own AI/ML servers on-prem if you're saturating some $20k/each GPU boxes, for example.

But ... most web applications are still IO bound, and labour is surprisingly expensive.

Very true,

Running something on the open Internet is super annoying. If you properly subnet, or restrict to VPN, I can see it as useful but then you have a whole other host of problems to conquer.

If you're doing a static page or such, no issue, but the second you're hosting a webapp or need to have a database, yeah.

When you're uninformed and don't know what you're facing, things seem complicated and cloud solutions tend to look very attractive but what is so hard about running a web app?

For sake, just spin up a single server somewhere cheaper and run it all in there and vertically scale up after you've put the appropriate monitoring to know when you need to scale up.

And put your money where it's worth it.

I agree with you. As counterpoint, I was embarassed for our profession when something like 70% of people in the industry said they were confused with the syntax for httpd.conf

I agree. I’m a contractor and my client is running Azure. Having the possibility to snap my fingers and have a Redis cache appear ready to go is a no brainer - it costs per month what I cost per hour.

Despite the naysayers, you're completely correct. The effort required to build and maintain AWS is so much lower than on-prem. The only time this isn't true is when you have highly optimized, calculated needs where investing in that specialized infrastructure will pay off.

I want my team focused on building stuff that makes us money.

Reply on news.ycombinator.com