Signals says cloud repatriation has already saved $1M | Hacker News Reader

Signals says cloud repatriation has already saved $1M

theregister.com

31 pointsrcarmo47 comments

"37Signals" , i.e. basecamp, Hey, etc. I know HN strips out numbers automatically to remove clickbait articles like "37 signals you need to watch for when buying a SaaS" or whatever, but pretty annoying when it butchers something like this.

Thanks, I missed the second S and thought this was about the Signal messenger.

The absolute bottom of the market lowest price AWS GPU spot instance costs 15 cents an hour.

That’s USD$1,314 per annum for an extremely underpowered machine.

The numbers don’t add up. Just buy a physical machine, plop it on a network somewhere, you’ll have dramatically more computing power and you won’t be paying for it again next year, and the year after that etc

“The cloud is where Moore’s law goes to die.”

Except the need for managing it securely. Oh, it is very successful? Now the 50 dev team each want one.

Oh, you maxed out the number of ports on the switch.

One of those GPUs have customer information that only select individuals should have.

One of them needs to write only for off-site backups.

Every time this on-prem vs cloud stuff comes up folks just skim over all the other pieces included. Didn't even talk about racking, network and or power.

Yes, there is a spot and time where on-prem makes sense and there is also a time for cloud.

I barely cover some of them in this 30 min video, https://youtu.be/8i7i1VFSJrw?si=pLyIhwgxe4d3efvb

You’re adopting an almost religious dogma about this. At the end of the day, it’s a dollars and cents decision.

During development and growth of services, cloud makes total sense as you abstract the capital costs. If you’re a startup making Uber for cupcake makers, you’d be dumb to do anything else.

Depending on your needs, once things settle down, it’s often cheaper and more effective to capitalize your costs yourself.

It’s akin to a car. If you go somewhere for a day or two, Uber makes sense. For a week or more, rent a car, for a year, you probably want to buy or lease a vehicle.

Again, it depends on you. For me, I have large workloads running in GCP and Amazon. But 80% of the production I’m responsible for is “on-prem”, because it’s about 30% cheaper. YMMV

Lease a Server

You get all of above (mgmt, etc) for the same cost.

For example, you can get for the same $1200/year:

- 16 core (Zen4 current gen)

- 128 GB RAM (ECC/DDR5)

- 2 x 2TB NVME

For just $100/mo.

https://www.hetzner.com/dedicated-rootserver/ax102/configura...

I feel like the greatest marketing move AWS has played is getting people to forget that Leasing Servers is a thing.

Hetzner has some weird and very strong opinions on what you can and cannot do on their servers, likely because they are so cheap they attract all sorts of shady customers.

I've had good luck with OneProvider.

> Hetzner has some weird and very strong opinions on what you can and cannot do on their servers

My company has never used them... what have you experienced?

> "very strong opinions on what you can and cannot do on their servers"

Another way of saying that is, "they don't want spammers as customers since that can negatively effect other (good) customers".

Which I'm just fine with.

It's quite a bit broader than that [1], and so worth being aware of as it's stricter than many US providers, but unless you happen to fall in one of those categories it's also not a big problem:

"This includes in particular, but is not limited to, pornographic or obscene material, extremist content or content that offends common decency, gambling, material that could seriously endanger the morals of children or young people or violate the rights of third parties (copyrights, name rights, trademark rights and data protection rights). This also includes the publication of defamatory content, insults or disparagement of persons or groups of persons."

On top of that there's a blanket ban on crypto.

Personally I've used them for 15+ years at a number of companies and never had a problem. But I accept some people will want to set up content that violates or skirts close enough to break those rules to be uncomfortable.

[1] https://www.hetzner.com/legal/terms-and-conditions

Not with a GPU

Until some junior uses an instance store for customer data or creds.

I've done racking, wiring up the network, etc. and if that adds up to more than ~3%-5% of your total spend, you're bad enough at it that you should lease spent at a managed hosting facility. That also makes your power predictable - it's baked into the rack rental most places.

In which case most of the steps you list boils down to "go to their website and fill in a support request" or tick some boxes, and if there's a wait, spin up a cloud instance in the same network in the meantime.

Very few people mean actually on prem when they say on prem. Usually the comparison is with managed hosting or at most colo providers where you can but don't have to wire things up yourself.

Sure, there are use cases where cloud is cheaper, but most people are nowhere near them. There are also cases where actual on prem makes sense. But unless you're somewhere where power and real estate is cheap, odds are your cheapest option will be managed hosting, by a large factor over cloud, and a quite a bit smaller factor over on prem.

> Except the need for managing it securely.

Putting a server in EC2 doesn't make it secure.

> Oh, it is very successful? Now the 50 dev team each want one.

Cool, now you're paying 50x as much.

The tasks you are talking about is not rocket science.

Before the cloud most companies did this themselves.

You assume the only people with the capability to do this stuff are cloud vendors.

Of course it saves money. The main selling line of the cloud is you're too dumb to administer, so just trust them and throw money at them.

If you're not too dumb to administer, then running your own hardware is invariably cheaper.

What's surprising is that so many otherwise technical people believe the cloud BS without really thinking things through. Anyone who can do basic maths can see that aside from trivial uses renting is always going to cost more in the long run.

I don’t think that’s the main selling point of cloud? The main selling points are that (1) you can scale capacity up and down, so you aren’t purchasing, provisioning, and operating hardware for peak demand (2) with infra-as-code tooling, software teams can configure and deploy significant infrastructural capabilities without scaling specialized operations roles. Dumbness does not play into it.

IME, much of the “need to easily scale” is fixed by understanding your bottlenecks, being able to understand the metrics being captured, and how to address them. You would be shocked (or not) at how few people raised on cloud can explain what IOPS are, or what IOWAIT is, let alone linking the two up.

Re: IaC, you definitely should understand what it is you’re instantiating (and why), no matter how you’re doing it. AWS Community TF Modules are all well and good until something breaks, and you realize you don’t really understand what’s going on.

> how few people raised on cloud can explain what IOPS are

I think you would be shocked at how few people raised in a fully on-prem world can adequately explain even that simple concept and its first-order implications.

Thanks, I hate it.

We did an analysis for a local data center leader on their SAN IOPS before we would sign on and they asked if they could have the results because they'd never done it.

I was utterly shocked. Not only because they hadn't done it but that NO ONE ELSE had ever asked and/or done it either.

Literal insanity.

My experience in doing devops on contract for years, though, is that what instead happens is that a bunch of different teams scale things up beyond what they need, don't take things down again, and nobody understands which servers are actually needed or why they're being billed what. There's good money to be made in just helping companies figuring out their cloud billing.

I also don't for a second agree that software teams can deploy significant capabilities without specialised ops roles - if anything my consulting showed me the clients I had that deployed to clouds needed far more help on average.

It may be a selling point, but it's a fiction.

And almost every single colo-provider and managed hosting provider these days also provides cloud services, so you can lease servers or racks, and get cloud instances often in the same data centre tied into your network if you suddenly need to handle a spike. As a result, you can typically load your managed servers far higher today than you used to be able to.

Right, and we're talking computers here.

Sure, they can easily be set up in a complex, failure-prone way. But if we keep it simple and stick to known fundamentals, they can run almost entirely passive.

Think: a basic debian stable server with SFTP vs. a cluster of some complex storage protocol.

The basic debian SFTP server can run for years and years with minimal intervention.

Cloud has become a cargo cult.

Indeed.

Too often I hear about how people need "features" without even knowing what they think they need them for. It's like the drug companies selling drugs on TV - suddenly people are asking their doctors if they need some random drug that has absolutely nothing to do with anything wrong with them.

Your Debian SFTP server example is apt. Many times I've asked people why they think they need all sorts of complicated things when everything they need could be run off of a Raspberry Pi.

I've even set up Raspberry Pis to show people that they literally don't need more than that.

A whole lot of people want to believe because hardware is a big unknown they don't understand that often has complicated requisition paths for them. It's tempting to buy all the arguments when it means you can just get more resources with an API instead of having to justify the need for them.

Had a student do a research thesis on this last year and IIRC whether it's a massive cost saver or an expensive disaster boils down to good old fashioned "system analysis".

Know your system, every part. Big cost savings are often made at this stage just discovering a shit-ton of obsolete stuff.

Don't just map the network, map the _function_ of the network. Then look at its logical versus physical topology for anomalies (also a great security exercise).

What you bring back on-prem must be systematically viable.

Factor in on-prem energy costs, they are not trivial.

The biggest overall factor is people and wages. Any significant repatriation project ends up needing to hire back the system administrator you fired a couple of years ago.

> boils down to good old fashioned "system analysis". Know your system, every part

This type of answer has managed to talk me out of some amazing roles (and dodge some horrible people).

Apparently the correct answer is "there is a magic formula whereby you dont need to understand anything or do any analysis and it will deliver you sunshine and rainbows"

Reminds me of a (public) talk by DIRNSA on security a few years back.

The talk boiled down to: know your network. Know the traffic on it. Then you have a chance of being secure.

NSA isnt selling anything, they can tell the truth.

$Corp who sells Worlds Best network intrusion detection firewall will be telling a different story.

Actual on prem is not for most people. Managed hosting in a data centre run by someone else is the halfway house (and what 37Signals is doing). You get most of the savings, and very few of the pain points.

One of the biggest benefits besides egress should be people and wages. When I consulted in this space, clients on AWS were gold dust, because they tended to spend far more.

A significant reason is that you tend to be able to drastically simplify when you can pick hardware that fits your workloads rather than what fits your cloud provider, and can forego all the multi-tenancy complications your cloud provider needs to account for.

Are you able to share that research thesis? Or any key papers on how this should be done? Knowing SOTA on these approaches is always nice.

Sorry no, I quit teaching at that uni and don't keep contact with the cohort now.

But a little more info I remember:

It arose under a project suggestion three or four students took me up on, and that was "Long term resilience".

The obvious antecedent being "If Google goes down tomorrow....."

"De-clouding" was in the title. And the conclusions, perhaps predictably, were that hybrid systems were the sweet spot across a range of business types, with no panaceas, or even decent maxima to be found in fully on-prem or total cloud.

The students artifact IIRC was a "matrix" (a graph theoretical model) of the edge-costs for traversing well modular systems into different states - something that could be used to plan a "repatriation"

FWIW the student was African and we had a good laugh at the racist undertones of that language and it's subtle implications about "Empires".

Hybrid is interesting because once you set a system up for hybrid it tends to drive down costs without increasing cloud usage much in my experience.

When you can spin up cloud instances, that capability in itself means you can justify loading individual servers closer to 100%. Couple that with managed / leased capacity in data centres rather than "true" on prem, and managed providers often delivering many server models very rapidly, and in most hybrid setups I've worked on, we've ended up using the ability to spin up cloud instances very little, while slowing down the rate of growth of new server capacity overall.

So I agree, they're the sweet spot, but in terms of dollars spent my hunch is that most people deploying them will find their spend dominated by "more fixed" servers, ranging from own/leased servers via managed hosting depending on local power and real estate costs (last placed I ran a physical hosting facility, what finally tipped it towards managed hosting for us was that the price of land grew to a point that empty rack space near us vs. at the managed hosting provider we chose was so much more expensive that the difference paid for our servers).

I dunno, to me the argument is basically, "We can outcompete our competitors and the cloud provides at once". I doubt it, actually, because it requires 37 Signals to make tradeoffs against their primary business, which is whatever people actually pay them for, in favour of something which is not their primary business, i.e. running a cloud platform.

But sure, if all they buy from AWS is VMs, may be better.

The trick is that 99% of the engineering expense at cloud providers is to enable weird multi-tenancy use cases and scale to obscene size. You don't have to do either of those if you run your own hardware - your infrastructure can be quite simple.

AWS is always more expensive than owned hardware + good team -- it's after all what they generate their profits from. However, it's also probable that they were using the cloud wrong. Their software (afaik) is a typical 3-tier application coded on Ruby on Rails, which probably implies that they used just EC2s, with probably an orchestrator on top.

Also, if they didn't need any new staff after moving to their own server rooms, this implies that either their staff was having lots of free time or that their staff currently is overworked.

When I've done contract devops, clients on AWS always tended to need more of my time. The level of devops complexity for AWS is often far higher than self hosted.

It's really hard not to beat AWS on cost, including person hours to manage the setup.

I spent over ten years in the Rails community and as a result consider myself fairly well versed in the things that tend to come out of the mouth of DHH.

This particular one is very much in line with some of his other greatest hits where the more you dig into his claims and assumptions the more things don’t seem to always add up in quite the way he is presenting them.

Without going super deep on the topic, he came to cloud with his on-prem architecture choices, made very few adaptions to take advantage of what cloud is good at besides trying to slap K8s around it and then was upset that the benefits never materialised.

I can understand that being a shitty and painful experience but he is now acting as an anti-cloud evangelist of sorts while arguing against a series of constraints and choices that he picked and clearly weren’t appropriate for his business.

But now for him that’s all that cloud can ever be as it’s all he ever knew.

So in reality they've done two unnecessary migrations, burned a ton of cash on cloud compute and are selling it as a success?

While I don’t doubt your personal experience, I’ve spent time at a cloud-native company and boy, do they know how to spend money. Just because you’ve bought into every .aaS the cloud has doesn’t mean you’ll save money.

RDS, for example, is stupidly expensive. I maintain that it’s not that hard to use Xtrabackup or any of the other free and open tooling to run regular backups for RDBMS. It’s also not that hard to set up HA. If those things seem hard, your Aurora Serverless bill is going to be even harder to swallow when it hits.

Sounds like a great business opportunity for you that will make millions and millions.

Touché.

I have no desire to start an IaaS company. However, if I ever launch a product, it will be running on my own hardware (to be clear, I own the hardware now; this isn’t a future CAPEX) until there is a clear financial incentive to not do so.

The challenge is selling it.

I did devops contracting for years, and I gave my customers a standing offer: If they were not sure they'd save money, I'd guarantee they'd save money, including on my ongoing fees, if they moved to managed hosting instead of cloud, and I'd charge them a percentage of their savings over the first few months only to do the transition.

Even in the face of that, and extensive spreadsheets, it was always a big challenge to convince people to try even with smaller parts of their services. AWS is a known entity. It's not a career risk. If something goes wrong, they have someone to blame. Of the ones that did take me up, one cut their costs (including the devops) by 90%. More typically 50%-70%.

And you're making that effort to make far narrower margins.

How practical is it to run a server at your house? Is fiber a must? What if the power goes out? What about exposing your home ip?

Colo is a much better option, but i've hosted at home before. I have fiber, UPS, and a generator. The home IP wasn't a concern, but you could proxy behind cloudflare or something?

Source: https://world.hey.com/dhh/our-cloud-exit-has-already-yielded...

"We’re still storing eight petabytes of data in AWS’s S3 service. We’re leaving that one for next year. " - DHH in July [0]

That means 37 signals probably still have a 7 figure cloud bill

[0] https://37signals.com/podcast/leaving-the-cloud-the-finale/

Reply on news.ycombinator.com