Hacker News Reader

Comment by mabbo | Hacker News Reader

It's the same story every day, the same pattern. And it's the same because it works.

Startup: use <aws|azure|gcp> because it's quick to get started. It's easy to scale it up 100X in 5 minutes for when you make it big. And it provides a long list of easy to use services that are overpriced per use, but which free you to focus on your business rather than complex DevOps.

Then once you scale to the point that you're comparing your salaries and your ops cost, you find smaller services to handle the expensive part of your business. Ones that do exactly what you need, not much else, and charge accordingly. Maybe they're self-operated, or maybe they're other companies with a narrow focus.

Then you write a blog post about how everyone is crazy to use <cloud> since it's so expensive, and you don't know why you ever thought they were a good idea.

View on HN

Same pattern on HN too - a post about some outrageous bill due to a configuration error or whatever, followed shortly by a post like this one.

I mean it makes sense in a way and I'm not saying it is a bad thing, just a pattern I've noticed.

The third step of this pattern is that vantage company popping up in the comments. We should get bingo cards made

E: hehe there it is, they’re already here

Might want to include a relevant link in your comment? Where is said vantage company in the comments?

The commenter was correct, you can find the appearance via searching 'vantage' with the Find in Page browser feature. Direct link to comment: https://news.ycombinator.com/item?id=30066823

Honestly I decided against linking it in case people decided to go mass downvote it or something dumb

Seeing some of the replies to tweets that appear here I’ve lost faith in the HN audience’s ability to observe and discuss

Pattern, not problem :)

And then there's someone like you who detects that we have a pattern here and that it's repeating itself.

This is about levels of abstraction.

You can ask the cloud to deploy a container to Kubernetes. You can rent a VM with Hetzner. You can rent rack space in a data center. You can build your own data center. You can build your own server hardware.

Depending on the scale you operate, you'll draw the line somewhere where it makes sense to you (which may change over time).

Dev teams will still operate best when they can deploy containers to Kubernetes and without having to design their own server hardware. You now need to build abstractions in house. Maybe you can sell these to other people? Congratulations you are now a cloud provider.

> Dev teams will still operate best when they can deploy containers to Kubernetes

Jeez really? We have like, binaries deployed to machines over here.

Nobody said you can't operate with old-style VMs and ansible or puppet or whatever, but tbqh yes, you're going to move slower.

It may not be fatal to your business; it's not going to be a top 3 reason you succeed or fail! But it does make a difference.

> Nobody said you can't operate with old-style VMs and ansible or puppet or whatever, but tbqh yes, you're going to move slower.

To stay in your metaphor, moving faster is not a guarantee to arrive at your target earlier. As long as everything works fine, using k8s is great. When you have to dig deep to solve a problem, a stack with less layers might be a huge advantage.

Is everything about to move fast? When you read a book, is it a race to get to the last page? When you sing a song, is the goal to get to the last word as soon as possible?

If you'd do it for an employer, the answer to all these questions is yes.

Sounds like you moved your complexity from machine virtualization to container orchestration. Doesn't sound like you won anything, least of all agility.

Binaries are so last-century, now everyone's shipping the entire OS!

Containers can contain just a single statically linked binary. Infrastructure doesn’t care.

They practically never do, nothing that you use or would read about it assumes that you do. That makes this response a little pedantic, even though it is strictly true.

It's actually quite common in the Golang world.

Don’t come here speaking sense against a decade of marketing!

Realistically I wish someone would just write an orchestrator that runs Go processes and nginx and glues it all together.

Can't a container mostly be a Go process?

"Marketing" as you said, made this concept of "container", but I think using the words "process" and "namespaces" is still ok.

I don't agree "container" is marketing-speak; you could say it was appropriated, but I was using linux-containers in early 2000s when it was still a kernel patch. We all knew that bare metal offers a lot and our servers are underutilized most of the time, and VMs were considered a heavier option.

I still think using "container" lose the fact that it's just a process with some tweaks :)

I'm not saying "marketing" as to say it was coined by non technical people, but rather that it's a way to "sell" the feature. Like you go see your manager and say "I'm going to containerize the application" vs saying "I'm going to add some isolation settings to the process"

Consul and nomad would like a word with you.

See my post history. No thanks :)

https://news.ycombinator.com/item?id=30020427

> Realistically I wish someone would just write an orchestrator that runs Go processes and nginx and glues it all together.

Not quite what you're asking for but you can get close, on bare metal, with:

- ssh to set up a systemd service, scp to deploy your binary, then ssh again to restart your service

- dokku (containers are optional) with its cli

For the first option, once you've set up nginx on the server you can easily automate the addition of a new systemd service for a new app with a simple shell script. That done, it is a matter of less than a minute to add a new app and even quicker to update and restart an existing one.

For the second option, it is installed with a single bash command and supports Heroku build packs which means for many systems there's virtually (or exactly) no config needed. Just run the cli from within your app folder. If you do want containers they are automatically found and deployed too.

I've done most deployment methods from the simplest (above) through to complete Terraform builds, and once they are automated all that matters is that the method you choose supports your needs. Go for the simplest option that does that.

I still remember spending money for an FTP client that could “live edit” Perl CGI scripts on the server.

I deploy Kubernetes to Hetzner with https://github.com/vitobotta/hetzner-k3s

And with hetzner you can rent bare metal servers as well. It is somewhat equivalent to renting rack space, but they bring the servers, not you. The advantage is that if a server fails, they can provision a new one in less than 1 hour. The disadvantage is that you are probably running on somewhat refurbished hardware that has a higher prob of failing (at least these were the rumors in the past). 5-6 years ago, this was (by far) the most cost effective solution for Hadoop/HDFS deployments. IDK what the status quo is now.

As others have pointed out it is relatively easy to deploy small k3s clusters on Hetzner Cloud.

> The advantage is that if a server fails, they can provision a new one in less than 1 hour.

I can confirm that. I've been running a few servers with them and my experience is that the older ones had a failure every 2-3 years. Usually it was a disk, sometimes the power supply. Once 2 of 3 disks failed but I managed to rebuild the array (it was the HDD era so it took several hours), no data needed to be restored from backup.

But since 2018 I had no failures at all. And I'm happy to see their ecosystem grow.

Lol you wish. My genius corporate employer has whole data centers with cheap paid-for servers, but thinks moving to the cloud will result in better software because that is what all the startups do. duh

They might not be wrong. In govt at least it was 6+ months to get a single rack of machines provisioned / assignees if you were lucky.

The machines were there fully paid. But w aws I’d be up in 10 minutes w a lot less arguing . I’m convinced this might be part of AWS edge in big players

Same thing in a private company. 45 days to get a server + approval from a VP depending on the cost.

It can be really tough to get people out of the beige box mentality if they have never encountered anything different. The first time you blow away their production server(s) and replace them with what is checked into git can be rough.

Then you get into serverless and suddenly they can’t log into production anymore, and they have to instrument their code, and work within finite resources - all that stuff they ignored in college about manipulating datasets larger then core memory are suddenly relevant.

When we did our transition to serverless we found that the people who complained the loudest about their traditional servers being mysteriously corrupted were actually the same that had obtained admin rights “so they could work faster” and were going into the servers and blowing away configuration files, altering permissions, and making other changes. Once there was no more ssh and no more root privileges our outages dropped two thirds. Today it’s mostly third party outages impacting us and people deploying untested code to production.

We didn’t find that.

What we did find is our operational expenditure over three years tripled. So basically capital expenditure.

I’m not sure how they warped that into self congratulatory success.

The people complaining now complain that it turns out the promise of total automation was a lie. None of it is very robust and all of it is painful.

From a personal project perspective, I've maintained a few VPSes for quite a while now, but now I run them as a Docker Swarm, and deploy containers onto them - or better, I use serverless.

From a maintenance perspective, it's brilliant. Over time, cruft accumulates on my VPSes however disciplined I intend to be when I spin one up. Now, I no longer have to remember all of the ugly hacks I had to implement to get a particular new app I wanted to try out working, I just deploy and it's done.

What makes you think the process for getting a new EC2 instance won't get the same amount of red tape as a physical instance over time?

In some companies, pre paid takes a whole lot of time and paperworks, while post paid only need invoices and maybe half an hour meeting with higher ups to explain the cost.

Because you only talk to finance not IT.

Because compared to a staff of 50 doing anything the costs of an ec2 instance for a week are irrelevant?

Because finance already has AWS in system as a vendor?

Because pricing is public?

Because costs hit your budget not ITs so you have more authority?

Orgs can easily spend 5x what it would cost internally on AWS because of friction avoiding factors like this.

I find it easier to deploy Docker inages to VPSs than to learn how to use countless AWS services. My go-to combo for a new project is DigitalOcean and Cloudflare.

If we're talking about backup and recovery strategy, cloud is undoubtedly easier.

Setup one that doesn't eat your wallet is what makes it hard.

A better starting point would be a vps like DigitalOcean or Linode or Vultr. These days I’m not recommending anyone to use “cloud” (aws/azure/gcp) unless they have a specific need.

Hetzner provides exactly that: a VPS. They're also cheaper than DigitalOcean, I assume, because EU web hosting is way more competitive and US VPS offerings have traditionally been way overpriced.

Let's also be realistic here: CloudFront is a good solution if you need a file to be served over CDN with low latency in multiple regions. Raw s3 or just a web server is much cheaper if you don't need low latency over multiple regions.

>EU web hosting is way more competitive and US VPS offerings have traditionally been way overpriced

US offerings being way more overpriced than EU alternatives also boils down to the much larger capital/war-chest US tech companies have at their disposal to spend on Ops to conquer a market, a fact reflect in those prices but also in things like dev salaries, office equipment, etc.

EU offerings are better priced because most EU companies, who aren't bankrolled by some mega VC fund, cannot afford to pay US levels of pricing.

For what I've seen recently, it feels like hosting might be cheaper in EU than in US, but housing/colocation is so much cheaper in the US.

I think OCI has pretty damn good offering that sit between DO and AWS. Especially when you can mix and match your CPU / Mem Config.

There are two numbers that matter: days until the product is ready, profitable; days until the money runs out. Startups make decisions to improve the relationship between those two numbers.

Using Cloud services that are obviously more expensive than self-hosted options reduces both numbers- less time to launch stuff, but less money. The point is that in early-stage startups, the costs are so low that paying an order of magnitude more for Operations may not move the second number by very much at all, since most of the money is paying for developers.

One strategy has a high fixed cost and a low variable cost. The other has a low fixed cost and a high variable cost. Neither dominates - even for the same firm over the course of a year.

Hetzner has no fixed cost. It's billed just like AWS EC2 - just (a lot) cheaper.

Hetzner is fixed cost. You add things, click checkout, get resources, and your bill stays the same until you change something.

Depends on who is "you". When I provide a Kubernetes cluster on the Hetzner Cloud I usually need to provision dynamic storage as well - that gets allocated through developer's deployments. Also provisioning VMs for autoscaling can be automated. Of course I can lock everything down but then it's no longer a "cloud".

Indeed. I don't even see this as a bad thing - it means you pick the best option given your shifting criteria which is what we should be doing. Blindly doing one or the other makes no sense. I don't particularly enjoy the blog posts that are hyperbolic (not necessarily this one) - both AWS and other things can both be simultaneously a good choice for any given company.

But moving off of the cloud is not always realistic. If the start-up leveraged cloud-specific services, they are now locked in. If they remained cloud-agnostic, they were under-utilizing the cloud.

How many companies actually shift their infra vendor?

Yep, and most likely the service they switched to is running at a loss until they get enough customers, and then will start uping their costs, continuing the cycle (either migrate for cost savings, or stay)!

In actuality, how many startups have switched from <cloud> to an alternative setup? Truth of the matter is that once a startup gets to a certain size, they use these to lock in savings versus moving to bespoke infrastructure: https://www.parkmycloud.com/blog/aws-edp/

A lot of startups perpetually run on VC money and don't particularly care about becoming profitable. But those that do make it sometimes shift at least some of their infrastructure - see Dropbox or Netflix (with edge caches), Cloudflare, etc.

It probably depends a lot on whether the startup is selling a service or collecting eyeballs to sell to advertisers. Cloud could be a lot more efficient (all told) for the former than the latter.

I'm not sure why people try to revise their logic from the past. It probably made a lot of sense to start with a cloud provider. It made sense to move the critical parts out as the company grew. They were all completely valid choices.

Exactly. In the devops journey cloud cost optimization comes after deployment maturity.

Granted, thinking about this up front is ideal, but it’s better to start an mvp and build from there than to be paralized by perfection at the outset and not build anything at all.

Correct except in the vast majority of cases it still isn't worth it for the business to switch to the smaller, specialized service because the operational costs of rewriting/training/managing multiple clouds are too high.

Yep - this exactly - except some businesses have good margin or being close to customer compute is valuable - snowflake following this advice and moving to hetzner would be value destroying in some cases

Is aws|azure|gcp really easier to get started than VPS?

Depends on what you're doing. I've been pretty find of Digital Ocean, but there's lots of smaller or in between options.

If I'm targeting AWS, Azure, etc. Then I'm going to leverage their services in a lot of cases. Wether it's AWS Dynamo DB, or hosted PostgreSQL, there are places you will save in either resources or personnel.

A set of Dokku servers behind a load balancer can go a long way. Kubernetes further still. Taking the issue of DB management, static assets or backups goes a long way to freeing up time to make solutions without buying into lockin too much.

Good question. It seems there is far more to learn for each different cloud provider than buy a vps and i. e. just learn how to install postgresql on Linux and run your application.

it also depends on your usecase. if you are very bandwidth outbound heavy, yes aws is crazy.

Reply on news.ycombinator.com