Saving on egress switching from AWS to Hetzner | Hacker News Reader

Saving on egress switching from AWS to Hetzner

blog.fleetdm.com

326 pointszwass207 comments

It's the same story every day, the same pattern. And it's the same because it works.

Startup: use <aws|azure|gcp> because it's quick to get started. It's easy to scale it up 100X in 5 minutes for when you make it big. And it provides a long list of easy to use services that are overpriced per use, but which free you to focus on your business rather than complex DevOps.

Then once you scale to the point that you're comparing your salaries and your ops cost, you find smaller services to handle the expensive part of your business. Ones that do exactly what you need, not much else, and charge accordingly. Maybe they're self-operated, or maybe they're other companies with a narrow focus.

Then you write a blog post about how everyone is crazy to use <cloud> since it's so expensive, and you don't know why you ever thought they were a good idea.

Indeed. I don't even see this as a bad thing - it means you pick the best option given your shifting criteria which is what we should be doing. Blindly doing one or the other makes no sense. I don't particularly enjoy the blog posts that are hyperbolic (not necessarily this one) - both AWS and other things can both be simultaneously a good choice for any given company.

But moving off of the cloud is not always realistic. If the start-up leveraged cloud-specific services, they are now locked in. If they remained cloud-agnostic, they were under-utilizing the cloud.

How many companies actually shift their infra vendor?

Yep, and most likely the service they switched to is running at a loss until they get enough customers, and then will start uping their costs, continuing the cycle (either migrate for cost savings, or stay)!

Same pattern on HN too - a post about some outrageous bill due to a configuration error or whatever, followed shortly by a post like this one.

I mean it makes sense in a way and I'm not saying it is a bad thing, just a pattern I've noticed.

The third step of this pattern is that vantage company popping up in the comments. We should get bingo cards made

E: hehe there it is, they’re already here

Might want to include a relevant link in your comment? Where is said vantage company in the comments?

The commenter was correct, you can find the appearance via searching 'vantage' with the Find in Page browser feature. Direct link to comment: https://news.ycombinator.com/item?id=30066823

Honestly I decided against linking it in case people decided to go mass downvote it or something dumb

Seeing some of the replies to tweets that appear here I’ve lost faith in the HN audience’s ability to observe and discuss

Pattern, not problem :)

And then there's someone like you who detects that we have a pattern here and that it's repeating itself.

In actuality, how many startups have switched from <cloud> to an alternative setup? Truth of the matter is that once a startup gets to a certain size, they use these to lock in savings versus moving to bespoke infrastructure: https://www.parkmycloud.com/blog/aws-edp/

A lot of startups perpetually run on VC money and don't particularly care about becoming profitable. But those that do make it sometimes shift at least some of their infrastructure - see Dropbox or Netflix (with edge caches), Cloudflare, etc.

It probably depends a lot on whether the startup is selling a service or collecting eyeballs to sell to advertisers. Cloud could be a lot more efficient (all told) for the former than the latter.

Correct except in the vast majority of cases it still isn't worth it for the business to switch to the smaller, specialized service because the operational costs of rewriting/training/managing multiple clouds are too high.

A better starting point would be a vps like DigitalOcean or Linode or Vultr. These days I’m not recommending anyone to use “cloud” (aws/azure/gcp) unless they have a specific need.

Hetzner provides exactly that: a VPS. They're also cheaper than DigitalOcean, I assume, because EU web hosting is way more competitive and US VPS offerings have traditionally been way overpriced.

Let's also be realistic here: CloudFront is a good solution if you need a file to be served over CDN with low latency in multiple regions. Raw s3 or just a web server is much cheaper if you don't need low latency over multiple regions.

>EU web hosting is way more competitive and US VPS offerings have traditionally been way overpriced

US offerings being way more overpriced than EU alternatives also boils down to the much larger capital/war-chest US tech companies have at their disposal to spend on Ops to conquer a market, a fact reflect in those prices but also in things like dev salaries, office equipment, etc.

EU offerings are better priced because most EU companies, who aren't bankrolled by some mega VC fund, cannot afford to pay US levels of pricing.

For what I've seen recently, it feels like hosting might be cheaper in EU than in US, but housing/colocation is so much cheaper in the US.

I think OCI has pretty damn good offering that sit between DO and AWS. Especially when you can mix and match your CPU / Mem Config.

There are two numbers that matter: days until the product is ready, profitable; days until the money runs out. Startups make decisions to improve the relationship between those two numbers.

Using Cloud services that are obviously more expensive than self-hosted options reduces both numbers- less time to launch stuff, but less money. The point is that in early-stage startups, the costs are so low that paying an order of magnitude more for Operations may not move the second number by very much at all, since most of the money is paying for developers.

it also depends on your usecase. if you are very bandwidth outbound heavy, yes aws is crazy.

One strategy has a high fixed cost and a low variable cost. The other has a low fixed cost and a high variable cost. Neither dominates - even for the same firm over the course of a year.

Hetzner has no fixed cost. It's billed just like AWS EC2 - just (a lot) cheaper.

Hetzner is fixed cost. You add things, click checkout, get resources, and your bill stays the same until you change something.

Depends on who is "you". When I provide a Kubernetes cluster on the Hetzner Cloud I usually need to provision dynamic storage as well - that gets allocated through developer's deployments. Also provisioning VMs for autoscaling can be automated. Of course I can lock everything down but then it's no longer a "cloud".

Lol you wish. My genius corporate employer has whole data centers with cheap paid-for servers, but thinks moving to the cloud will result in better software because that is what all the startups do. duh

They might not be wrong. In govt at least it was 6+ months to get a single rack of machines provisioned / assignees if you were lucky.

The machines were there fully paid. But w aws I’d be up in 10 minutes w a lot less arguing . I’m convinced this might be part of AWS edge in big players

Same thing in a private company. 45 days to get a server + approval from a VP depending on the cost.

It can be really tough to get people out of the beige box mentality if they have never encountered anything different. The first time you blow away their production server(s) and replace them with what is checked into git can be rough.

Then you get into serverless and suddenly they can’t log into production anymore, and they have to instrument their code, and work within finite resources - all that stuff they ignored in college about manipulating datasets larger then core memory are suddenly relevant.

When we did our transition to serverless we found that the people who complained the loudest about their traditional servers being mysteriously corrupted were actually the same that had obtained admin rights “so they could work faster” and were going into the servers and blowing away configuration files, altering permissions, and making other changes. Once there was no more ssh and no more root privileges our outages dropped two thirds. Today it’s mostly third party outages impacting us and people deploying untested code to production.

We didn’t find that.

What we did find is our operational expenditure over three years tripled. So basically capital expenditure.

I’m not sure how they warped that into self congratulatory success.

The people complaining now complain that it turns out the promise of total automation was a lie. None of it is very robust and all of it is painful.

From a personal project perspective, I've maintained a few VPSes for quite a while now, but now I run them as a Docker Swarm, and deploy containers onto them - or better, I use serverless.

From a maintenance perspective, it's brilliant. Over time, cruft accumulates on my VPSes however disciplined I intend to be when I spin one up. Now, I no longer have to remember all of the ugly hacks I had to implement to get a particular new app I wanted to try out working, I just deploy and it's done.

What makes you think the process for getting a new EC2 instance won't get the same amount of red tape as a physical instance over time?

In some companies, pre paid takes a whole lot of time and paperworks, while post paid only need invoices and maybe half an hour meeting with higher ups to explain the cost.

Because you only talk to finance not IT.

Because compared to a staff of 50 doing anything the costs of an ec2 instance for a week are irrelevant?

Because finance already has AWS in system as a vendor?

Because pricing is public?

Because costs hit your budget not ITs so you have more authority?

Orgs can easily spend 5x what it would cost internally on AWS because of friction avoiding factors like this.

Yep - this exactly - except some businesses have good margin or being close to customer compute is valuable - snowflake following this advice and moving to hetzner would be value destroying in some cases

This is about levels of abstraction.

You can ask the cloud to deploy a container to Kubernetes. You can rent a VM with Hetzner. You can rent rack space in a data center. You can build your own data center. You can build your own server hardware.

Depending on the scale you operate, you'll draw the line somewhere where it makes sense to you (which may change over time).

Dev teams will still operate best when they can deploy containers to Kubernetes and without having to design their own server hardware. You now need to build abstractions in house. Maybe you can sell these to other people? Congratulations you are now a cloud provider.

> Dev teams will still operate best when they can deploy containers to Kubernetes

Jeez really? We have like, binaries deployed to machines over here.

Nobody said you can't operate with old-style VMs and ansible or puppet or whatever, but tbqh yes, you're going to move slower.

It may not be fatal to your business; it's not going to be a top 3 reason you succeed or fail! But it does make a difference.

> Nobody said you can't operate with old-style VMs and ansible or puppet or whatever, but tbqh yes, you're going to move slower.

To stay in your metaphor, moving faster is not a guarantee to arrive at your target earlier. As long as everything works fine, using k8s is great. When you have to dig deep to solve a problem, a stack with less layers might be a huge advantage.

Is everything about to move fast? When you read a book, is it a race to get to the last page? When you sing a song, is the goal to get to the last word as soon as possible?

If you'd do it for an employer, the answer to all these questions is yes.

Sounds like you moved your complexity from machine virtualization to container orchestration. Doesn't sound like you won anything, least of all agility.

Binaries are so last-century, now everyone's shipping the entire OS!

Containers can contain just a single statically linked binary. Infrastructure doesn’t care.

They practically never do, nothing that you use or would read about it assumes that you do. That makes this response a little pedantic, even though it is strictly true.

It's actually quite common in the Golang world.

Don’t come here speaking sense against a decade of marketing!

Realistically I wish someone would just write an orchestrator that runs Go processes and nginx and glues it all together.

Can't a container mostly be a Go process?

"Marketing" as you said, made this concept of "container", but I think using the words "process" and "namespaces" is still ok.

I don't agree "container" is marketing-speak; you could say it was appropriated, but I was using linux-containers in early 2000s when it was still a kernel patch. We all knew that bare metal offers a lot and our servers are underutilized most of the time, and VMs were considered a heavier option.

I still think using "container" lose the fact that it's just a process with some tweaks :)

I'm not saying "marketing" as to say it was coined by non technical people, but rather that it's a way to "sell" the feature. Like you go see your manager and say "I'm going to containerize the application" vs saying "I'm going to add some isolation settings to the process"

Consul and nomad would like a word with you.

See my post history. No thanks :)

https://news.ycombinator.com/item?id=30020427

> Realistically I wish someone would just write an orchestrator that runs Go processes and nginx and glues it all together.

Not quite what you're asking for but you can get close, on bare metal, with:

- ssh to set up a systemd service, scp to deploy your binary, then ssh again to restart your service

- dokku (containers are optional) with its cli

For the first option, once you've set up nginx on the server you can easily automate the addition of a new systemd service for a new app with a simple shell script. That done, it is a matter of less than a minute to add a new app and even quicker to update and restart an existing one.

For the second option, it is installed with a single bash command and supports Heroku build packs which means for many systems there's virtually (or exactly) no config needed. Just run the cli from within your app folder. If you do want containers they are automatically found and deployed too.

I've done most deployment methods from the simplest (above) through to complete Terraform builds, and once they are automated all that matters is that the method you choose supports your needs. Go for the simplest option that does that.

I still remember spending money for an FTP client that could “live edit” Perl CGI scripts on the server.

I deploy Kubernetes to Hetzner with https://github.com/vitobotta/hetzner-k3s

And with hetzner you can rent bare metal servers as well. It is somewhat equivalent to renting rack space, but they bring the servers, not you. The advantage is that if a server fails, they can provision a new one in less than 1 hour. The disadvantage is that you are probably running on somewhat refurbished hardware that has a higher prob of failing (at least these were the rumors in the past). 5-6 years ago, this was (by far) the most cost effective solution for Hadoop/HDFS deployments. IDK what the status quo is now.

As others have pointed out it is relatively easy to deploy small k3s clusters on Hetzner Cloud.

> The advantage is that if a server fails, they can provision a new one in less than 1 hour.

I can confirm that. I've been running a few servers with them and my experience is that the older ones had a failure every 2-3 years. Usually it was a disk, sometimes the power supply. Once 2 of 3 disks failed but I managed to rebuild the array (it was the HDD era so it took several hours), no data needed to be restored from backup.

But since 2018 I had no failures at all. And I'm happy to see their ecosystem grow.

Is aws|azure|gcp really easier to get started than VPS?

Depends on what you're doing. I've been pretty find of Digital Ocean, but there's lots of smaller or in between options.

If I'm targeting AWS, Azure, etc. Then I'm going to leverage their services in a lot of cases. Wether it's AWS Dynamo DB, or hosted PostgreSQL, there are places you will save in either resources or personnel.

A set of Dokku servers behind a load balancer can go a long way. Kubernetes further still. Taking the issue of DB management, static assets or backups goes a long way to freeing up time to make solutions without buying into lockin too much.

Good question. It seems there is far more to learn for each different cloud provider than buy a vps and i. e. just learn how to install postgresql on Linux and run your application.

I find it easier to deploy Docker inages to VPSs than to learn how to use countless AWS services. My go-to combo for a new project is DigitalOcean and Cloudflare.

If we're talking about backup and recovery strategy, cloud is undoubtedly easier.

Setup one that doesn't eat your wallet is what makes it hard.

I'm not sure why people try to revise their logic from the past. It probably made a lot of sense to start with a cloud provider. It made sense to move the critical parts out as the company grew. They were all completely valid choices.

Exactly. In the devops journey cloud cost optimization comes after deployment maturity.

Granted, thinking about this up front is ideal, but it’s better to start an mvp and build from there than to be paralized by perfection at the outset and not build anything at all.

For those in the Bay Area who are a bit more adventurous, Hurricane Electric charges only $400/mo for a full rack in their Fremont datacenter, along with 1Gbps of bandwidth, and 12A of usable power.

It’s totally overkill for the article’s use case, but it enables many cool things that need sustained bandwidth, and has been pretty fun.

That is a surprisingly small amount of bandwidth for such a large amount of space. A single machine in the rack could easily saturate the network.

Same with 12A of power. It's just the base offering. It's nice to get a full rack though b/c you can be more flexible with organization, have some (physical) storage space for spares, console, accessories, etc.

If you can get three in a row you can have a place to sleep for $1200/mo.

You can't sleep standing?

Of course, but you have to budget space for your closet

You could be on to something. Start a company that sells rackmount kitchen appliances.

If you have trouble sleeping with the fan noise, the CO2 fire suppressant will put you right to sleep :)

> If you can get three in a row you can have a place to sleep for $1200/mo.

Remember to set them crypto mining so you're kept warm at night.

What's a full rack size? 42U?

Also agreed with sibling comment that 1Gbps is pretty slow nowadays. Do you know if that can be customized?

There are several carriers in the building that offer $200/mo/Gbps or cheaper; HE has pretty competitive additional bandwidth as well.

(And yeah, 42U.)

This is 1gbps transit so not that low considering you will need a cdn anyway and can also peer

They quoted me $900/month for a 10G port with a 3 year term at a third party data center. I'm certain you can get that deal or better as a customer in their facility.

Isn't 12A kind of small for an entire rack?

12A is small. We used to colocate there back around 2010 and at the time could fit anywhere from 6-10 servers depending on load.

We used the extra space for spare parts.

Yes. The power and Internet are always the expensive parts of any colo. That is probably about 4-10 servers depending on how you spec them.

How do you deal with planned downtime / maintenance / all those nice things that clouds can manage for you?

You get multiple ones and failover to them or you schedule a maintenance window. Yes, it'll be more expensive, but you probably already have a good reason & the associated budget if you colocate as opposed to renting from Hetzner/OVH/etc.

Is this so different from the cloud? Even an EC2 instance should be assumed to be a single point of failure and you also need multiple (preferably in different AZs or regions) if uptime is important.

Depends on the type of load.

Old and boring things, DNS, load balancer, yada-yada.

If you save thousand(s) it is still could be cheaper.

Not sure what you are talking about here but for example an RDS instance is simply an EC2 instance with a bunch of automation.

If you want to save some money write the scripts yourself and run them in a cron job. Took me a couple hours and I'm no sysadmin.

To me the best thing to do is move all your services into k8s and automate you DB (use open source). Then you can move to any provider when you realise the cloud pricing is shit.

Hurricane electric are one of those companies that have been around since the dawn on the internet, I usually run into them when I want to look at BGP information and stumble upon their BGP toolkit.

I went to their home page to try and find more information about colo pricing, but it's all hidden behind a contact form.

Their site design seems very 1998, makes me feel really nostalgic.

I still use their telnet looking glass service from time to time.

Now THAT sounds fun!

> Hurricane Electric charges only $400/mo for a full rack in their Fremont datacenter, along with 1Gbps of bandwidth, and 12A of usable power.

Wonder what if Netflix turns around, and blue oceans [0] its video delivery infrastructure [1][2], like Cloudflare did for text/html. We do know that video has all but eaten up the attention that text/html once had on the Internet [3]

[0] https://en.wikipedia.org/wiki/Blue_Ocean_Strategy#Concept

[1] https://news.ycombinator.com/item?id=28584738

[2] https://openconnect.netflix.com/en/

[3] https://www.wired.com/2010/08/ff-webrip/

This could be an interesting competitor to Mux (https://mux.com/)

There are some fundamental constraints there for both cases you mentioned, but more so Netflix. Their business model and physical infrastructure are structured around a small, popular, catalog. This enables dense storage utilization at the edge, optimizations like pushing content during troughs, etc. Because of that they're edge is generally optimized for a small number of bytes stored and a very large number of bytes served to clients.

Storing a large number of "cold" or "long tail" bytes is a different physical problem when it comes to storage and IO. And that's also the exact problem you have when hosting lots of third party content (or apps, or lambdas, or anything else). There you're also optimizing for bytes stored:served, but have to place it on cheaper media and with less duplication to make the physical space/power/dollar constraints work.

They certainly could change their business model and infra. But then they end up looking a lot more like a commercial CDN with hierarchal stores, more centralized POPs at IXPs, etc.

Disclaimer: Principal at AWS. Used to work on Amazon CloudFront and a bit of time in AWS Elemental, served a lot of video & streaming bits. Everything above is public domain knowledge.

If you mean selling their video delivery infrastructure as a commercial offering, there isn't much Netflix can do that isn't already covered by YouTube, which has a massive head start to boot.

I actually asked about this 3-4 years ago and finally got an answer from [1] jedberg @Netflix. Basically they are not interested. I do wonder if they think the same now that their subscribers base has somewhat saturated and are looking for additional revenue.

[1] https://news.ycombinator.com/item?id=18954868

It’s still a 5x savings over AWS!

I think you missed a zero there

$400/mo 42u at what power? ;)

I am always on the lookout for dedicated server providers. However the minute I see the site not having configs / procing info I go elsewhere. The last thing I want is waste my time on negotiations.

I've been using https://www.hivelocity.net for years for my dedicated server provider. There's been an insane amount of consolidation in that space in the last few years. Independent operators are getting harder and harder to find.

$400/mo for the first term. Then it basically doubles. I inquired about it:

> For example, with a 3-year term for a 42U cabinet, it would be $400 a month for 3 years, and that includes 1G already. After that, for another 3 years, would it be $340/mo for 1G, plus $500/mo for the 42U cabinet, which is $840/mo (more than double)? In both cases, is a /27 always included with the 1G port (pending justification)?

> Yes, you are understanding what our standard rates are without the promotion. Yes, we include the /27 with service. Our promotion is currently our most cost effective solution. Let me know if you have any questions.

Do you really want to build a whole 42U rack full of big heavy servers and have your price doubled after your contract expires?

I have no inside knowledge on HE pricing, but I had my cabinet on a 1-year term and it is the same price today even though the term lapsed and I am just month-to-month.

It is certainly possible that the rate increases at some point, but I don't see why either HE or you would want to commit to anything beyond 3 years from now?

I think you're looking at this the wrong way. You are receiving a > 50% discount for 3 years after which you pay the non-discounted price. If you'd prefer to pay full price from the start, I'm sure they'd be happy to let you.

The non-discounted price still isn't outrageous (I think?), but it's disingenuous to advertise a temporary discount as the price of a service. Unless you intend to only use it for 3 years, I guess. Maybe the cost savings do justify that.

At 12A, you're probably building a whole 8U of servers with 34U of airflow baffles.

Beware that Hetzner is quick at blocking IPs, with zero recourse or explanation.

If they receive any kind of abuse report, they instantly put you on a 24h timer, but they won't show you contents of that message. If you do not respond in time, they will simply block the IP. What's strange is that I was using Cloudflare in front of Hetzner, but it seems that Cloudflare is ratting their users out and forwarding any abuse mail to the owners of "cloud protected" IPs, all without any notification or warning. Hetzner then sends you an email like "Please remove <sitename> from our network within the next 24 hours. This site violates 6.2 of Hetzner's terms and conditions.", where site in question is an obvious mirror/search engine of another social network.

What's perhaps more irritating is that Hetzner Cloud has a tiny LIFO queue of ipv4 addresses unique per customer, so you will get the same IP as soon as you release one. You can imagine what happens to a Rancher-managed k8s cluster with Hetzner node driver when one of those IPs gets a no-route: it will get stuck trying to re-deploy a failed node. Mine eventually fell apart though because I did not attend to it for a couple of years.

I'm now hosting from a proverbial garage (NixOS and cron) but proxying traffic through a cheap cloud VM, which saves me at least $1500/mo.

I've done tons of sketchy shit (security research) from lots of different providers and never seen this type of response.

Whatever you (or your customer) were doing likely hit the point where they assumed the police would be by for a chat.

Absolutely; every host does review abuse complaints because there's a lot of fake garbage out there.

For this kind of response, you're probably looking at (1) extremist material or (2) CSAM.

> For this kind of response, you're probably looking at (1) extremist material or (2) CSAM.

Both of which can be found in copious amounts on Reddit and Twitter, and there are multiple services for these where the description of "search engine / mirror for a social network" applies.

Additionally, Hetzner is a German company. Our laws are very strict regarding Nazi content, and absurdly ridiculous when it comes to CSAM - about three quarters of what you can find on fanfiction sites is theoretically illegal here (§184c StGB).

Watch out that electricity costs went up significantly in Europe in the past months. I am a happy hetzner customer since 2012 I think. Their server monthly costs were just a bit higher than my electricity bill would've been if I were to keep a machine up 24/7 at home, so I rented a machine from them having the hardware basically for free.

Last week I got an email from them saying my server cost would increase from 30eur/month to 48eur/month. I can take afford the increase, but I wonder how many companies with infra bills of tens or hundreds of thousands will be able to afford a 60% increase in hosting costs.

Great article. I wrote about a similar experience 9 months ago while switching from Google Cloud Storage to Scaleway Object Storage:

https://pellegrino.link/2021/05/31/scaleway-object-storage-s...

Similarly, I saved about x100. For sure, the services do not offer the same level of guarantees and speed but depending on my case it was worth a migration.

Looking forward to trying Cloudflare R2 soon!

Thank you for sharing! We are also interested in R2, but it sounded like it would not be available for quite some time when we spoke to Cloudflare.

Depending on the size of the objects, one could use Workers KV ($5 per 1M writes, $5 per 10M reads, 25M size limit) as a pseudo-R2 store [0] via Workers Unbound (~$1 per 6M requests), especially with the recent optimizations bringing the bills (of Workers Unbound proxying KV) down by 70% [1]

[0] https://news.ycombinator.com/item?id=28747798

[1] https://blog.cloudflare.com/workers-optimization-reduces-you...

Yeah +1 for scaleway been using them for years and they are great.

One thing Hetzner doesn’t have yet (but it SHOULD!) is Object Storage, and it’s something I’m working on over at NimbusWS[0].

Another awesome thing about Hetzner is that bandwidth internally is free and automatically negotiated (i.e if you send traffic to a Hetzner IP it will flow internally).

One of the hardest things is being able to tell the bandwidth usage and when it’s inside/outside Hetzner.

[0]: https://nimbusws.com

Scaleway has object storage: https://www.scaleway.com/en/pricing/

Scaleway has awful customer support, beware.

Yes, run.

Everyone has (relatively) poor support if you are a budget customer. Scaleway support replied to all my tickets and was professional. Not a 1 hour response but can’t complain for a €5/mo. Been with them for ca 4 years since the ARM days until recently.

Scaleway and Hetzner are cool. What I'd love to see is a lower-cost (if slightly higher-friction?) version of AWS as it was early on. Some sort of analogues for:

- S3

- VPC

- Compute instances (types don't have to bee too fine grained)

- SQS, SNS

- Some sort of Dynamo and/or RDS functionality.

- Some basic API coordinators, I guess Teraform has providers for lots of stuff these days.

Lambda- and fargate-like things would be a plus, but not strictly necessary.

I feel like 90% of the projects I've ever built could be easily made and scaled with nothing else. Further features hit diminishing returns really fast, and serve to muddy the waters up around what tooling is best, or even what exists.

Digital Ocean has S3, VPC, Compute, managed databases, some apis.

OVH Cloud might have more of your list.

Though, neither has all of it, and some portions have bad reviews, like DO's object storage is apparently slowish and has other issues.

Have you actually used Scaleway? It has more than half the things on your list.

Note there is a huge difference in costs between these providers cloud offerings and dedicated servers.

Yes, but if you factor in the traffic costs, it still comes out 10x cheaper than AWS &co. Plus, the parent was not asking about the cost vs bare metal.

Noted! So general compute is something I want to add to Nimbus but given how thorny it can be, I'm thinking of going for splitting into very specific use cases:

- Run a container (you give me a container, app.json or a standardized format, and I run it)

- Give me a bundle of files (static hosting), or specify a bucket

- Give me a single function and I'll run it, optionally hooking up a managed domain name through us to it)

Notes on the other things:

- VPC this is actually a bit more complicated, but possible

- SQS/SNS => NATS/Kafka is this enough?

- Dynamo/RDS => Managed Postgres +/- extensions like Citus and Timescale for scale

- Some basic API coordinators => Probably will not do but writing a TF provider is probably far in the future

Paradoxically Lambda and Fargate-like functionality is like one of the easiest to deploy and manage (standing on giants like OpenFaaS/Fission/etc), and would strike me as the easiest for people to actually get started with if I could roll it up with managed DNS.

It has (sort of): https://www.hetzner.com/storage/storage-box

Which is more versatile than an object storage.

Oh yeah I'm aware of the storage boxes! They're actually fantastic, and what I plan on building on for Nimbus -- it's not quite as easy as just running MinIO (in fact I won't be using MinIO), but the efficiency is great.

Traditional object storage scales automatically and is very easy to integrate with (most people write apps to interface with S3 these days, and less SFTP), so there's just that small edge! That "sort of" is what I want to get rid of.

Other smaller clouds have services that are a better fit (OVH has object storage, Scaleway, etc), but Hetzner doesn't yet.

I mount my storage boxes as CIFS volumes, but random access latency is high, and I can't do multiple heavy IO operations concurrently. It's best suited for use as a backup drive.

Is there a faster way to mount storage boxes? I suppose SSHFS would be even slower.

At least you can tune sshfs performance with a few options.

https://www.admin-magazine.com/HPC/Articles/Sharing-Data-wit...

Are you using Ceph for object storage?

I'd be super interested in hearing from someone who has set it up whether it's something an experienced but generalized sys admin could build, or if you need an expert in the matter.

My suggestion from a company that failed to set it up properly, is to find an expert.

As someone who worked at Red Hat, and has setup Ceph Clusters.

Get an expert, or a consultant. You'd need a really experienced storage admin at the least, and possibly more.

Go with RookCeph if you can. Still challenge to maintain

Yup, Ceph is going to be the solution to power it. There are quite a few options in the space but I was basically down to Swift (powers OVH) and Ceph (powers Digital Ocean Spaces) and OpenIO (power's OVH's new experimental offering).

Ceph has much more public resources available (SwiftStack recently got acquired, seemed like they had most of the knowledge in the space), and I've actually set it up a few times now thanks to the excellent Rook[0].

It's definitely a lot easier to set up with Kubernetes (the tradeoff being you need to understand Kubernetes), but it's definitely manageable for a generalized sys admin (albeit one with a bit more experience). I've written about the process:

https://www.google.com/search?q=site%3Avadosware.io+ceph

[0]: https://rook.io

I am in the process of migrating off of Rook Ceph after using it in production for two years. Setting it up is easy thanks to Rook, but wait until Ceph gets under load, then the real fun begins. If you only need object storage, I suggest looking into SeaweedFS[0]. It's a far more lightweight and performant solution.

[0]: https://github.com/chrislusf/seaweedfs/

Thanks for the suggestion -- I'm definitely aware of SeaweedFS and it was actually a really strong contender but I didn't choose it (and didn't mention it) for a couple reasons:

- Some sharp corner cases are definitely out there (issues/bug reports)

- Supported APIs aren't quite as extensive as the other options yet

- The requirements/expectations for

There's also some previous discussion from 2020[0] that was interesting. I actually planned to use SeaweedFS and dip my toes with what I'm calling the "CloseCache" feature -- on-demand nearby proxies for the data that's really in your object storage. The idea was to take advantage of seaweed's excellent proxying features and kick the tires at the same time.

Somewhat off topic but I'd love to pick your brain, would you mind if I sent you an email?

[0]: https://news.ycombinator.com/item?id=24716319

How would you define object storage vs. block storage? How do I know which one I need? This question is coming from a web dev who mostly uses databases on the backend so I'm unclear what constitutes an "object" in this case.

(Yes, I googled it, but I'm looking for a more practical example)

Block storage is usually for putting a file system on top of. Object storage is kind of like an abstraction of a file system itself.

You would use block storage more for a file system mount as a raw device (virtual).

There's overlap and many platforms support both close to interchangeably.

This is pretty spot on, so there's not much to add but a some somewhat disorganized thoughts:

Object storage and block storage are similar -- the difference is usually how they're accessed but isn't necessarily (ex. s3 via FUSE projects can be thought of as block storage). Some examples of the blurring of this line:

- https://github.com/minio/minfs

- https://github.com/archiecobbs/s3backer

- https://github.com/yandex-cloud/geesefs

Hard drives are in the business of "block storage" -- you offer them a block of bytes (if you wanted to you could tell the hard drive exactly where to write the bytes, rather than using the file-based interfaces that are common), they put it on some sort of media (rotational, solid state, whatever), end of story.

Applications often only need a higher abstraction on storage -- not bits and bytes, but instead files or "objects". Most applications don't need the ability to access one or more bytes (in... a block) of an area on disk, they often only want access to entire files (roughly a 1:1 mapping with objects, with how 99% of the people use object storage).

Getting back to the question, block storage is usually interacted with via a completely different set of technologies -- iSCSI[0][1], NVMeOF[2], etc. The idea here is that you're talking to a hard drive on the other end, and it make sense to speak the language (or a language) of hard drives. Object storage is normally interacted with via HTTP. The expected/tolerated latencies using these different technologies are different orders of magnitude.

To rehash, object storage is similar, but seeks to up the level of abstraction and give you access to an consistent but opaque access to files. How is the object storage storing your files? You don't know and you don't care (probably) -- what you care about is how fast your HTTP request returns (or doesn't). This interface here is similar enough to reading and writing files locally but more importantly enables multiple applications to share the same storage without sharing the same hard drive. You can also write certain slices to files as well (this requires some more complicated locking and what not just like it would do locally).

I want to also note that there's a conceptual step between "traditional" block storage and "new age" object storage that others might skip over, that's distributed storage systems like NFS. It's NFS's job to present a file system that when changed, prompts a file system on a completely different machine to change in the same fashion. It's easy to imagine a simple way to make functionality work but of course reality is more complicated. Object Storage arises from the realization that you don't actually have to interact with a "fake"/shimmed version of the local filesystem that appears local but is actually remote -- you can just send a HTTP request over the internet to a machine asking for the file you want when you want it.

Here's a fun thing to think about -- is a database an implementation of object storage? You normally don't ship bytes to databases, you ship records (which you happen to decide the structure of) -- records are closer to files conceptually than they are to a "block" of bytes, even though at the end of the day the digital storage we're referring to is going to be bytes on a storage medium somewhere.

If you really want to get a good instinct for this, dedicate some time to skimming/reading through the Ceph documentation and you'll see how they layer the object storage (and other things) on top of a lower level "block" management[3]. The picture on the first page should be quite instructive.

[0]: https://www.truenas.com/docs/core/sharing/iscsi/

[1]: https://stonefly.com/blog/what-is-internet-small-computer-sy...

[2]: https://blog.westerndigital.com/nvme-of-explained/

[3]: https://docs.ceph.com/en/latest/architecture/

Hey check out this repsonse: https://news.ycombinator.com/item?id=30068565

I work at Microsoft (not on Azure) and run Tor relays. My Tor Exit hosts are Psychz Networks, Terrahost, and OVH, along with a middle relay on CenturyLink Fiber.

Why? Azure egress bandwidth is too expensive for a Tor relay, even for a well-paid software engineer like me. I have ~1.5 Gbps Tor bandwidth. If I were to put that on Azure, I would be paying a mortgage worth of bandwidth fees. Same can apply to AWS and Google Cloud.

However, Big Clouds charge more for bandwidth because (a) they're a leader and (b) they have paid peering deals with Big Telecom (e.g. AT&T, Deutsche Telekom, Telefonica, etc.). Smaller hosts like Hetzner don't peer with those ISPs and rely on "transit", but that also means slightly worse performance to Big Telecom.

In short, big clouds can use their market power and advanced technology to command a premium, whereas smaller clouds have to win over customers with lower prices to make up.

Tor relays are ill-suited for Big Clouds, but then your 5000-node Kubernetes cluster is ill-suited for OVH or Hetzner.

I do use the free Azure credit for Tor Bridges, however.

> However, Big Clouds charge more for bandwidth because [...] they have paid peering deals with Big Telecom (e.g. AT&T, Deutsche Telekom, Telefonica, etc.). Smaller hosts like Hetzner don't peer with those ISPs and rely on "transit"

Wait, this seems backwards to me. I'd expect companies that can peer with Big Telecom to charge you less for bandwidth than companies that have to pay for transit. What am I missing?

They aren't paying more, they're charging more.

I'm not sure about the others, but as a Deutsche Telekom customer I have discovered that they actually want to be paid for peering deals through occasionally extremely slow connections.

Nice web service you've got there...would be a shame if it was slow for a lot of your users, right?

I still can't believe that there is no regulation in place to prevent this sort of racketeering.

> This came out to about a 2x savings over AWS.

Is that a typical way to describe cost savings? I would have said 1/2 the cost of AWS. It almost sounds like cloudflare would pay them money (e.g. you go to the store and see something marked 200% off, so you get paid to buy it, like when oil prices went negative).

Related: I've always found the 2x, 10x, etc notation confusing. Does a 2x improvement mean a 100% increase (e.g. 10 -> 20) or a 200% increase (e.g. 10 -> 30)? Would someone ever say a 1x improvement or a 0.5x improvement?

It's funny how your first paragraph makes it clear you're not confused about this at all.

2x savings means you need to double the new price to get the old

10x savings means multiple new price by 10x to get old

59x savings...multiply by 59.

Yeah, I could guess based on context. I would prefer notation that's clear on its own without needing to look at context. "1/2 the price" is clear on its own.

And it still makes me feel nervous that 3x improvement = 200% increase. It seems to me like a big opportunity for misunderstanding.

Thank you so much for saying this! It has always bothered me

1/59th the price isn't as clear.

If even a person asking how it works knew how it worked, how confusing can it be?

I think 1/59th the price is clearer than 59x cheaper. In 59x, what is the x for? Multiplication? What is being multiplied?

"1/59th the price of AWS" is clear because you take the price of AWS and multiply it by 1/59. Probably more clear would be "1.7% the price of AWS".

When I think there's an opportunity for misunderstanding, I point it out. I don't think I'm more likely to misunderstand something than the average person, but I am more likely to speak out if I see something potentially confusing.

I would be interested in seeing a poll where we ask people to estimate "59x cheaper than $200", vs "1/59th of $200" vs "1.7% of $200". Obviously the last one is easiest to do exactly in your head, but we can still get a fair comparison by just seeing whether people are at least in the neighborhood vs way off.

I would also like to see it with 2x, 1/2 and 50%.

> DIYing it does come with additional maintenance burden, but so far we’ve found this manageable.

Tune in next month for their article on how much fun manually scaling everything is.

The reality is that most startups don't scale exponentially unless they are injecting huge amounts of cash into their company (and in that case they can afford AWS). The company in this article just has a single server.

However, most services that are around for a long time experience bursty traffic so provisioning is still difficult.

Fun fact: Hacker News runs on a single core.

Sarcasm?

"HN's main software runs on a single core", wrote a moderator when I wrote to ask about why rapid (<2s?) subsequent actions were rejected by the server (ie. like a post and then favorite it).

This was November 2020 and he mentioned a big overhaul was in the works (of both HN and Arc, the programming language in which it is written).

We've seen new features since then, but the throttling is still there so it seems to still be the same situation regarding fundamental infrastructure.

I didn't know that. thats very impressive.

It really depends on a lot of factors such as technology, amount of traffic, etc. My server handles about 200 simultaneous users on a few % of cpu. I think most startups will have a smaller load than this unless they become incredibly successful or pour tons of cash into growth.

Bare-metal is so cheap that you can turn everything to 11 to begin with, never have to worry about "scaling" and still come out ahead in terms of cost.

In fact, before the cloud craze came around, internet services used to run just fine despite autoscaling not being commonplace. For a lot of use-cases, autoscaling is a problem created by the cloud's high prices and is unnecessary if hardware is cheap enough to run continuously.

The way I look at it is that the cloud doesn't run on the cloud. It's totally artificial.

That's only true if you don't factor people hours into your costs.

The vast majority of companies have one if not more DevOps/infrastructure engineers on payroll. The "people costs" only realistically works either for super-early stage companies where the founding engineer(s) also manage the infra or something that's simple and standard enough that it can run on a fully-managed PaaS such as Heroku.

But hey if you're gonna bring up people costs, shouldn't the simpler stack (because it doesn't need to autoscale) mean less management & maintenance overhead anyway?

I'm not familiar with Hetzner at all. But it seems they are running a single self managed VM to serve these updates globally, and then comparing that to the cost of egress on a globally distributed content delivery network? What am I missing here? This is not the same thing. Nevermind the fact that AWS is an ecosystem with hundreds of products working in tandem to deliver solutions not just EC2.

I don't think you're missing anything! We didn't need 100% uptime or a globally distributed CDN. What we did need is orders of magnitude cheaper costs. So we moved this particular part of our offering to Hetzner.

I think the interest in this comes from the idea of identifying the needs and optimizing appropriately.

Yes, but AWS egress on EC2 is still 0.02c outbound from EC2 (4x savings). The egress fees are still egregious via a comparable EC2 instance. Then of course, once you factor in that AWS push "S3 via CloudFront" as the gold standard in static asset hosting it smells even more.

I feel it’s unwise to simply accept “additional maintenance burden” as a “free” cost unless it’s under an hour or two a month.

Fair enough. So far it's been under an hour or two a month! We'll see how that pans out in the longer term.

The learning curve is definitely different but once that's done, the maintenance is minimal (modern hardware is surprisingly reliable from my experience). Most maintenance you'll be doing is software-level stuff (system updates, etc) you'd also need on a cloud-hosted EC2 or similar.

It likely is under that threshold.

Also it doesn't matter. Cloud vs vps/colo/rented bare metal basically comes down to surplus of cash vs surplus of tech talent.

Any small tech focused company is gonna have more of the latter then the former, and in that situation, cloud doesn't make sense when ansible, k8s, and the like exists.

Security updates are automatic. I haven’t touched my physical servers (not even via ssh) in 6 months. The only reason I did it then was that we moved an app from Node to Go, so I was able to completely remove Node from those servers (which felt great). I think people overestimate how hard it is to run a server.

Semi-unrelated, but how was your experience moving from Node to Go? Any particular reason/requirement that prompted the change?

Surely running a physical server = like running a VM + physical maintenance requirements + any quirks of the hardware (drivers etc.)

What's interesting to me is that from my limited experience with cloud services it feels like the technical skills to set up cloud hostings and a good architecture to be just as much work as setting up dedicated servers or even vps.

Or maybe being 38 makes me really old in the way I approach things...

It's a different skill set. Cloud feels closer to developer skills, where as your VPS, VM, dedicated servers, are traditional sysadmin skills.

Funny, I see it the opposite way. Managing VPS, VM and dedicated root servers is something that a developers often have to deal with just at home, doing it with remote servers is just the same skills but applied to bigger servers, remote setting, more instances available than your personal computer, easier to congest bandwidth and so on. If you're a developer you're already a sysadmin, no matter if you use Windows, macOS or Linux. It's just a matter of depth in your sysadmin skill.

People who manage huge "cloud" installations feels more like what the traditional sysadmin role was. Everything is so different than your personal setup, you have no use of the same skills outside of that particular provider, and there are many concepts that you have to deal with that you never deal with locally.

With cloud, millennials just reinvented the mainframe. Anything new is a well forgotten old.

Configuring cloud is likely more complicated than configuring a few VPS because its configuration interface designed for large scale deployments and corporate/government needs. Cloud scales, not just in terms of compute and storage capacity but it can also address your growing org demands. For example - Protect data flowing thru the wire such that another server sitting on the same wire cannot eavesdrop. - Control access to servers and resources. - Force configuration policies. - Keep audit logs of what people do on your servers in a tamper resistant way. - Have single sign on into multiple apps - So on an so forth.

All that can be done by a competent admin/dev/devops but it takes nontrivial amount of time to setup and maintain. Having solutions integrated into your cloud of choice is a huge time saver.

https://www.cloudflare.com/bandwidth-alliance/

although I hate to shill for Cloudflare, as I believe they are evil, but anyway there you go

Why do you think Cloudflare is evil? I think they're a big net good, especially in their efforts against censorship and surveillance.

I'm reluctant to get too into it as it's a downvote magnet. This reply already should be legitimately downvoted just for being OT. Yes, they offer a magnificent service. Best of class or nearly so, for all their offerings and a decent free tier to get the mindshare. But: they want to dip their beak into all internet traffic. The power that comes with such concentration is impossible not to corrupt.

What efforts against censorship? Like how they (actively) cancelled 8chan and Daily Stormer?

I won't go into anti-surveillance much because it's too contentious. Very few recognize or want to acknowledge the downsides of 1.1.1.1.

I honestly don't want to get into it more than that in part because I've forgotten more than I used to know about their various business practices. (I used to work near to this space.) I have a vague idea that they have made centralized (power concentrating) choices in some cases, where they could have made decentralized choices, but I can't recall specific products ATM and, well, I'm only willing to invest so much time in this post. By their literal mission "to build a better internet", we can see from those choices that better is in the eye of the beholder. Just as evil has always been as defined by Google.

I don't mean that Cloudflare is willfully evil. At the time that I was intensely interested in them and formed this opinion I could find no evidence of willfulness, and their products and their operations have all the good protections. But they do get to define 'better' in their own image and in favor of their own interests.

I'm sorry to be so vague.

> What efforts against censorship?

Their efforts to roll out DNS-over-HTTPS and eSNI/ECH.

> Like how they (actively) cancelled 8chan and Daily Stormer?

I'll admit that was bad of them, but they're still way less cancel-happy than basically the entire rest of Big Tech.

[Disclaimer: I'm Co-Founder and CEO of https://www.vantage.sh - a cloud cost platform]

On this topic as its relevant, at Vantage we launched an automated cost recommendation that keeps an eye on egress out of Cloudfront and S3 and recommends when it makes financial sense to switch to another provider (in the linked blog post, Cloudflare) https://www.vantage.sh/blog/cloudflare-specific-cost-recomme... -- this has been very popular with current customers/users.

We're actually in process of adding a bunch of new infrastructure and service providers and should include Hetzner in our list of recommended providers but thought I'd mention for anyone wanting an automated approach to knowing when you hit the financial tipping point, this is one option.

Hetzner has some strange pathing issues to the US, such that using an EU vpn almost always gets you much faster speeds.

I blame level3 as that’s where it seems to go pear-shaped.

Whenever I see posts like this, I'd like to point out that both is possible at the same time.

Full on self hosting your own cloud comes usually with a lot more services than "the" cloud does because thats usually why the hoster is considered "small".

In Hetzners case you will for sure at least have to self host a decent RDBMS and it better be replicated, which means more maintenance and servers.

But you do not have to put all your eggs into one basket! There are more than enough ways to split the load and you can even add dedicated servers in the mix too!

It should not be "yep we are on azure, or yup we are on hetzner" but "my job as an architect is to combine the two for maximum value" ;)

> Having heard of Hetzner’s low egress costs (20TB free + €1.19/TB/month), we investigated what it would take to run our own server.

My understanding was that they dropped the per-TB billing a few years ago and now it's just uncapped?

Dedicated machines have uncapped bandwidth, VMs get 20TB free.

1G/s connected machines have uncapped bandwidth, 10G/s machines have a 20TB cap, then €1 /TB.

A couple of talking points come to mind:

The linked blog post mentions October 2021, while in December 2021 AWS lowered pricing for CloudFront where the first terabyte out to internet is now completely free:

https://aws.amazon.com/about-aws/whats-new/2021/11/aws-price...

For larger use cases most cloud providers have custom or private pricing options available. For example in CloudFront pricing page AWS publicly states that "Custom discounted pricing is available for customers willing to commit to a minimum of 10 TB of data transfer per month for 12 months or longer."

I have noted that most of the blog posts like the one linked above talk about migrating workloads around for savings of tens or hundreds of dollars while focusing on low level IaaS (VMs), and completely ignore the benefits of leveraging cloud services, APIs and related software ecosystems instead of DIY.

While I mostly agree with you, you should take into account that not all start ups are venture funded. Sometimes you want to test an idea and you are funding it yourself (spare time and money). Testing an idea is not a one month gig. You often have to be patient and try different things until it works or you are confident you know why it didn't.

I'd be most interested to hear about situations one step up from there, where Hetzner's per-TB-per-month pricing doesn't work out favorably and you want to serve content reasonably quickly anywhere in the world.

It seems like you could build out a CDN on OVH using 1-5Gbps unmetered+guaranteed bandwidth and place servers in the US, Europe, and a few PoPs in Asia for relatively cheap, then use GeoDNS for balancing traffic. But it seems like OVH's pricing offerings have been shifting over the last year to remove the "guaranteed bandwidth" in favor of "unmetered bandwidth" (potentially throttled 50%) on more machine types.

It would still require monitoring and managing bandwidth saturation per host, and it's unclear how much extra hassle OVH adds. But in theory it seems easier than setting up and managing colos in many countries?

It's quite easy when you only need to replace EC2. Since last year Hetzner also has a Loadbalancer service that can be used with VM instances but also in Kubernetes. It's not so easy to replace AWS RDS, though (or other services for that matter but RDS seems to be one of the more basic services).

Could someone explain what the « website assets » limitation of CloudFlare is about?

I can think of two possible reasons:

1) Website assets are generally optimized for size (images, etc.) and so reduce Cloudflare's total cost to service.

2) Cloudflare believes that free tier users who use it for their website assets are more likely to want the other Cloudflare services and therefore convert to paying customers.

For example CloudFlare's free tier is not intended to be free transit or storage for other services that are substantially about selling storage and transit. If you make an image-processing CDN type thing, like imgix, CloudFlare does not intend to subsidize almost all of your costs.

They did not mention Terraform in the article but it might be of interest for people considering a switch that there's a Terraform provider for hcloud.

What’s the best way to use generic compute to create an Serverless functions equivalent? I feel that's the best case for this sort of compute here - well, that and build jobs, heh.

i also would be very interested in this!

tl;dr Cloudflare offered a savings over AWS CloudFront, but we saved multiple orders of magnitude DIYing it on Hetzner. So far ops have not been too painful.

nice, but the article is the opposite of "too long"

This is interesting... Is there some distributed caching server one could use for this, for increased uptime? Basically some CDN.

This isn't a direct answer to your question, but our use-case is not latency sensitive (and not even particularly uptime sensitive). So far we've got near 100% uptime and if we needed more we could put multiple servers behind a load balancer. That wouldn't help much with latency -- for that I'd think you need some geographic distribution -- exactly what a CDN is good for.

Ah yep, makes perfect sense.

I saw that the article mentioned maintenance being minimal right now but at some point, you will need to hire an SRE or some such individual to handle just the Hetzner subsystem for you. Is that part of the TCO calculations?

E.g., if you’re paying someone $5-7000/month (no idea what an SRE costs) to manage a fleet of say 10 subsystems, their salary should factor into the cost calculations, right?

But you also need to hire someone to manage the AWS infrastructure.

And the need to use-only-as-needed creates additional burden that doesn't exist with cheaper monthly-paid hardware.

The answer might very well be "it will take less time to manage AWS", but it's not clear. Some might need a pay-stack to handle the cloud's pricing.

tl;dr the title was enough. I am so surprised by how this somewhat normal post (we got burned on aws cryptic billing model) makes it on the 1st place - is like people discovering warm water. This is not high-tech/rocket technology/science.

Bragging about switching to Hetzner always reminds me of my friend bragging about buying a $500 honda civic. What a great deal! He soon had a much larger tool chest and oil-stained jeans, and learned a lot more about cars :)

Inevitable conversation:

“Why doesn’t everyone do this?”

“I did the other thing so I can do … other things.”

I think it's the other way around. People like to brag about their complex AWS setup and how easy it is... well, except for this lifecycle policy, and you need to edit this XML, and yeah, this limit on VM types is not documented you have to request it be lifted, and you did remember to whitelist your IP range so you can log in right, and yes I know you'd think you could serve that file but the CORS headers need to be set in the AWS console, and redirects have to be uploaded with a special config option in s3cmd it's not just a list of regexps and...

Meanwhile on my Hetzner, I log in, tweak stuff directly, and it works. My static site wound up being much easier and more comprehensible when I dropped all the AWS stuff. I don't miss AWS one bit.

Reply on news.ycombinator.com