I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers.
I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments go missing with no automatic recovery. I didn’t realize this was happening to a number of my services until I started monitoring with 3rd party tools, and it became evident that I really couldn’t rely on them.
It’s a shame because I do like a lot of other things about them. Even for hobby work it didn’t seem worth the trouble. With digitalocean, everything “just works”. There’s no free tier, but the lower end of pricing means I can run several Go apps off of the same droplet for less than the price of a latte. It’s worth the sanity.
collect some evidence, maybe someone wants to do something about it.
This is the one thing I need in every app and don't want to do myself.
The main app is node/express running on Digital Ocean and it connects to directly to the Supabase hosted Postgres for most operations, but then uses the Supabase auth API for auth related stuff.
Saves a lot of time sending password reset emails etc and the entire project costs less than $5/mo in hosting costs.
I don't like self hosting anything that requires its own process. And if I did decide to self host I would choose a more mature project.
This is a very young one man project delegating the heavy lifting to another one man project. And it doesn't appear to support social logins.
Part of what inspired me to give fly.io a shot was that I didn’t love the monorepo deployment story on the app platform. Fly doesn’t have a solution to that, but I suppose I felt less tied to DO at the time because I wasn’t totally content anyways. I’ve discovered since then that I was actually doing it wrong, so I’m way happier. I’m pretty big on monorepos so their whole system fits my workflow remarkably well now.
I’d like to figure out how to prevent deployments when my code doesn’t change in one app, but does in another. At the moment, pushing anything at all will trigger all apps to rebuild and deploy again. Not a huge deal and several orders of magnitude less painful than not being able to deploy at all, haha.
I know that’s probably pretty easy for many, but I was pretty new to k8s and it felt like magic.
Things like they don't give you the postgres root user on their managed postgres. And I ran into issues trying to capture the deployments in code. Their terraform providers are pretty good, but still leave something to be desired. For all its many warts, I'm much happier back on AWS. It did end up more expensive, but it's worth it for the fine grained control in my case.
But I spent the last 5 years as a DevOps/SRE, so... uh... I'm picky.
I’m nowhere near as picky as you are, but maybe I’ll need to be at some point. As it is I mostly just build stuff and send it to the internet. If it builds and it does what I expected, I’m pretty happy! I don’t often need anything too special.
Higher number of outages at Vultr over 5 years, but none longer than a few hours. I can’t remember the last DO outage lasting more than a few minutes.
Experienced a Vultr routing problem that lasted several hours; they communicated about it, but it was still a long time to fix.
DO once did an auto-migration of a server to another cluster with an attendant outage that lasted a few minutes at most. No IP changes, completely transparent.
You boot one up in less than 30 seconds, and get ssh access to it almost immediately. It's very BS-free.
I think fly’s tooling feels better than doctl, but the infrastructure is incomparable at the end of the day. doctl has improved over time too, and with added pressure from newcomers I don’t doubt that it’ll continue to improve.
Additionally, you still keep the full ssh access to the machine if you ever need it.
Contact is in my profile but I'd love to have some more people kick the tires and tell me what they want built the most.
They don't run on AWS. Not sure what sort of rumors are running :(
> data centers from?
The major players e.g. Equinix, Coresite, etc. Varies per location. Even AWS don't build most of their data centers.
I feel like there might be more to it, especially considering the situation with electricity prices in some places in EU recently.
I used (and still use) a Lithuanian platform called Time4VPS which was cheaper than Hetzner previously, yet had to increase their prices somewhat for that reason. Now only some of their plans are competitive with Hetzner, while Hetzner also provides some managed services as well.
Hetzner docs also went into some of the details regarding the pricing: https://docs.hetzner.com/robot/general/pricing/hetzner-prici...
And yet, I can't help but to wonder why they don't give in to the desire to maximize profit margins, like happened to say Scaleway (good platform, but as expensive as DigitalOcean).
Maybe that's not actually the dominant cost, or they've optimized everything else so well they can just eat the electric bill.
like the longtime workhorse was a high performance skylake desktop cpu w/o ecc ram
We used to use nianet to house our hardware in Denmark. Basically these companies does hardware renting and they also do hardware renting with more steps which is where you rent rack space but own the hardware. They provide the place for the hardware and they also have multiple locations so that you have both backup and redundancy, and while it doesn’t scale globally in 20 years I’ve literally never worked on anything that needed to beyond having some buffer caches for clients logging in on their vacations or something like that.
What Hetzner seems to be doing with the DO styled hosting, and this is just a guess, is that they are one or the many EU companies preparing for the big EU exodus from the non-EU cloud. Which is frankly a solid bet these days where both AWS and Azure are increasing prices and are becoming more and more unusable because of EU legislation. Part of this is privacy which Microsoft and Amazon are great with in terms of compliance, but part of it is also national security. I work in an investment bank that builds solar plants, since finance and energy are both critical sectors we risk being told that half of the finance/energy companies in the world can’t use Microsoft because the EU seems it as a single point of failure if our entire energy sector relies on Azure. Which is sort of reasonable right? But what this means for us is that we can’t vendor lock-in, not really, because we need to have up-to-date exit strategies for how we plan on being fully operation a month after leaving Azure. Which is easy when you just containerise everything and run it in VMs or similar, and really annoying if you go full in on things like AKS. Which doesn’t help our Azure costs.
Anyway, right now we are planning on leaving Azure because of cost. Not today, not next week but sometime in the next 5-10 years and a lot of these EU cloud alternatives that actually operate the hardware instead of renting it are likely going to be a very realistic alternative. And that is the private sector, I spend time in the EU public sector which is a massive amount of money and I’m guessing it’ll leave both AWS and Azure by 2050. Some of these EU cloud initiatives is going to explode when that happens, and right now, hetzner is one of the best bets.
To get back to your question, DO rents server space. I have no idea where they’d rent it in Germany but they could potentially be renting it from Hetzner.
You do the same on the other side of the table. Companies like Hetzner knows that EU cloud sollutions are likely to see growth, so it's only natural that they invest in the tech to put themselves in a prime position to jump on the opportunity. Selling a good product while you do so is the way I would do it personally, but you also have EU cloud initiatives backed by VC money going straight for the endgame.
To add on to the comments about Hetzner building their own custom hardware, they also custom built their own software stack. They rejected the hype that was OpenStack and worked diligently on their own hypervisor platform (that they are incredibly secretive about) and that appears to be paying off in spades for them. Most sovereign cloud plays end up being suffocated by the complexity, and incoherence, of the OpenStack ecosystem. It just becomes impossible to ship.
For a fascinatingly different take on how to build a datacenter: https://www.youtube.com/watch?v=5eo8nz_niiM
* Edit: remove speculation about Kubernetes and Hetzner, that was based on hazy memory.
I am asking for this since a while and was told there is no way Hetzner would offer such a service. Certain Posts on Social Media have also never been answered with any kind of indication that they are actually working on it.
Please provide some Details on this.
So huge grain of salt, you are totally right. It could be internal platform work only.
1. Strict rules and strict customer verification. Crypto mining that wastes SSDs is not allowed. Portscans, mass emails, etc. are not allowed. They also don't offer GPUs to the general public because it has been abused in the past. You usually need to send in ID documents just to open an account. My guess is this allows them to avoid most bad actors and, thereby, waste less money on fraud.
2. Extremely long-term investments. They typically build their own hardware and then use it over 10 years. They have their own flea market where you can rent older server models for a steep discount. That means they will have a long time where the hardware is fully paid off and still generating revenue.
3. Great service. With a mid-sized company, I can call their technicians in the middle of the night. The fact that we could call them in case of a crisis has generated A LOT of good will. But I would be truly surprised if they didn't make a profit off those phone calls, as they charge roughly 4x the salary cost.
4. High-margin managed services. In addition to just the cheap servers, they also offer a managed service where they will do OS and security upgrades for you. It's roughly 2x the price of the server and it appears to be almost fully automated. I know some freelance web designers who will insist on using Hetzner Managed for deployment for their clients, because it is just so convenient. You effectively pass off all recurring maintenance for €300 a month and your client is happy to have an emergency phone number (see #3) in case the box goes down.
Does anyone know if that's still the case?
As far as I know, they do not require any ID when canceling the service.
[1] https://registry.terraform.io/providers/hetznercloud/hcloud/... [2] https://community.hetzner.com/tutorials/howto-hcloud-terrafo...
EDIT: For what it's worth, I have had good experiences with app servers hosted on Hetzner Cloud and managed Postgres provided by ElephantSQL (https://www.elephantsql.com/) for Germany-based apps.
I'd love some feedback, specifically:
- Which services do you most want to use/have managed
- What databases do you find yourself using the most
- Concerning caches, do you use memcached or mostly Redis?
[0]: https://nimbusws.com
I want to like Fly, but the reliability is one of those were I feel like every time I investigate moving workloads over I'm disappointed by these stories over and over again.
You shouldn’t need to multi region a postgres yourself - they should have at least 2 data centre redundancy for the region and it just works.
Hope they get some magic sauce to become better at this.
When I saw them describe their multiregion SQL replication architecture I thought "what crazy person thought this wouldn't eventually open up a spider's nest of distributed systems errors?"
Their license would require a company like fly.io to pay them though, so I'm sure this resulted in fly.io instead trying to whip up an improvised infrastructure on the back of stock Postgres. I bet this cost them a whole lot more than paying CockroachDB would have, but devs have been conditioned that you should never ever pay for software even if it's the result of tons of deep engineering and solves massive brutal problems for you. I also bet there's some not-invented-here ego involved.
P.S. I don't work for CDB but I would absolutely consider them and we may end up using them at some point. They let you do a ton for free. They only charge for stuff you need if you get really really huge or if you are running a SaaS reselling DB services like fly.io would have been doing.
It's actually the only region to serve the entire AU and NZ population with any reasonable latency. (Ok, Singapore can do in a pinch for at least sub 200ms.)
You'd wanna hope its more than one machine!