Popular Linux orgs Freedesktop and Alpine Linux are scrambling for new webhost
arstechnica.com
arstechnica.com
> https://gitlab.freedesktop.org/freedesktop/freedesktop/-/iss...
Sure, servers might die at times, but this also happens at AWS and can be avoided by using multiple servers in a HA configuration.
Everybody I know is happy with what Hetzner provides, even at production level. OTOH, "arguably better" DigitalOcean sent me a "Your physical host died, so we restarted it. If it persists, we'll migrate" e-mail, which just shows the reality of the hardware.
On your question, while I do not have services on Hetzner yet, I manage a lot of servers, so I know dynamics of a datacenter and what it entails to keep one up.
Your beef is not with Hetzner, but whoever decided to run the service. Unless the customer violates local legislation or the hosting providers ToS, the appropriate action is to leave the service running, be it AWS, GCP or Hetzner.
I would quite frankly have been very disappointed with them if they had done anything in response to you request.
Hetzner is German company and subject to German law. The website they were hosting did not have the (in Germany) mandatory legal notice (Impressum) or any contact details. This shifts the responsibility to the provider (Providerhaftung). Ignoring legal requirements is hardly professional.
Also I would like to note that I did neither request them to give me their customer's details nor to shut down the site. All I wanted was them to work with their customer to have the offending image removed.
Also I am convinced the porn image was not malice but an accident. The scraper replaced all profile images with ones they probably scraped from a forum. I was just unlucky to get a very indecent one.
Had Hetzner collaborated I'm pretty sure this could have been resolved in no time.
I get your frustration in the situation, but even if it would have been helpful to you, it would be extremely unprofessional for a hosting provider to invervene in the operations of its customers unless the customer violates the Terms of Service (an agreement you have no part in), a court order is made, or law enforcement makes a legally supported demand. Random legal notices sent by mail from strangers are not on this list, but could lead to Hetzner evaluating if a ToS violation is in effect that must be resolved.
If the customer is a business, it could be that they have breached the Impressum laws. It would have helped you if they had the impressum, but until legal action is taken it remains undecided if there is a violation, and if there was the consequence would be that the customer is liable for a written warning and in some cases a fine, as decided by the relevant authorities or a court of law.
Nothing that would involve you as a third-party individual.
It's highly available though and the redundant replicas run on their separated physical machines
https://docs.hetzner.com/cloud/placement-groups/overview/
Single machines die (and can't be started again for a few minutes to hours) every few months, but that's acceptable for me, and we also similar things happen at AWS.
US sanction laws are legal nightmare, quicksand that constantly changes. Major global infrastructure projects like Freedesktop should avoid US!
PS: Sorry for the off-topic discussion.
"Hey your customer is hosting a website with random pictures on it. One of them of me is unflattering. Please make them fix it"
The only difference to the original was my profile picture, which was an explicit porn image.
For several weeks everyone searching for my name or my StackOverflow answers saw that. I'm glad that I was not looking for a job at the time.
I exhausted all possibilities on all channels to rectify this situation short of using a lawyer. I put a lot of effort into getting things in order and seriously considered going the legal route but ultimately decided against it mainly because courts are hit and miss here when it comes to reputation damage of regular individuals as opposed to companies or celebrities.
EDIT: I should have answered your question how this is Hetzner's fault more directly. The website they were hosting did not have the (in Germany) mandatory legal notice (Impressum) or any contact details. This shifts the responsibility to the provider (Providerhaftung). Also I would like to note that I did neither request them to give me their customer's details nor to shut down the site. All I wanted was them to work with their customer to have the offending image removed.
Also I am convinced the porn image was not malice but an accident. The scraper replaced all profile images with ones they probably scraped from a forum. I was just unlucky to get a very indecent one. Had Hetzner collaborated I'm pretty sure this could have been resolved in no time.
The channels to reach you were pretty limited and hard to find when I tried. I assure you, I tried everything short of getting a lawyer to send you paper mail.
I still have the communication history in my archives, but it would be a little effort to dig it out.
EDIT: Found the documentation. My first complaints were in March 2011. At some point I got assigned an abuse ID MU-000B0F55-1:18. In May the issue was still unresolved.
Seriously, Hetzner provides so much move value per dollar, sometimes I fear that one day they will find out and just jack up the prices to match the rest.
Yes yes there are cloud features now offered by VPS providers, but they are add ons to chase demand, they aren't positioning their offering to appeal to users wanting a comprehensive suite of services on the platform. Managed databases, SMTP as a service, deployment as a service etc etc etc. For that reasons market rates are different.
For Hetzner to bump their prices significantly they would need to build a cloud platform a la AWS/GCP/Azure. Won't happen by Xmas even if went all in. They are good at what they do and make money so they stick to that.
Their cloud always had on-demand, per-hour billing of servers and block storage volumes, all very easy to manage and provision via their API. Recently they got into object storage space. They even provide a switch to connect their cloud servers with a dedicated one, so you can have, e.g, a beefy GPU server running a LLM model and your web service auto running on the cheap.
I believe that the only thing that really holds Hetzner at their price levels is that the price-sensitive people can always threaten to move to OVH.
The barrier to entry for these providers is simply, "low". So margins got to be low.
The thing that holds Hetzner and the likes is you can't purchase a package, follow the setup instructions to make cross disciplines engineering departments. It is no wonder Amazon, Google and Microsoft built a comprehensive cloud. They were in the engineering business.
It doesn't imply Hetzner aren't doing a stunning job at what they do or that running an infrastructure farm is a walk in the park.
Meanwhile for CI runners you probably could split the big bare metal servers down into smaller individual machines and run less jobs of them. Depending on the CI load profile it might also make even more sense to scale out to the cloud on high demand as opposed to having a bunch of mostly idle machines.
Hope you find a host soon!
The code is small and integrated into the kernel at this point.
Aren't your needs primarily for distributing Windows/Mac packages at this point?
They host quite a few open source projects there. And seem to be one of the few that also hosts for ARM and POWERPC projects.
Thank you for wireguard - it's been a hugely impactful piece of software.
Do you think it would be helpful to outline what hardware resources you would need to successfully migrate the project and all the CI/CD computations to a new home? This would help people determine if they can help with hosting.
You are making all of this sound way more definitive than the ticket. The donation money approach is the personal opinion of the sysadmin. The OSUOSL is brought up, some explanations are added that make it more attractive and remove some doubts, and beyond that it's waiting for the board to decide what's next.
If the issue is cost, slightly older Epyc hardware is quite affordable, and colo deals can be found for extremely reasonable costs. If it's expertise, then all they have to do is ask.
Storage is MUCH cheaper when you colo, and bandwidth requirements are a large part of why you colocate instead of just running servers out of an office building that has at least two upstream connections.
I'm really curious what you think they're using now. Certainly you read the article... It says they're using bare metal servers. That's basically colo where the provider owns, but doesn't control, the hardware.
Probably because it's not redundant or automatically backed up at any interval. The worst days of my life have been during hardware failures at colos.
Perhaps the comparison should be apples to apples when thinking about price.
Set up two locations. Set up rsync. It's still cheaper than cloud storage by far, and that's even after paying yourself handsomely for the script that runs rsync.
I do not see any concrete data from you either. This is a forum. Typically we'd just call this a "conversation."
> Anything can increase cost and complexity. Using Amazon S3 increases cost and complexity.
Yes, and you get something in return for that cost and complexity, so do you care to map out the differences or are you just going to stick to your simple disagreement?
> Set up two locations. Set up rsync. It's still cheaper than cloud storage by far, and that's even after paying yourself handsomely for the script that runs rsync.
You forgot monitoring. You forgot that when this inevitably breaks I'm going to have to go fix it and that you can't schedule failures to only happen during working hours. You're ignoring more than you're considering.
You seem to have a bone to pick. I said owned storage is cheaper, and you made up something about it not being redundant. You're not making a salient point as much as you're trying to handwave and dismiss owned hardware as complex, expensive, not worth the "value" one might get from S3, whatever.
If you REALLY think that storage can't be cheaper than "cloud" unless it's not redundant, then show us numbers. Otherwise, you're just making shit up.
You mention all your other things as if you're saying, "Gotcha!" You still have to monitor Amazon. You still have to manage backups. You still have to monitor resource utilization. You're not being clever by trying to imagine that other admins are as bad as you are because all those things seem hard. Good admins do those things no matter where their stuff is running.
And "when stuff breaks" happens regardless of whether you've chosen Cloud or to run your own servers. It all breaks on occasion (though hopefully rarely).
What I don't know is how to make your web servers failover graceful if one goes down (the really hard problem is if the internet splits so both servers are active and making changes). I assume other people still know how to do this.
With the likes of AWS they tell you the above comes free with just a monthly charge. Generally open source projects like having more control over the hardware (which has advantages and disadvantages) and so want to colo. They would probably be happy running in my basement (other than they don't trust me with admin access and I don't want to be admin - but perhaps a couple admins have the ability to do this).
Purchased, fully redundant storage is MUCH cheaper than anything in the cloud when talking about any time frame of a year or more.
Obviously you can "rent" storage for a month for less than the cost of purchasing it, but only idiot startup CTOs try to argue a comparison like that.
Two sets of storage are still cheaper, and we all have rsync.
Does it automatically repair itself when it fails? Are you sure you're making a proper comparison?
> Two sets of storage are still cheaper, and we all have rsync.
Yes, you can in fact solve 90% of the problem with 10% of the effort, what's your plan for the rest of the problem? Just call in sick?
Still more economical than cloud, but it seems like this has become far too common.
RHEL benefits from freedesktop and X, and as a show of good faith they could support Alpine too.
But as we all know, RHEL/IBM only wants to take free labor and not really give back these days :(
Of course they still get tarred and feathered with the "Red Hat wants to control Linux!" brush because they...contribute the bulk of development to projects like GNOME
I do open source in my free time because my family can't sustain itself otherwise. Kudos to all the devs out there working on open software.
Ludicrous. Red Hat and IBM are far from perfect, but they are absolute heroes for open source. Listing all the projects that Red Hat pays to develop would be very difficult because it's so long. They've even acquired proprietary companies and open sourced their products (while the product was still selling and highly useful!), something virtually nobody does.
Also, Red Hat typically only "sponsors" open source projects that they have some business dependency on. Freedesktop.org might be a good candidate, but Alpine could be harder to justify. I don't know of any RH product that uses Alpine directly. (Most enterprises only have exposure to Alpine through container images.)
Alpine Linux: https://opencollective.com/alpinelinux
Freedesktop [edit]: ..no crowdsource option at the moment
Now go ask your employer to donate.
I can think of maybe three exceptions my entire career, and none of them were especially huge contributions.
Slackware's the same way, most donations come from individuals and very small companies.
I'm not sure how many mirrors are run by the university directly, though AFAIK MIT and RIT host theirs directly.
There are 3 major concepts: understanding how to run the comnand, understanding the idea of public key crypto, and actually using it (i.e. NOT imaging the ISO unless the signature passes).
What it needs is something like a torrent client that 1) doesn't let you download unless you supply the expected SHA first, and perhaps 2) that it verifies that the hash came from the signed webpage where you got the torrent link. Too many people (myself included) think it's not going to happen to them (download a backdoored program/OS).
After 20 years in the industry I'm just now learning how certificates work and how to work with them.
This is what I always emphasise: usability first. If a solution is secure on paper, but confusing to use, then it's not secure - the user can get confused and do the wrong thing. Defaults matter.
> What it needs is something like a torrent client that 1) doesn't let you download unless you supply the expected SHA first, and perhaps 2) that it verifies that the hash came from the signed webpage where you got the torrent link.
This is already a solved problem. Just provide a magnet link. You already have to trust the website to provide the checksum, so why not trust the link?
As for packages, Debian experimented with a BitTorrent transport for apt a long while ago, but I suppose it didn't catch on. Perhaps this was before BitTorrent had HTTP fallback? Either way, this would be an interesting avenue for research.
At the moment, we operate 900 servers. We evaluated the idea of hosting mirrors on some of our servers, but our servers are not super powerful, and we have to pay for bandwidth. We use these servers in our production pipeline. Maintenance alone is a massive task, and hosting distro mirrors could be incredibly challenging. We are not at that scale yet.
We could provide IP location data sponsorship to popular distro mirror systems, which would make traffic routing and load distributions more effective.
But i don't know who is responsible for that.
The question of "who is responsible": anyone is free to run their own mirror, after all this software is freely redistributable.
As for "why not", if I were to lead a project like Alpine, I would insist that the org stays in control of its own infrastructure. Mirrors are also only one chunk of the problem; you also need builder machines.
presumably because it's a silly idea, given Cloudflare isn't a colo or dedicated server company and they won't let you just rack machines in their DCs for the same reason Google won't.
Running on 'large donations' is not a viable strategy for any long-term goal. Perhaps its time for Linux to consider running a tiny datacenter of its very own to both dog-feed itself and give itself extra momentum or inertia from donation stall-out?
You know running an anctual datacenter with all its cooling ,storage, networking and power requirements is a full time job for several people right? Why not put bare metal in someplace which already does that for far cheaper and better?
Imagine if Wireguard were backdoored because someone working for the ISP that runs the VMs compromised their VMs through the hypervisor. How would a project audit an ISP? How could anything be trusted? Bottom line: it can't. ISPs don't give that kind of information to customers unless you're special (government, spend crazy money).
While it's still possible to compromise a machine through physical access, it's MUCH more difficult. How do you bring it in to single user mode to introduce a privileged user without people noticing that it's down, even momentarily, or that the uptime is now zero? Compromise like this is possible, but worlds more difficult to pull off than compromise through hypervisor.
Even Microsoft updates can use end user device distributed hosting.
Admittedly I am rather surprised by the storage requirements from Freedesktop.
Small by on premise enterprise standards is 4-16 cores and 128GB of RAM.
Our big VMs have 64 cores and 1TB of RAM (db servers).