Sourcehut network outage post-mortem
sourcehut.org
sourcehut.org
Maybe I'm weird, but I'd consider colo to be a closer cooperation than just renting some virtual servers from wherever. And just getting told "Yupp, your null-routed. No, we can't give you access for specific sources over a different path. Get fucked" - or, in fact, not getting told that - is ... one of our ex-hosters was like that.
And as a service provider, I have strong feelings about the customer service there.
Maybe I don't know big infrastructures, but this just leaves me with a weird feeling in my guts.
But hell. Make sure to give your engineers - and their family - something. After some hell-weeks, we've given people some budget to do something fun with their family, because the company had to take so much private time during those weeks.
I had a ten year colo relationship with AT&T, mostly but not entirely datacenters, where the salescritter would be changed every single time I tried to talk to them. 3-4x a year, they either left or were re-org'd elsewhere or our account was moved to a different group. (Datacenter techs: competent. Datacenter manager: competent. Sales staff: chaotic.) Eventually I learned to open a ticket to find out who the salescritter would be.
On the other hand, there was a competent server salesperson at company X who I used to buy lots of stuff from; company X started raising prices and flubbing tech support returns, so on my next order I tried company Y. I was rather surprised to get the same server salesperson -- who had just taken a job at Y.
Good coverage of the event: Security Now! Podcast https://www.youtube.com/watch?v=ehfV7cRLkFE
But, even if you only have a single action available, isn't communication with the customer key? We've pulled customer systems offline, and parts of the customer were glad because we could provide information and help with handling harsh situations they had.
And even if this isn't a 1% revenue customer, if they are receiving a DDOS-whopper you cannot happen, that would probably be a customer you want communication with. Even if the goal was just to figure out if there was glory in hosting them or to just fire them.
Just null-routing and being unavailable seems unprofessional, though I know quite well how .. well-structured these acquire-everything hosting providers are internally. Doesn't change that this is bad customer service.
> Following our initial quote from CloudFlare, we understand that some CloudFlare employees undertook a grassroots effort internally to convince the leadership to sponsor our needs, and eventually CloudFlare came back to us with an offer to sponsor our services for us free of charge. This was a very generous offer for which we are very appreciative; in the end we did not take them up on it as we had made substantial inroads towards an alternative solution by that time. I have had my reservations about CloudFlare in the past, but they were there for us in a time of need and I am grateful for that.
That said, what will happen when more companies publish their experiences with "enterprise sales"? There's an article from HEY[1] about how broken the sales process is. To get a quote, you normally have to endure 2 or 3 zoom calls before the price is unveiled.
There's probably room for an innovator to fix all of this.
1: https://world.hey.com/dhh/the-only-thing-worse-than-cloud-pr...
This right here is invaluable and something you only get from experience. Planning and theory only get you so far.
I extend this thinking to deploying large infrastructure changes you've never done before - you can only plan so much before pulling the trigger and just doing it and seeing what happens.
> We also did our best with hg.sr.ht, but it is community maintained
It looks like git.sr.ht is hosted on OVH in France, while hg.sr.ht is hosted on High5! in the Netherlands.
It's not entirely clear to me how this affects their product roadmap or support, but definitely good to know.
> restoring service was delayed until we could get the community maintainer, Ludovic Chabant, online to help
Maintainer, singular!
The only reason i use Sourcehut, and the main reason i pay for it, is because i stubbornly still use Mercurial, and want first-class support for it. With the utmost of respect to M. Chabant, that is not exactly first-class.
It would appear that Ludovic Chabant is working full-time at Epic Games. He is unlikely to have the capacity to be on call for Sourcehut.
They explain it here:
> However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS
Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with an anti-establishment manifesto than any practicality. Then again, I might be wrong.
They can at least reach and reason with OVH, as mentioned when they got flagged as an out bound DDoS.
> Being that this is Drew, I wouldn't be shocked to know that this provider choice has more to do with a anti-establishment manifesto than any practicality
I feel this is a pretty unfair barb considering one of their first moves was reaching out to Cloudflare. Unfortunately, non-http traffic + the need for tls termination on their own servers (pretty sure cloudflare calls this Keyless SSL) squarely lands them as an enterprise customer w/ enterprise pricing.
Drew probably had already entered into agreements with OVH when cloudflare came back around, and we don't have insight on the terms or period for which Cloudflare's second offer was good for.
Not wanting your traffic MITM'd is anti-establishment. That's where we're at LOL.
I appreciate the service though so I hope the differences between maintained and operated doesn't mean anything in the long term.
That said I'm very happy to use Sourcehut and I think they'll overcome these challenges over time. They seem to have the staying power.
There's also the question of whether the DDOS is still even trying the old infrastructure. The post says it's unreachable, but that would be true if the null route hadn't been removed yet.
> However, we found that OVH’s anti-DDoS protections were likely suitable: they are effective, and their cost is amortized across all OVH users, and therefore of marginal cost to us. To this end the network solution we deployed involved setting up an OVH box to NAT traffic through OVH’s DDoS-resistant network and direct it to our (secret) production subnet in AMS; this met our needs for end-to-end encryption as well as service over arbitrary TCP protocols.
They could still try to knock your entire datacenter offline but that is much harder.
You're still depending on either a "secret" IPv6 network, or your upstream provider performing some source-based routing to only route packets from the VPN connection. I doubt that's available to a simple colo customer.
The article mentions how setting up SourceHut from scratch is a complex undertaking with hundreds of small tasks. Some of those are understandable, especially given the amount of data they surely handle, but setting up a complete environment and restoring from a backup should be a simple and mostly automated procedure, not a gargantuan undertaking that needs to involve the entire team several days. There are always difficulties when production is down, and you're trying to restore a full system while undergoing a DDoS attack, I get that, but the reason we have modern cloud providers and tooling is to make creating new environments as painless as possible. It seems foolish not to take advantage of that.
I'm not a fan of Kubernetes either, but it's good that they're experimenting with it. Hopefully it leads to quicker deployments if this happens again.
This sounds like a case of treating your infra as pets and getting stuck when it suddenly needs to be replaced.
1) Test/demonstration of a DDOS against a random target.
2) Attack against a project hosted on sourcehut to make it unavailable (there was even the speculation of disabling a master repository so an end-user could not check that his own local version was the correct one, thus using it with a security hole or a trojan)
3) Attack against a page hosted on sourcehut (I joke that someone wrote "Putin = Fag" on his sourcehut hosted blog).
4) What else ?
I'm actually serious there - school/university holidays invariably bring with them a shitload of fairly random DDoS attacks aimed at ... whoever the attacker feels like.
I don't understand and I don't know where they get that volume of packets from, but I've seen it happen a quite sufficient number of times over the years to consider it as a possibility here.
(excuse the vagueness; if you're ever at the same conference bar as me you're welcome to follow me outside when I go for a smoke and ask for the full story then)
So, to me, this is quite unlikely.
No it wasn't. The outcome is due to major networks being shite. Not accommodating newer technologies and gate keeping services to resolve DDoS attacks.
All major network upstreams could do so much more to make the net more reliable and resilient to small ISP. Myself included.
peer neutral networking, not having tons upon tons of e-waste prone to botnet behaviour, it wouldn't be like this.
I'm not a big fan of some of his hot takes, but I still respect him and trust him with my data.
For the curious, the terms are here: https://man.sr.ht/terms.md#permissible-use
And even if you agree with the current set of restrictions, are you sure it will not be further expanded? I am not.
As for banning sex content on a paid service, you'll find it's more common than you think, since payment processors tend to drop customers who permit that sort of thing. Porn-enabled services have chargeback and failed-charge rates orders of magnitude higher than services which forbid them.
There are a ton of reason to fire a client. The two mentioned here are completely uncontroversial from a business standpoint.
For the same reason GitHub does? GitHub's AUP at https://docs.github.com/en/site-policy/acceptable-use-polici... says:
"We do not allow content or activity on GitHub that: ... is sexually obscene or relates to sexual exploitation or abuse, including of minors".
Atlassian's AUP at https://www.atlassian.com/legal/acceptable-use-policy says "Inappropriate content" includes "Posting, uploading, sharing, submitting, or otherwise providing content that ... Is deceptive, fraudulent, illegal, obscene, defamatory, libelous, threatening, harmful to minors, pornographic (including child pornography, which we will remove and report to law enforcement, including the National Center for Missing and Exploited Children), indecent, harassing, hateful"?
GitLab's AUP at https://handbook.gitlab.com/handbook/legal/acceptable-use-po... says "unacceptable use of our services [which] applies to all users of all GitLab services including those on the Free, Premium, and Ultimate GitLab tiers" mean "you must not: Create, upload, submit, execute, transmit, or host anything that ... is vulgar, obscene, or pornographic, or gratuitously depicts or glorifies violence."
Now, there are differences between "explicit sexual content", "sexually obscene" and "pornographic", but if you are worried about possible further expansion, you shouldn't use any of these code hosting services.
Instead, I was pointing out that since all the providers I looked at have essentially the same restriction, you likely shouldn't use any of them. Certainly there are a lot of people who use GitHub despite having no guarantee the ToS won't be more restrictive in the future.
Sourcehut's ToS is certainly not exceptional in that regard, so really you are objecting to essentially every 3rd party code hosting provider, yes?
Or is there one you had in mind where you aren't concerned about further expansion?
A simple Google search on the topic should be educational.
On an account that you pay _Drew_ for. Do you also complain because someone renting you a garage doesn't want you running a strip club out of there?
However, Sourcehut is actually FOSS software.
IE: if you wanted to run one of their banned things, you could, just on your own hardware.
It's fine, in my opinion, to moderate your services if people have an escape hatch to get out of your service if you require them to move along.
This is a far cry from services such as GitHub, or even Gitlab (with their open core) as transferring to your own system is actually possible, though not without some relative pain.
I don't like crypto projects, so of course I am biased here. But if you like free speech then there's not many options and I think sr.ht is the best one (especially if you plan to self-host).
GitHub is well known to be controlling of speech and even championed some measures that affected the entire industry, and as others have mentioned they have restricted projects on a relatively arbitrary basis. Sometimes even due to geographic region.
I believe generally letting things happen as long as money comes without any regards to values behind the things might have been detrimental.
Also, I always have observed this. This reminds me. Not targeted at gray_-_wolf but a general observation which I just remembered...
There is a trend where people attack small actors/entities for smaller mistakes or opinions. But give free card to big players you cannot touch for atrocious BS. Because they are monopolies or filty rich.
Especially in tech, you don't talk shit about google or ms a lot publicly. Cos that makes you less hirable.
Social/collaboration features are explicitly deprioritized by design; I think this is a natural consequence of srht being built by and for lone wolf developers. GitHub and Gitea (which is basically a github clone) seem much more geared toward collaboration by groups, something most small-time f/oss developers don’t need.
Also, the emphasis on email and irc is bad, imo. The web won because it is better. A lot of the anti-web stuff is just tradition.
Also, I hate when I’m looking for useful forks of something on GitHub and have to sift through tens of useless forks that were created just to be able to submit a pull request.
Sometimes useful when the main project gets nuked or deleted, if you can find them.
I also self-host Forgejo in my homelab and really enjoy it.
See also their clarifications on Gitea the company[1]:
> Gitea Ltd. will be open to building special versions for special clients and will contribute any features back to the main repository when possible
This was in a followup to the original announcement.[2]
Forgejo (i.e. Codeberg, a FOSS non-profit) maintains that the project should be led by the community, not a company[3]:
> Sadly, Gitea Ltd broke that trust by a lack of transparency: its existence was kept a secret during months. After the initial announcement, Gitea Ltd published another blog post but it was still vague and there has been no other communication since. Who are the Gitea Ltd shareholders? Who, among the Gitea maintainers, are employees of Gitea Ltd?
[0] https://about.gitea.com/pricing/
[1] https://blog.gitea.com/a-message-from-lunny-on-gitea-ltd.-an...
[2] https://blog.gitea.com/open-source-sustainment/
[3] https://blog.codeberg.org/codeberg-launches-forgejo.html
If someone contended that SourceHut optimizes for devs who've been writing Linux kernel code for 25 years, so you weed out all the newbs and can get the hardened veterans involved in your project, I could buy that. I'd disagree that it's what I'd want for my project, but to each their own. I couldn't recommend it as an alternative to other services that require participants to know how to use a web browser.
With the PR flow, people need to sign up to the website, create a fork, clone the repo, make their changes, go into a slow web ui etc. It mostly works because everyone is on Github. However, even that solution sucks if you are having a polyrepo setup and need to make changes in many places.
For bazaar style development where you accept contributions from anyone and don't use Github, the email flow is so much faster and simpler. Yes, you need to set it up once. But the other day I contributed to a open source project that was self-hosted, and it's amazing that I just can clone the repo, make my changes, commit and then git-send-email, bam done. Had I needed to sign up and create an account, set up a fork, I probably wouldn't have bothered because it was a small contribution. However no need to register to a website, no need to click through a slow ui, no need to create a fork, it reduces the ritual to make contributions by quite a lot, given that you've set it up.
There is also https://git-send-email.io/ which provides a nice tutorial for people.
I am glad that there is a good alternative that supports this flow, because I think it is superior. There are a ton of alternatives if you want the PR flow (Gitlab, Gitea, Github, Codeberg).
But in some ways it isn’t. Like any fool (like me) can just get some Emacs configuration for free from others. There doesn’t seem to be that kind of sharing for all the fiddly little things you need to do with git-send-email and the rest. All I’ve heard so far is that, oh yeah I usually deal with this specific issue by running some Perl scripts that I wrote eight years ago and that I’ve been nurturing ever since. But it wouldn’t be very useful for you because it’s very, very idiosyncratic. Might not even work outside Debian and my Apt state...
To be clear though: there are programs and tools beyond the git(1) tools themselves that help you with email workflows. Like b4 which is for the Linux workflow.
So what can you do with that? Doesn’t seem like you can do anything.
Outlook seems especially bad. [Here] is some HN hearsay about how even Microsoft had to run Linux machines in order to contribute to the Linux project.
It's not the case that SourceHut treats collaboration features as out-of-scope. Drew seems to draw a bright line between social and collaboration features, and intends for SourceHut to offer solid collaboration features, but without any hint of trying to be a social network.
From an old comment of Drew's: (edit: I now see Drew's comment was replying to you yourself)
> SourceHut is designed to facilitate collaboration, of course, but it's done differently from platforms like GitHub and those that seek to emulate it. And of course it is more than a git frontend, providing tools specifically to facilitate collaboration such as mailing lists and bug trackers. SourceHut is an engineering tool, not a social network. It is designed to get your work done and then get out of your way.
> GitHub is explicitly designed like a social network, and this is a design that we reject.
(From https://news.ycombinator.com/item?id=31963467)
> I think this is a natural consequence of srht being built by and for lone wolf developers
Built by lone wolf developers, plural?
Minus that last part, I'd just stick with plain Git. It's everything I need for my own personal, only-for-me projects.
There’s not really much need for a “forge” without collaboration. I wouldn’t pay the price of SourceHut just so that I can fetch and whatever between my machines. That’s like a pricey sneaker net.
If all you want is a lightweight web UI for a git repository, git's official GitWeb project offers this. SourceHut isn't just an alternative to GitWeb.