Statement regarding the ongoing Sourcehut outage
outage.sr.ht
outage.sr.ht
1. Indian police/government
2. Original user
3. Total coincidence
When it happens you just realize: this should not be possible. This is a game that hugely benefits the most powerful players whether they just don't bother solving the problem or actively play in it.
Like the police in India?
In fact we were contacted by multiple police authorities from different regions.
Beat cops definitely aren't, but most police forces in India are state level and a couple are federal, and will usually have a Task Force devoted to offensive and defensive security, and if not, will contract out to companies like Appin Security Group (who were able to force Reuters and SentinelOne to remove their report into how the private-public cyber model in India operated [0])
Also, after the 2008 Mumbai Attacks, the Indian Ministry of Home Affairs began working on coordinating and centralizing Metadata Analysis and Monitoring [1][2][3], because they unable to trace the VoIP calls used by LeT [4].
If it was actually National Security related, it would have went through a couple fusion centers that the Indian government formed [5][6]. If it was some local PD asking the platform, they probably wouldn't have the capabilities to DDOS
[0] - https://www.reuters.com/investigates/special-report/usa-hack...
[1] - https://en.m.wikipedia.org/wiki/Central_Monitoring_System
[2] - https://en.m.wikipedia.org/wiki/NATGRID
[3] - https://en.m.wikipedia.org/wiki/DRDO_NETRA
[4] - https://www.wired.com/2008/12/mumbai-and-voip/
[5] - https://en.m.wikipedia.org/wiki/National_Cyber_Coordination_...
[6] - https://en.m.wikipedia.org/wiki/Indian_Cyber_Crime_Coordinat...
Because terrorism related stuff falls under the Indian equivalent of Homeland Security, even if it's local PD triaging.
> Americans like Scammer Payback report giant scam callcenters with credible evidence, they take well over a year to raid it
Raids that are not National Security related need coordination between the Federal Government and State Government in India. The scamming call centers are located in a state called West Bengal. West Bengal is ruled by a regional opposition party called the TMC. West Bengal removed it's consent for Federal Police in India to raid without the consent of West Bengal Police. This ended up in the Supreme Court for a couple years [0].
On top of that, the scamming call centers are closely tied to the ruling party machinery in that state, as you need to get a license from the state government to operate a call center, and these kinds of call centers will often be donating to local political parties to look the other way.
Watch Jamtara on Netflix. It's a good overview on the economics of scam calls in India.
[0] - https://www.deccanherald.com/india/cbi-independent-legal-ent...
Many thanks for the context!
Think of Indian democracy as being similar to American democracy in the 1890s-1930s, when local despots like Huey Long and populists like William Jennings Bryan roamed the planet.
> that is the reason why it's always West Bengal
Yep. In other states they either will get raided by the Federal Police (eg. CBI) or the economics of running a call center doesn't make sense.
To run a scamming call center you need a low cost English speaking population AND Political Backing. Most states in India will have 1, but not 2 (or at least, not for call centers).
[0] https://www.wired.com/story/modified-elephant-stan-swamy-hac...
That entire apparatus is rotten due to the incentive structure - if you as a cop don't listen to politicians, you'll get a last minute transfer to some village in the middle of nowhere with no running water
If it was one of these forces within the MHA, I wouldn't be surprised, because it falls under "National Security"
Cloudflare. And any other DDoS protection vendors.
Not saying they caused it but obviously they benefit the same way roof tile manufacturers benefit from hurricanes.
I'd just assume that any service is going to get randomly DDOSed soon after launch, even without any sort of blackmail/targeted attack. Even if you can figure out who did it, there's pretty much no chance of chasing them down.
My takeaway from this wouldn't be "this shouldn't be possible" but "this has been commonplace for decades, and will keep happening, and we should prepare for the next one".
It doesn't really matter whether it's some corrupt local agency or some script kiddie... half the world's connected, on various shitty devices, and DDOSes are gonna keep happening.
Nor is it a good thing that Cloudfare has keys to half of the internet.
And there are alternatives to Cloudflare.
The point is just that DDOSes have been common for decades and they shouldn't be a surprise for anyone. They're an inherent part of the internet protocols we have and the freedom of routing. Saying "they shouldn't exist" is like saying bad actors shouldn't exist. Sure, that's a nice thought, but they do exist (and have existed long before Cloudflare) and we can't just pretend otherwise and believe that our service won't be affected. It's just a matter of time/luck.
I treat this as a foundational flaw in the protocols powering the internet. People shouldn't need to pay off DDoS protection companies as a standard line of business.
https://web.archive.org/web/20231005165737/https://srht.site...
The attribution wasn't subtle - a substantial fraction of Baidu's ads/analytics traffic served to domestic Chinese users was rewritten to hammer that specific repository directly.
NYT coverage at the time: https://www.nytimes.com/2015/03/31/technology/china-appears-...
> You may have noticed that Hacker News was down on January 10th; we believe that was ultimately due to Cogent’s heavy handed approach to mitigating the DDoS targetting SourceHut (sorry, HN, glad you got it sorted).
It was also in https://news.ycombinator.com/item?id=38939532 but I did not see it earlier.
This results in an entire /24 network not being routed to your network and being dropped by your peer instead.
Either Cogent didn't buy our product (or a competitor's equivalent), or they have a network op who's a fresher and only knows how to blackhole things. Either way, it's a bad look for Cogent.
Many ISP also have DDOS mitigation using bgp flowspec to block dirty traffic only and let valid traffic pass through.
A DDoS on AWS would most likely bankrupt Drew and Sourcehut, even if Amazon managed to absorb the traffic. Not to mention the principial issues that they have with AWS in the first place.
It's A M S, not A W S.
so, yes for really important stuff i could get patches through to projects i care about. but in practice the handful of projects i'm involved in on sr.ht seem to be mostly stalled.
That being said, I did send a small patch directly by email just for the fun of it.
Maybe someone higher up in Cloudflare here can escalate this subject and help them along for a proper quote?
The PR gained in the development community to help a service beloved by hackers in this site could be very beneficial.
I feel awful for sourcehut, but that doesn't mean that cloudflare should be obliged to support them for free or below cost rates.
Though honestly it’s totally possible that cloudflare was offering a real discount but it’s still just very high for something like sourcehut.
I am unable to edit my original post anymore to reflect that is also my opinion :(
99.999999% chance they were not.
“Look, if you’re trying to sell DDoS services, go right ahead, demonstrate your ability on our infrastructure. That way you’ll also know not to target our infrastructure”
But at the same time that might genuinely be a positive for the net. Just like the drug epidemic — you can’t stop people from doing it, but you can reduce the potential harm
"However, this is not an ordinary DDoS attack; the attacker posesses considerable resources and is operating at a scale beyond that which we have the means to mitigate ourselves."
:(
The things under attack are the message, not the attack itself. XD
Of course, GitHub and Microsoft have significantly higher resources than SourceHut to endure the DDoS.
Kind of, but not really. A lot of times you'll see UDP blocked from EMEA, which stops a good amount of attacks but doesn't solve the problem. It also creates problems for services that rely on UDP like VOIP. These days, even if the command originates from EMEA many of the participants are IOT devices that've been compromised - and those may live in the host country!
Blocking an entire country can do something, sometimes, but it also opens up a wormhole of optics when users who are not knowingly part of malicious activity complain they can't access a service that the rest of the world can. Of course, the host country that operates with a decent degree of CYA acts like they have no idea why someone would do such a thing.
Mitigating this stuff long term is often a game of 4D chess on a rotating board.
At that point I wonder why peering partners of Chinese ISPs used in these attacks don't go and drop connectivity unless the abusive traffic gets stopped.
Drew, you're great, Simon and Conrad are great, you'll get through this, and you will be fine.
Keep doing your great work, forever.
Has there been actual cases of prosecution?
https://krebsonsecurity.com/2023/02/finlands-most-wanted-hac...
Motivation can often be like why everything is re-written: "Because we can" or just "Because it's there".
I'm not in the circle of these things, but history/news suggests that many are performed by those under the age of adult prosecution, or from countries that don't care, so there is minimal risk, and even when there is those involved are not the sort to believe they will get caught.
The Chinese government regularly targets GitHub because it hosts VPN software.
This attack expedited it greatly.
Maybe they have another plans under that, a better server or set of servers, some hand-rolled mitigations, etc. I have no idea. I'm a user as everybody else.
> One of our main concerns right now is finding a way of getting back online on a new network without the DDoS immediately following us there, and we have reason to believe that it will.
More likely, though, they're restoring the service to a fresh IP range and put the servers behind some kind of DDOS-protection or, alternatively, they simply choose to do the switch now as they need to do a full restore anyway and it's not related to the DDOS mitigation.
2. If I host a service on AWS, Azure, Linode, DigitalOcean am I also susceptible to layer 3 DDoS?
2. Depends if DDoS protection is part of the offer, I suppose.
https://web.archive.org/web/20240111132224/https://man.sr.ht...
Respectfully, if you guys asked this question...how come you don't have a cluster slave as a replica in another data center where the whole thing is synced?
A switch on a wall with an arduino in it where you flip it and DNS is updated to point there & a message is displayed to the users.
If you want a Klugey non-production IoT solution that bodges up something really important THESE days, all the cool kids are using ESP32s.
And as much as I think that would be a totally inappropriate solution for src.ht, I kinda wanna go make a "black-hole" switch for my office.
They also have servers in multiple countries. Some of these servers being blackholed by Cogent, by the way.
I don't get it. Cloudflare proudly advertises unmetered DDoS protection on any plan level. Is that just a lie, or what am I missing here? They don't need to be on a custom Enterprise plan.
would be cheaper to pay a developer to add websocket support to openssh
The Cloudflare HTTP CDN cannot protect SSH any more than a condom will make a good umbrella just because they both are designed for protection!
Looks like level 3 ddos protection is only available on the enterprise plan, it's not included in the unmetered ddos protection.
There are (expensive) ways of mitigating but a project like Sourcehut couldn't afford or justify what will likely be a 5 to 6 figure sum.
For DDoS attacks, you need to have enough capacity to absorb the attack. Major cloud providers tend to have that, as do DDoS mitigation services (Cloudflare amongst others).
> We spoke to CloudFlare and were quoted a number we cannot reasonably achieve within our financial means [..]
Typically what you want to do though is stop the traffic from reaching you at all, so ideally your network provider, who is upstream from you, blocks the illegitimate traffic so your servers never see it and don't get overwhelmed.
What happened here is that due to some administrative lapses, the victims (Sourcehut) of the attack got disabled by the network provider. That was the initial outage. Imagine if your ISP decided to stop routing traffic to Google. Being hosted on GCP, a major cloud provider, would be of no use, since there wouldn't be a network path to them in the first place.
In general, Cogent seems to be doing a rather bad job at dealing with this attack and there's been fallout for many services beyond Sourchut. Google or AWS or Microsoft might've handled it more gracefully, or might not. Though major cloud providers tend to have their own connectivity between their datacenters, they too have peering/transit agreements with other major network providers. If those upstreams stop forwarding traffic to them, the same thing would happen. It's just less likely to go unnoticed.
Cogent is a massive provider, so you'd think they'd be a bit better at this. But they also have a reputation for being awful.
That's not what happened based on my understanding. The provider nullrouted their traffic (which is common if a customer is under attack), but Sourcehut couldn't talk to the customer support as their support panel wasn't working for them.
That service is the easiest and most profitable to abuse, there are certain providers in certain countries that price inbound SMS very steeply, and are willing to share the profits with you. If you if you can get an attack going.
DNS_PROBE_FINISHED_NXDOMAIN
They're allowed to decide what they'll host, and it looks like they're just being clear about a boundary. Unless they're playing favorites and allowing some crypto, what is the problem?
They aren't removing code simply because they don't like it. They have defined a term of service. You can pretend it's more capricious than that, but it isn't the truth unless you have some additional evidence beyond that one thing.