Hacker News Reader

Comment by fabian2k | Hacker News Reader

There was no intentional blocking of Tor there in the past as far as I understand. But there are some mechanisms that block users from IP addresses that posted spam or abusive content. Those are automated and based on flags and not directed towards Tor specifically, but of course TOR addresses are more likely to have been misused that way so your chances of encountering these blocks is dramatically higher when using Tor. But these blocks only apply to low reputation users, I think once you're above 100 they don't matter anymore. And they also don't block reading the sites.

There have been DDoS attacks recently on Stack Overflow and Stack Exchange, I think it's quite plausible those have something to do with this. They still seem to be under active attack intermittently.

View on HN

It was never "TOR", it's; Tor: https://matt.traudt.xyz/posts/2021-02-22-tor-spelling.

The new blocking completely blocks access to the site, not just posting. See for yourself; https://www.torproject.org/

Tor is next to useless for DDoS attacks, as it doesn't offer any amplification. For every byte you send in via TCP, you get one byte out. For attackers with large botnets, it doesn't make sense to DDOS over Tor, as the number of exiting IP addresses are limited and it's easy to block them all. It makes more sense to use your thousands of available botnet IP's that aren't on any lists.

Their blog is claiming most of the ddos traffic came over tor.

  > Tor is next to useless for DDoS attacks, as it doesn't offer any amplification.

It seems that these attacks are not being carried out to take down the Stack Exchange network. It seems that these attacks are to take down Tor as a legitimate technology. Here me out.

In order to carry out these attacks, the attacker already controls enough machines to DDoS one of the largest websites on the internet. So, huge botnet or nation state. Now, we have one of the largest websites on the internet telling it's tech audience:

  > An immediate solution for users who find themselves blocked is to access our site
  > from other IP addresses, via home internet, work internet, or other VPN services

https://meta.stackexchange.com/questions/376060/update-on-th...

They are normalizing the "workaround" of using an insecure IP address when Tor is inaccessible. This will lead to all non-secretive and non-illicit Tor usage to go back to the open insecure internet. Thus, everyone still using Tor "has something to hide" (as if that wasn't the case already). By forcing all but the most desperate users off Tor, Tor can be discredited as a nefarious tool.

>They are normalizing the "workaround" of using an insecure IP address when Tor is inaccessible. This will lead to all non-secretive and non-illicit Tor usage to go back to the open insecure internet. Thus, everyone still using Tor "has something to hide" (as if that wasn't the case already). By forcing all but the most desperate users off Tor, Tor can be discredited as a nefarious tool.

I think this is a reach - after all, if you wanted to discredit a tool for people in oppressive regimes, people who care about their privacy, and people doing illegal things, why start with the programming help site? (I know SE has other sites, but the biggest ones are for tech help)

...That said, I could see this being a way LE could try and unmask a very high-value darkweb programmer. Still a reach.

> I think this is a reach - after all, if you wanted to discredit a tool for people in oppressive regimes, people who care about their privacy, and people doing illegal things, why start with the programming help site? (I know SE has other sites, but the biggest ones are for tech help)

My guess is they'd target websites that are important to TOR users, and tor.stackexchange.com is one of them. I also don't think they started with SE. TOR IPs are continually filtered by Google, Cloudflare, and other automated firewalls.

For good reason, though. Bad actors using tor not for privacy but to evade IP bans on websites/games/etc or to shit post or disrupt legitimate communities-- this is the issue IMO. It's easier to simply block all to IP addresses, especially if you just need it to stop and don't have the resources of a larger company. And for larger companies that block in an automated fashion as has been mentioned elsewhere, there is so much stuff coming in via those IP addresses that is shady that they all tend to get blocked or at least added to watchlists by companies like cloudflare.

Maybe SO has blocked all the the attacker's available IPs after their previous attacks, and is now resorting to Tor to mask their traffic?

That doesn't explain why you can't read the site. OP is mentioning that they can't access the site at all.

That's why I suspect it's something new and might be related to the DDoS. The main spam and abuse problems are handles by the system I mentioned, which is not the reason here. So this is likely a defense against something else.

You're looking at simple, static pages. Unless something is horribly wrong with their infrastructure, these should take milliseconds to generate, if even that.

>You're looking at simple, static pages.

Stackoverflow pages have many dynamic components like vote counts, reputation points, sidebar related-questions-links, new comments, etc. An excerpt from their blog explains they can't cache the output : https://nickcraver.com/blog/2019/08/06/stack-overflow-how-we...

Even though Stackoverflow's website io access pattern has higher reads than writes, the resultant generated html is still not as static as Wikipedia pages. Even if they're efficiently using cpu to generate the dynamic elements, the cost of egress traffic may also be a factor.

All that said, I don't have any insight into what heuristics they use to block certain ip addresses.

OK, that's true, they're not static, but we're still talking about a few database queries which quickly hit indexed columns. If you're running a public website like this, it's not like each page takes several seconds to load.

The egress traffic is also trivial - a page seems to clock in at below 50k. If they're paying $0.02/GB, that's $1 / million requests.

More importantly, if it really were a DoS attack, there's way less obtrusive methods, such as a CAPTCHA or similar verification screen.

That blogpost is intriguing!

> variants of cache... anonymous, or not? mobile, or not? deflate, gzip, or no compression?

I don't understand why the markup would vary between mobile/desktop, or why they would even consider compression a varying factor to account for in cache. Maybe that's because their backends produce such specific variants that they have a hard time caching in the first place?

If 80% of pages are only requested every two weeks, then it doesn't make sense to cache those. There's still probably lots of stuff you can cache, such user profile/stats and question/answer scores for several minutes. You can also cache many parts of the markup that are not going to change often. I mean it's always even better when you cache the entire markup, but there can still be lots to gain if you cache only small bits that are expensive to acquire/template.

> But the cost of memory to store those strings (most large enough to go directly on the large object heap) is very non-trivial. And the cost of the garbage collector cleaning them up is also non-trivial.

It looks like their cache implementation could (should?) have been based on better foundations. First, i think the cache doesn't have to reside in memory: disk accesses are fast, certainly much faster than running a database query over the network which will need to access several files and cross-reference data with extra latency on top. Then, because if you're gonna store long-lived stuff in memory you should probably use a garbage collector based that's tailored for this usecase, not your language's (.Net) default GC... maybe Redis? Don't get me wrong, i find it pretty cool if some engineers want to develop a homebrew cache, but that sounds like a huge project in itself.

> the cost of egress traffic

I'm not aware of SO tech stack, but i'd be surprised if they have much egress fees. They're a very big site so they probably run off unmetered dedicated servers if not their own hardware on cheap transit. Who knows, they may even have their own AS and peer with other providers in some locations? From a quick request, it looks like stackoverflow.com is served from Fastly AS but i personally don't understand why: i don't remember seeing any heavy content (video/images) on SO so in that kind of situation a CDN would hurt more than help on slow links. Maybe that's because they bundle megabytes of javascript crap? Now that they're blocking tor users i can't check for myself :-)

They do use Redis. They have a page on their tech here: https://stackexchange.com/performance

That makes sense. From what i remember of using redis a few years back, it doesn't do file-based cache, only in-memory. Something like nginx is very good at file caching, though.

They're not static pages, the lists of questions, users and everything else are dynamic. Some of them are obviously heavily cached, but you can't cache everything. It's certainly possible to find more expensive pages if you put in some effort.

Grant that some of them are more expensive. But why can't I visit any page?

It potentially could be related to all of the "knock-off" websites that scrape StackExchange data. Maybe they are going outbound on various Tor nodes and getting the IPs blacklisted as a result of reading thousands of pages too rapidly.

Given my experience with the network quality of Tor, I'd be surprised if scraping was A) efficient to do over Tor and B) that Stack Overflow would even notice it because as I said, the network speed is too slow, so can't add that much traffic compared to the absolutely staggering amount of traffic they get from non-Tor.

Also, what would be the point? It's easier to download a data dump from https://archive.org/details/stackexchange

I wasn't aware that existed and obviously that would make any scraping utterly useless

You would be surprised/disappointed at the amount of abuse the bigger sites have to handle

Things like this https://news.ycombinator.com/item?id=26072025

Interesting, but doesn't fit the context of Stack Exchange blocking Tor. Your example there is regarding a mobile app hotlinking a image of a flower, which seems easy enough to block/fix, while Stack Exchange blocking all Tor users from even reading Stack Overflow doesn't make so much sense.

1 image abused enough, they dig, find the culprit being an app.

Fixes, updating the app, blocking the image, blocking all requests with empty user agents.

------

1 person abused enough SO, they dig, find the culprit being someone using Tor network.

Fixes, identify the user and ask them to stop, block all traffic from Tor.

-------

Do you propose an alternative fix?

Reply on news.ycombinator.com