Hacker News Reader

Comment by simonw | Hacker News Reader

"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler."

Wow. That's some seriously disrespectful crawling.

View on HN

10TB/day is, roughly, a single saturated 1Gbit link. In technical bandwidth terms, that is the square root of fuck all.

The crazy thing here is that the target site is content to pay ~$700 for the volume of traffic that you can move through a single teeny-tiny included-at-no-extra-charge cat5 Ethernet link in one single day. And apparently, they're going to continue doing so.

Yeah that's an insane price for bandwidth, they need to move providers ASAP if that's the kind of fees they get.

Hosting documentation shouldn't need that much bandwidth. It's text and zip files full of text. Without bots, that's a very small cost even if the bytes are relatively costly.

Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.

Why is it "classy" to not name names when a business (which likely holds it self out there as reputable) behaves badly, especially when it behaves in a way that costs you money? Everyone is so vague and coy. These companies are being abusive and reckless. Name and shame!

In the article, they mention that they are working with the crawling company to be reimbursed for the download costs.

Naming and shaming the company while you're trying to work with them is a real good way to not get what you want.

Naming names isn't really required. The hosts have a $5,000 bandwidth fee, but so do the consumers. There's maybe 10 companies with the financial & compute resources to let a $5,000-per-month-per-website bug run rampant before taking the harvesting service offline.

Meta/Google/Whoever may benefit from economies of scale, so they're not seeing the full $5,000 their side, but they're hitting tens of thousands of sites with that crawler.

I thought data egress is much more expensive than downloading it?

You know you can hit the data rate they were complaining about by using a residential fiber connection, right? 10 TB per day is about 1 Gigabit continuous if I’m not mistaken. There are probably millions of people that could to this if they wanted to.

There's millions of people who could do that to an individual website. There are remarkably few organisations who could do that simultaneously across the top 100,000 or so sites on the internet, which is how readthedocs has encountered this issue.

It is 100% likely a cloud provider IP range.

They are a persistent source of spam email servers, scrapers, and bot probes.

The simple reason is the operators quickly dump a host, and the next user is left wondering why their legitimate site is instantly spelunking spam ban lists.

It is the degenerative nature of cloud services... and unsurprisingly we end up often banning most parts of Digital Ocean, Amazon, Azure, Google, and Baidu.

Have a wonderful day, =3

It's the degenerative nature of assuming an IP corresponds to a user. They have not corresponded to users for over a decade. I once discovered I'm banned on my mobile phone connection from at least one app which doesn't know that CGNAT exists (a very poor assumption for mobile phone apps in particular). If you must block IPs, do it as a last resort, make it based on some observable behavior, quickly instated when that behavior occurs, and quickly uninstated when it does not.

Really depends on the use-case, but yeah the response happens in a proportional manner.

We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs.

And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along with the active Tor exit nodes and known proxies lists (the hidden feature in context cookies).

Have a great day friend, =3

Why are you ending all your messages with =3 ?

https://www.jpl.nasa.gov/images/pia22092-arp-142-the-penguin...

Don't worry about it friend =3

Read that before, read it again to make sure I didn't miss anything. The lack of clarity here is disappointing and only asks more questions than it answers.

Don't worry about it friend =3

What I'm getting from this is that you hate having users almost as much as Reddit (which enshittified their website and banned all non-shit mobile apps and all search engines other than Google).

Imagine a world, where people walk into your business with a mask over their face saying horribly abusive things... while pretending they are your neighbors... And poof... they automatically vanish along with their garbage content.

They may visit again, but are less likely to mess with the platform. Note, cons never buy anything... ever... it is against their temperament.

I find it interesting several of cons on YC are upset by someone else's administrative policies. Reddit should enforce these policies too, or at least drop a country or pirate flag icon beside nasty posts...

Have a great day, and don't fear the ban hammer friend =3

What is the term “cons” you use here?

In this context, a few users are scraping YC profile information to attempt various scam/wire-fraud schemes against community members.

1. masquerading as YC associated organizations

2. attempting to extract banking information with classic 419'er tactics

3. spamming member contact routes with several phone and email Grifts

These folks appear to be operating on a 4 month cycle, and redistributing target leads to other fraudsters.

Annoying, but hardly a problem limited to YC if that is your concern.

Best of luck, =3

5000$ for 73TiB seems excessive tough? Some EU cloud providers would price it 10x cheaper

Hetzner is $1 per TB and that's only if they decide your overall usage is excessive and needs to be billed for.

The more interesting thing for me is: the crawler didn't detect it on its own that it racked up 10TB from one site in one day.

If I would design a crawler, I'd keep at least some basic form of tracking, if only to check for people deliberately trolling me by delivering me an infinite chain of garbage.

I've been running guthib.mattbasta.workers.dev for years, and in the past few months it's hit the free limit for requests every day. Some AI company is filling their corpus with exactly that: infinite garbage.

That's also $73 worth of bandwidth on another server host. Please stop using extremely expensive hosts and then blaming other people for the consequences of your decision.

at that scale, they could just ship some usb disks and a prepaid return postage.

Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha.

If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days.

73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.)

Good luck =3

When the bots come out of cloud services then the IPs are all over the place: it's much harder to do right these days.

Rate limiting firewalls and spider traps also work well...

There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules.

One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3

I mean, blocking random IP ranges is your prerogative if you don't want to have customers. The scrapers will find ways around, while actual users will be unable to use your site. Residential proxies are something like $5 for 1000.

True, note domestic ISP IP ranges are published, and unless one deals internationally... don't bother serving people that will never buy anything from your firm anyways.

Domestic "Users" functioning as proxies will be tripping usage limits, and getting temporarily banned. Google does this by the way, try hammering their services and find out what happens.

Context cookies also immediately flag egregious multi-user routes, and if it is a ISP IP you know its a problem user. If it is over 15 users an hour per IP, than you can be 100% sure its a Tor proxy.

we ban over 243000+ IPs, and saw zero impact to our bottom line.

Have a nice day, =)

Those of us running sites for public information rather than sales cannot make the simple cut-off that you do.

And again, none of this is simple. It has taken me a few weeks to establish a usage mechanism that does catch the worst feed pullers, but it still can hurt legit new users. That is an opportunity cost.

One must assume most user IP edge proxies are compromised hosts. If someone paid for that list they were almost certainly conned, as the black hats regularly publish that content on their forums. These folks want as many users as possible in order to hide their nuisance traffic origin in the traffic noise.

Allowing users known to have an active RAT or their "proxy friends" on a commercial site is not helping anyone.... especially the victims.

https://www.youtube.com/watch?v=aCbfMkh940Q

Worth studying the problem from time to time when you get bored of the antics.

These folks are generally uninterested in positively contributing to any community, but rather show up to cause trouble for fun and profit.

User API quotas are popular for a reason. =3

Exactly, I'm banned or captchaed from half of all web sites these days, because of the AI.

Try updating your web browser, as sites often flag outdated user agent strings hard-coded in many bots/spiders.

Have a nice day, =3

I am sure that bots/spiders use the very best and latest and least suspicious user agent strings.

Some bad actors do indeed use ancient UA strings.

So do some users. Thanks for banning me.

The captcha trigger events on many sites will often keep nagging/blocking people till they update.

Don't take this trend personally friend, as if we see a fake iPhone sporting bandwidth >1300Mbps+... than the host is getting permanently banned anyway.

Have a wonderful day, =3

6 hours a minute? Are you joking?

Click around a few times on any of your sites and looks like I'll be banned?

Multi tasking? Opening multiple interesting links?

Like what.

Generally for sites:

1. gets incrementally slower until firewall user rate limiting tokens refill the bucket (chokes >6MiB/min bandwidth use, and enforces abnormal traffic ban rules.)

2. Pauses serving a page if you spider though 6+ pages a minute (chokes speculative downloading)

3. if you violate site usage rules 4+ times in the past hour, than your get a 2 day IP ban

4. if you trip a spider trap, than you get a 5 day ban

5. If you are issued more than 5 context cookies, than the IP will get spammed with a captcha on every page for 5 days

6. If you violate any number of additional signatures (shodan etc.) than you get your IP block and route permanently banned. There is only 1 exception to this rule, and we don't share that with anyone.

7. The site content navigation is programmatically generated in JavaScript

8. The legal notice is very real for some people

Have a nice day friend, =)

Who is "we"?

Plural of the deployment team admins.

Don't worry about it friend =3

> 73TB is a fair amount to have on a cloud

From the article:

"This was a bug in their crawler that was causing it to download the same files over and over again."

Must be a CDN choice issue, as most sites limit per IP daily file downloads, or have an account login with a quota limit.

People hitting the same files again sounds like a developer testing their code.

Reply on news.ycombinator.com