Hacker News Reader

Comment by Joel_Mckay | Hacker News Reader

Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha.

If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days.

73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.)

Good luck =3

View on HN

When the bots come out of cloud services then the IPs are all over the place: it's much harder to do right these days.

Rate limiting firewalls and spider traps also work well...

There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules.

One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3

I mean, blocking random IP ranges is your prerogative if you don't want to have customers. The scrapers will find ways around, while actual users will be unable to use your site. Residential proxies are something like $5 for 1000.

True, note domestic ISP IP ranges are published, and unless one deals internationally... don't bother serving people that will never buy anything from your firm anyways.

Domestic "Users" functioning as proxies will be tripping usage limits, and getting temporarily banned. Google does this by the way, try hammering their services and find out what happens.

Context cookies also immediately flag egregious multi-user routes, and if it is a ISP IP you know its a problem user. If it is over 15 users an hour per IP, than you can be 100% sure its a Tor proxy.

we ban over 243000+ IPs, and saw zero impact to our bottom line.

Have a nice day, =)

Those of us running sites for public information rather than sales cannot make the simple cut-off that you do.

And again, none of this is simple. It has taken me a few weeks to establish a usage mechanism that does catch the worst feed pullers, but it still can hurt legit new users. That is an opportunity cost.

One must assume most user IP edge proxies are compromised hosts. If someone paid for that list they were almost certainly conned, as the black hats regularly publish that content on their forums. These folks want as many users as possible in order to hide their nuisance traffic origin in the traffic noise.

Allowing users known to have an active RAT or their "proxy friends" on a commercial site is not helping anyone.... especially the victims.

https://www.youtube.com/watch?v=aCbfMkh940Q

Worth studying the problem from time to time when you get bored of the antics.

These folks are generally uninterested in positively contributing to any community, but rather show up to cause trouble for fun and profit.

User API quotas are popular for a reason. =3

Exactly, I'm banned or captchaed from half of all web sites these days, because of the AI.

Try updating your web browser, as sites often flag outdated user agent strings hard-coded in many bots/spiders.

Have a nice day, =3

I am sure that bots/spiders use the very best and latest and least suspicious user agent strings.

Some bad actors do indeed use ancient UA strings.

So do some users. Thanks for banning me.

The captcha trigger events on many sites will often keep nagging/blocking people till they update.

Don't take this trend personally friend, as if we see a fake iPhone sporting bandwidth >1300Mbps+... than the host is getting permanently banned anyway.

Have a wonderful day, =3

6 hours a minute? Are you joking?

Click around a few times on any of your sites and looks like I'll be banned?

Multi tasking? Opening multiple interesting links?

Like what.

Generally for sites:

1. gets incrementally slower until firewall user rate limiting tokens refill the bucket (chokes >6MiB/min bandwidth use, and enforces abnormal traffic ban rules.)

2. Pauses serving a page if you spider though 6+ pages a minute (chokes speculative downloading)

3. if you violate site usage rules 4+ times in the past hour, than your get a 2 day IP ban

4. if you trip a spider trap, than you get a 5 day ban

5. If you are issued more than 5 context cookies, than the IP will get spammed with a captcha on every page for 5 days

6. If you violate any number of additional signatures (shodan etc.) than you get your IP block and route permanently banned. There is only 1 exception to this rule, and we don't share that with anyone.

7. The site content navigation is programmatically generated in JavaScript

8. The legal notice is very real for some people

Have a nice day friend, =)

Who is "we"?

Plural of the deployment team admins.

Don't worry about it friend =3

> 73TB is a fair amount to have on a cloud

From the article:

"This was a bug in their crawler that was causing it to download the same files over and over again."

Must be a CDN choice issue, as most sites limit per IP daily file downloads, or have an account login with a quota limit.

People hitting the same files again sounds like a developer testing their code.

Reply on news.ycombinator.com