Hacker News Reader

Comment by Minor49er | Hacker News Reader

While interesting and well-intentioned, the advice in the article can cause issues. The agent isn't downloading CSS or images? Could be a blind user. Pages being downloaded in quick succession? Could be browser prefetch. Lack of mouse movements during navigation? It could be a user on a mobile device or screen reader.

What if the bot is making individual requests from unique IP addresses? What if it's scraping many pages over time rather than a single smash-and-grab approach?

The article admits that this is a hard thing to solve. In most cases, it's probably not very worthwhile to try to detect bots in the ways that are suggested. Focusing on patterns in the server logs to find out what's being targeted might be more beneficial. Then slap a login or captcha around anything valuable that bots shouldn't have access to

View on HN

I use a keyboard-driven browser to reduce wrist strain, and I sometimes get tagged as a bot, probably for lack of mouse movement.

basically I've written bots for scraping governmental sites, and fashion sites. I've never yet had a problem (although the code on one fashion site looked like all classes were randomly generated with implied some attempt to keep scraping from working), so what I'm wondering is - what kinds of sites actually go through the trouble of tagging you as a bot, since you've got experience care to share?

Ticketmaster

Good response, I definitely never though of that before. Serious case where you don't want Bots buying stuff up.

Ages ago I was one of those bots. Most fun I've had playing cat-and-mouse. Their team was actually quite good. And I learned that Perl can solve any problem.

Have you written about this? Sounds like an entertaining story.

I have not. But I will and will post to HN.

Yandex Search does it to me regularly. Sometimes it takes 5 attempts to get through the captchas.

I joke that I only have two real skills. Asking questions and identifying good ideas.

In the latter category I have heterogenous load balancing, segregated by traffic type. For example, if you have a read-mostly web application, often the most expensive parts are in the administrative functions. If they run on separate hardware, then a regression doesn’t affect everyone, and a client side bug or flood of traffic doesn’t automatically lock people out of the admin workflow.

Similarly, you have server side rendering for web crawlers and anyone with JavaScript disabled for Angular and other interactive frameworks, which generally involves running a small quantity of specialized services on a few machines.

Shunting all users who smell like a bot to a small set of machines doesn’t lock them out, but it does preserve the experience for everybody else.

I somewhat agree, but those can be good signals though, you just have to gather statistical data on them from known good users and combine with other signals. Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low. The same way access patterns alone don't give you much, but are good against scrapers when combined with other signals.

And putting captchas or logins on anything you have to protect from bots also "protects" from real people, most of them simply leave and don't bother with those thing.

> Like what are the chances for an agent to be used by a real person if it doesn't download images, lacks mouse movements and, for example, is coming from a datacenter? Obviously pretty low.

Congrats, you just identified a visually impaired user with a screen reader who uses a VPN through AWS/GCP/etc.

And your point is?

that depending on your sites purpose and where your servers are located / where your main customer base is you might be opening yourself to extremely negative public relations by blocking this person as a bot, or even legal repercussions.

This is completely irrelevant to the conversation about technical solutions. My example is just about probabilities and how to use them for bot detection.

If that logic is extended, the true technical solution is to turn the site off entirely.

What exactly in that logic in any way leads to turning off the site entirely? It's literally the opposite: narrowing down the bots by calculating probability of them not being real people using multiple signals and like Bayesian inference or something like that, it's pretty much a primitive machine learning.

I'm surprised there are so many trolling comments here.

well if that was the case then the parent comment pointing out you could have had a false positive certainly seems relevant, but you wanted to know what their point was. I thought their point was just that these false positives could create negative consequences for anyone implementing a bot preventing solution on their site.

In the context of describing an idea to minimize false positives, pointing out a false positive in the given example is at best nitpicking, it's definitely not relevant.

The ADA exists and for good reasons.

Reply on news.ycombinator.com