Bad bots account for most internet traffic? Analysis

> Note that "Bad Bots" are never defined

This stuck out to me as well. What are "bad bots" and how do they differ from "good bots", and how are the authors determining which is which?

> world with Netflix and YouTube

Not in a world where pron exists.

"The top five categories of Bad Bot attacks are fake account creation, account takeovers, scraping, account management, and in-product abuse."

- second paragraph, first sentence

The actual report from Arkose (linked elsewhere in the comments) is much more detailed in what they’re measuring and reporting.

That said, the language around the “73% of traffic” figure is still a little ambiguous but most plausibly means “73% of requests/sessions to the significant commercial properties we studied”

Assuming that’s what they mean, it’s reasonable that bots are banging on major sites three times more often than legitimate users.

I think the linked article is doing the sloppy sensationalizing and the report appears to be much less so.

To build on the above, it's worth thinking about the transition that happens between those two quotes:

> Scraping, it must be said, is a legally murky area. It is not specifically illegal

There's already a jump happening here from "it's not specifically illegal" to "it's a gray area because it's immoral." Even if it was certainly and unambiguously immoral, that does not make it automatically a legally murky area. In fact, scraping seems to be reasonably established as specifically legal.

So the article is already kind of edging away from reality, where they acknowledge "this is not a crime" but they want to blur that and say that it's legally dubious, even though what they really mean is that they believe it's immoral.

Which, fine, I don't respect it but whatever. But then we jump to:

> But it is purpose built to commit crime.

Wait, no. Scraping is not specifically illegal. This is the emulator debate again; we have something that really is strongly implied to be legal and has won court cases before. It's not like it's an untested gray area, we know that there are scenarios where scraping websites against the permission of the website owner is not illegal.

So the article pretends it's more of a gray area than it actually is, and then once it has thrown doubt on it, drops the pretense entirely and calls it illegal. Not all scraping is fraud and not all scraping is illegal, and it's pretty galling for the article to admit that scraping is not illegal and then to turn around and pretend that they didn't just admit that.

> It is also a good example of crime-as-a-service.

No, it's not. The article admitted that scraping is not specifically illegal. Tools to help scrape websites are not "crime-as-a-service" because objectively (and there is legal precedent backing this up) scraping is not a crime.

----

The propaganda strategy being employed here is to admit a piece of factual information (scraping is not illegal), however hesitantly, and then to pretend that it was never admitted.

If anyone calls the propagandist out, they can point to their admission and say, "no, I acknowledged that it's not illegal, see". But then for every other part of the conversation, they just ignore that and act like it is illegal, even though at best all they've put forward is that they think it's more dubious than courts currently recognize.

They concede the point only to the degree they're forced to and only to the degree that would prevent someone from saying that they're lying or ignoring reality. But as soon as the context changes, they go back to acting like the point was never conceded.

This has the benefit of not only allowing them to make the same arguments they wanted to make before, but also sort of trains listeners to think of the original conceded point as more murky than it actually is. "Sure," the reader thinks, "it's technically not illegal, but at the point we're talking about crime-as-a-service and illegal tools, surely there's something legally dubious going on, this must be more of a gray area" -- even though not only has the article not backed up that idea, it's actually admitted that the idea is wrong.

> Note that "Bad Bots" are never defined, and that 73% will never be explained, either to say how they mean it

Arkose runs a bot detection service, OpenAI being one of their customers (check your requests on chat.openai.com). I’m guessing the 73% are the number of requests/sessions that fail their JavaScript fingerprinting.

Not sure how reliable these numbers are as I have a Go library specifically for passing their fingerprinting in pure Go (No JS) and it has been working for a while without much change

Bots might "watch" Youtube for ad money for the creators and Google, though.

I mean, it would not surprise me at all if bot traffic on Youtube is larger as in bytes/s of video, than actual visitors.

Remember that Google has no interest in disclosing how many bot views there are.

There was this news article about Spotify banning some musicians accounts due to listen farming.

https://www.musicbusinessworldwide.com/great-big-spotify-sca...

Since the payouts are not tied to the individual watcher or listener, scams run rampant. I guess the temptation to manipulate the money flow is way higher than having a "fair" distribution of each subscribers money to the artists.

I think it's possible if for nothing else but the fact that a bot can easily do 10-100x or more visits than a human can on the web.

I think that too.

I watched today a couple hours of Youtube videos. How many bots are needed to match that much traffic?

if we're talking about requests it makes sense. I just have to open my apache/nginx logs to see how many bots are attempting to get in.

I would definitely believe that Bots are responsible for %73 of the traffic. Not sure how one would necessarily characterize them as "bad" but definitely wasteful. Worked at site that published science. They collected page view metrics. Doing something simple like excluding page views that where done by IP addresses that continuously cycled through all articles resulted in over a %70 reduction in page views. Needless to say it was never deployed because the high view numbers made the authors feel good.

bots are still a subset of the traffic for streaming, porn, etc. Whether its for ad fraud or faking popularity on streaming those bots are running 24/7.

However I agree, on its face 73% sound ridiculous.

> people only do it when there's no API

Legitimate scrapers, maybe. Everyone else does it to circumvent the API limitations, by posing as real traffic. APIs imply API keys which can be traced and banned.

No, there’s a huge number of bots out there just doing stupid hacky stuff like trying for wp/admin/. Vast numbers of what I’d characterise as script-kiddy type attempts. Looking at the raw logs of a webserver not behind something like CloudFlare really opened my eyes to this. There’s an extraordinary number of garbage requests.

I think a more efficient approach would be to prosecute those who do it - bandwidth costs money - and to implement mechanisms to deter them. A not insignificant number of ai workers think are entitled to using people’s bandwidth, servers and intellectual property for free. Until a couple of such people are taken to task this wont stop.

> Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API.

Many sites intentionally make scraping as painful as possible because they don't want anyone scraping them.

You're better off using a combination of IP blocklisting and ratelimiting, both of which use some sort of binary tree to speed up lookups, rather than using a mega-regex which your solution proposes, which, with tons of backtracking, is very likely slower than serving the request would have been.

Not to mention that user agents are easily spoofable.

If a bot's user agent doesn't look identical to a normal user, either the bot developer is dumb, or they are relatively well behaving.

Anyone malicious who knows what they're doing will just send you a statistically common user agent string.

Under most http libraries this is a one-liner:

    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0'}

Try JA3 fingerprinting. It’s much more difficult to spoof than user agents

I propose: time spent offline.