Meta Bad Bot – facebookexternalhit/1.1 | Hacker News Reader

Meta Bad Bot – facebookexternalhit/1.1

3 points_ir0i2 comments

My website is receiving a massive amount of visits from the Facebook bot. The user agent varies and can be: facebookexternalhit/1.1 and meta-externalagent/1.1 There are several IPs, but practically all of them are IPV6, as in the example: 2a03:2880:13ff:18::face:b00c

My website is very small and has few pages and few accesses, probably not even 20 people per day. But I must be receiving at least 50 thousand hits from this bot, it seems that something is misconfigured and I don't know how to report this to Facebook/Meta. Maybe they are using this data to train Llama, I don't know, I just know that it is certainly wrong behavior on their part as there is no need to visit the same page several times a day.

View on HN

Here are some resources:

https://developers.facebook.com/docs/sharing/webmasters/web-...

The article states that these may bypass `robots.txt`, so you may not even bother with that.

There are people who use facebook as a proxy for scraping with the preview functionality.

https://datadome.co/threat-research/how-facebook-was-used-as...

>With technical detection only, the fraudulent requests would have been indistinguishable from legitimate API calls, since they had the Facebook user agent and IP address. However, our heuristic analysis uncovered that certain parameters, unlikely to be used by humans, were overrepresented in the URLs that Facebook requested. Certain use cases were also unlikely: for example, a human user would typically share a link to a specific ad, not to a page of search results.

According to that article, looking at the requests and discerning patterns of bogus traffic would be preferable to more drastic approaches such as blocking or rate limiting indiscriminately, and risking to impact legitimate traffic.

I added a robots against facebookexternalhit/1.1 and after a while the hits stopped. An interesting point is that my website had an error regarding the canonical url and the og:url meta, which basically created a link to a non-existent page. If the page /x was accessed, og:url would be /x/y and if /x/y was accessed then og:url would be: /x/y/y and so on, perhaps this was one of the causes. Although Facebook accesses were repeated on the same pages already accessed.

Reply on news.ycombinator.com