Hacker News Reader

Great question! There was a fairly even distribution of slow-performing requests across the many user-agent strings coming from the Facebook ASN.

My hunch is that the Facebook crawlers send user-agent strings that Facebook has seen in requests to their own services. This allows them to crawl content in linked posts masquerading as a device that their users would actually use.

View on HN

> My hunch is that the Facebook crawlers send user-agent strings that Facebook has seen in requests to their own services. This allows them to crawl content in linked posts masquerading as a device that their users would actually use.

Very interesting theory, I’d love a follow up post if this can eventually be confirmed or denied :). That would be very clever of FB but very, very rude to all of the small fish they crawl.

Reply on news.ycombinator.com