Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

Dupe (885 points, 513 comments) https://news.ycombinator.com/item?id=44785636