Hacker News Reader

Why do you think TikTok is ignoring robots.txt? This site is disallowing a lot of crawlers via User-Agent matching, but not ByteSpider.

The domain serving the images is allowing everything:

So I decided to google around and I think TikTok does ignore robots.txt. A few posts from around:

I just assumed that would have been the first thing they tried…