we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time