Maybe its only needed for AI review as opposed to generic google search results? Just realized one of the website we made already has a robots.txt, we were just unaware about. Thanks for sharing
I find this a bit hard to believe that they refuse to index it at all if the robots.txt doesn't exist. Would love to see more data or confirmation on this.