Crawler, AI or not, cannot afford to pay per visit. The entire model of crawling works because the incremental cost of each crawl is so low. Even fractions of a penny would be prohibitive.