check out https://spider.cloud - llm-friendly markdown support, crawl dozens of urls in secs, proxy and headless chrome etc