Excellent, another kind of copyright theft as a service that assumes your site is ripe for scraping unless you disallow yet another whackamole agent — FireCrawlAgent — that you won’t find out about until it’s too late. One tiny line in the README passing on responsibility to those who use this CTAS is the only nod to the idea that a site might not want to get ~sc~raped all while the specific agent string conveniently doesn’t appear in the docs and arming yourself against their presumption of default consent requires a search of the source code.
Interesting... Looks like it would be good for RAG. Maybe add Ollama support for local hosting?