This is great, love it!

Crawling sites to index the FAQ's and knowledge bases, into the vector search, isn't as intimidating as it sounds, at least on linux systems. Sometimes a thin wrapper function over plain old wget will get you 99% of the way

    wget -rnH -t 1 --waitretry=0 'https://{{domain}}' -P '{{domain}}'