I would generate a JSON feed for the site and upload that (I am curious if this can detect duplicate records. Edit: it can using the UPSURT call), that way you could even have full text search.

A crawler is probably not the best tool for this if you use some form of site generation, as you can also use it to generate the information that goes into Typesense.