Or, if you do, hand over the data in a nicely archivable form, and prevent the need for crawlers.