Ask HN: Created Scraping SaaS, best way to avoid misuse?
I created a service that's in very early alpha that's able to use lots of different techniques (pattern matching, NLP,javascript) to get data out of webpages.
For data gathering, I planned on having an interface to different kinds of APIs, along side the scraping, that users might need to do mashups.
It's already paying the bills for me, so I'd like to start scaling this out now. There can be a lot of potential for misuse here.
Besides the obvious things like robots.txt, naming my user agent, and users being able to opt out of things like emails gathered by the system and throttling. What other things should I should be aware of?