It sometimes depends on what sort of data you are trying to get. Hipmunk could be crawling travel sites. Mint couldn't have been crawling banks as those are protected, unless people were giving them the passwords to their accounts.

Are you familiar with web scraping? Depending on the volume, and type of data you are trying to get, it might work for you. It should at least get you through proof of concept.