Does Mint.com use screen scrapers? | Hacker News Reader

Does Mint.com use screen scrapers?

7 pointsfatalerrorx36 comments

Just curious if anyone knows the answer to this question..I'm pretty sure when it first was started they were basically scraping these financial institutions, but I don't know for sure. If they were initially screen scraping, do you think by now they have their own special access to the financial institutions that they deal with?

The reason why I'm interested is that I'm currently working on a site that's similar in nature but for a totally different industry and I'm worried that the websites I'm scraping from will be updated and will no longer work...I'm not sure how often the websites in question are updated but my guess would be that as the industry evolves over time the websites will be getting updated more frequently and it's currently only a 1-man engineering team (me)

View on HN

No, they used Yodlee initially, and now use the Intuit aggregator. However, Yodlee does some scraping, but rumor was (this was 2008) scraping was less than 10% of financial sites and they had more direct access to other 90%.

Thanks, I remember hearing about Yodlee when reviewing Mint...in the industry that this new service is targeting there is not very many things that are standardized, so there isn't currently a platform that I can use to make integrating with them easier...I wish there was but then again that's half the fun =D

I'm worried that the websites I'm scraping from will be updated and will no longer work

When I worked for Internet Brands, they bought a site called Tjoos (a coupon/deal site) that amassed what was at the time the largest number of merchant sites using a proprietary screen scraping technology. It is a very fragile method, and a good portion of the codebase was dedicated to config classes that needed to be updated with each change to a given site. The founders had to keep hiring developers to keep up with the demands of constantly updating scraping definitions, so the site had a lot of revenue, but little profit when IB bought it. Surely by now Internet Brands has updated the site to use the same engine that powers UlitmateCoupons, BensBargains, DealLocker, etc.

The newer coupon/deal aggregators are most likely using RSS feeds which is a more standardized format when compared to screen scraping, and thus less chance of breaking. There are still a surprisingly large amount of websites that are still stuck in the 90s in terms of data portability though

There was a previous discussion of this last year: http://news.ycombinator.net/item?id=1537825

I also recall (but can't find) another discussion of screen scraping where the consensus seemed to be 'be very careful of the legalities'. If you are small you might fly under the radar, but if you are successful you might just be opening yourself up to a whole can of worms.

Regardless of the outcome of any battle over this, the battle itself would be a severe distraction.

Good advice, I read the discussion you posted and there was a lot of insightful tips as well, which I've kind of worked out along the way. In terms of legality, this service will be scraping on behalf of the user, meaning they provide their login credentials and are granting permission to the server to perform the scrape..We'll eventually need to have a Terms and Privacy section which covers the possible legal issues, this is still in the prototyping phase but it's good to know before launch what is allowed and what isn't by each website

Reply on news.ycombinator.com