The same question applies to Blippy and inDinero. Anyone know what these guys are doing on the back end to get the transaction data?
The typical question is, why do Mint and inDinero use Yodlee instead of building the solution out themselves?
1) Security Liability. No startup should ever have to deal with the problems that go with storing passwords to financial accounts. Yodlee is in the business of security, they have direct feeds with major banks, making it much easier (and safer) to just integrate with them.
2) Mass quantity of banks. Screen scraping from so many banks is a pain in the *. It isn't standard either -- compare the bank website of Wells Fargo to that of a local credit union that asks 5 security questions upon login. In short, it's a brutal nightmare.
3) None of our businesses are in the business of screen-scraping. If Mint had to spend the first year of business integrating with banks, they wouldn't be successful. And even once the integrations are done, you have to maintain them in the event that the bank changes their login page or interface. In short, it's not worth any startup's time to do manual screen scraping themselves.
Would be happy to discuss further if you DM me.
The thing I've found frustrating about financial planning tools is that they seldom give me any useful information about the purchase I want to make right now. Instead, they tell me about the malleable resource called money, and offer me the chance to look at it from a 3rd person perspective.
I've seen a number of startups use Yodlee or CashEdge and I was always curious how they could afford this or how the setup worked. I can't imaging they pay what Mint was reported to have paid (about $2mil/yr) to license the software.
I had a few months solid of nothing but scraper-building.
Besides, what's Google if not a screen scraper? :D
If anybody thinks it's "silly" they probably have a characterization in their mind that's not really how it works in the wild when done skillfully.
I'd really like to hear more about the current state of the art (without you telling any company secrets). I have quite a lot of experience scraping utility and government websites (no javascript) in perl + LWP... but I'm getting a little tired of perl and am looking to give a new toolset a try. Perferably one that can handle a broader range of modern websites.
libxml_use_internal_errors(true);
This little function call is the secret to solving what seem at first to be intractable memory-leaks. The trouble is the scraper uses libxml, and libxml issues a notice/warning every time the HTML is malformed. Without this call, those errs will bubble up to the PHP error handler and it'll murder performance and memory usage.
One more i suppose...
If you scrape inside a loop (and unless you're using a distributed job queue,if you're scraping more than one URL at a time you almost certainly are), missing a unloadDocument() call is going to cost you each time you iterate. The objects it creates, IIRC, have some circular dependency issues and if you don't explicitly unloadDocument you'll run into trouble. (Suppose it should be OK tho if you've enabled that GC feature in PHP 5.3)
And, generally, a tip... sometimes it's tempting to write a simple regex instead of a chain like, say, pq($this->node->find('a')->get(0))->attr('href')
Write the chain. Regex is just too brittle.
I'm pretty intrigued by a library that can apparently handle ajax/json updates and content creation. Heh... I thought PHP was only for page generation and had no idea you could purpose it for something like webscraping.
So it should be fun playing with it.
My email's in my profile if you ever want to talk shop.
edit: we also built and deployed to production every night, so we had no problem keeping up. Sometimes we'd even deploy midday if we felt like a fix needed to go out immediately.
1) Write a program that can load webpages as if it was a user of the site.
2) Have it save everything it loads.
3) Write a program that can extract the data you care about out of the html and put it into a more useful format (or into a database or something)
Many languages have libraries for this or you can use a tool like cURL or wget. I do this a lot with perl and the LWP family of modules but the sites I work on don't use javascript or dom manipulation... There's so much javascript and ajax out there now though that I'm not sure if you can scrape those kinds of sites with perl.
But basically, 'look for the number in this div region, this is the account balance'.. etc.
HA! Just kidding about the regex.
I think this would be illegal in the UK under Computer Misuse Act, or somesuch, as it's unauthorised access to a computer system; the user doesn't have the authority to grant others access.
curl -d "username=foo&password=bar" https://yourbank.com | grep balance
http://www.yodlee.com/security_overview.shtml
A lot of the "how" is meeting security wickets (physical, application, transport, audit, examination).