Comment by imroot | Hacker News Reader

Comment by imroot | Hacker News Reader

Parent Full thread

Getting and keeping real estate listings up to date is a big pain in the ass. There's no standardized residential listing data, and while you might get RETS feeds or data dumps from the MLS nightly, it's still on you to ensure that all of the information is parsed correctly and displayed in accordance to the MLS'es standards (or risk being cut off).

Often, MLSes have terms in their contract that say that regardless of the source, that their information must be the canonical source for information in some cities -- which is a huge problem with dealing with REO'ed (Real Estate Owned) properties and the banks that want to sell them.

I know of a startup that eventually just resorted to scraping realtor.com (and I'm sure that they're still doing it to this day) instead of dealing with the various headaches of managing the contracts with the MLSes and RETS providers.

It's a nightmare. I used to work for one of the industry dinosaurs; they have an MLS team that was as large as their development team. They pull data from over 600 MLSes, with different requirements for each. Managing 600 business contacts is a big pain, then there are different data conventions that change without notice. They have had a long-standing arrangement to feed data to Trulia too.

The word is most MLSes use CDNs, so scraping might not be the best way.

which dinosaur?

Reliance

Supposedly in January there will be a new "standard" so vendors can stop mapping individual MLS fields

http://www.realtor.org/policy/mls-policy/real-estate-transac...

Link doesn't work -- do you have another source?

you gave me flashbacks thinking about "RETS", and how many times, it was a XML formatted CSV.

thankfully, someone else was managing the contracts, and we had built a fairly decent feed management system.

i still have nightmares, though.

yep, and even something as 'simple' as images is difficult. If an image gets removed from the feed, it typically must be removed from the listing page. Tons of little things like this make the data incredibly prickly.

aren't they still at risk of displaying mls data in a non-canonical format?

If they're scraping the site, then I'd assume they backed out of their contract for the MLS dumps and thus don't have a contract that says they have to display data in any format.

Reply on news.ycombinator.com