Hacker News Reader

Comment by rnk | Hacker News Reader

I think they got overloaded or too many queries on their backend. No results for several different zip codes in my big city.

View on HN

OP here, experiencing the HN kiss of death. Load balancer 502s .I’ve increased the number of docker containers across the entire stack. Hopefully it copes up with the load. My aws bill will be hiiiigh

It's working for me now 24 hours later. The results are interesting, thanks for putting this up there.

same. I think today, OP learns about caching ;)

Can't speak for Zillow, but some API endpoints don't allow you to cache results you obtain from them, license-wise.

API ? You’re funny, I’ve been begging Zillow for their API and no response. I’m scraping using html parsers

Already using redis for the cache. I’m using a rotating proxy to bypass Zillow captcha, they’re not able to keep up with the load

It'll be sad if they figure out who you and your bots are, and then deny you specifically access. Got backup plans for when you get zero results for every query because of that?

Yes lol. Realtor.com is also scrapable so that’s the immediate backup. Long term I have to speak FTP csv files from a bunch of archaic MLS integrations. If I need to go that route, I need money.

To me, that seems a much simpler route than screen-scraping (yes, despite problems parsing CSV format).

I've been involved with screen-scraping of req's for <major airplane company> to produce spreadsheets for others in the same company, and I'll take FTPing the data->CSV almost any day.

* Especially when the FAA is the eventual consumer *

The bigger challenge is not the ftp integration per say. The fact that all the housing data is distributed across nearly 500 MLS sites that closely guard their data and make it available only to licensed realtors. I would expect each of these MLS sites to have different formats. So trying to strike up a conversation and allowing them to let me use their FTP site is the biggest hurdle. Zillow and the rest of the big guys already have their claws deep into the MLS sites and they probably want to closely guard their pipelines. Small guys like me don’t know where to start.

I do integration work for healthcare; you want an interface engine to pull the data from the ftp site, then translate/convert it into your desired format.

We do that all the time.

What’s your favorite tool for this? Just starting to get my feet wet in this world.

But I want to see Realtor.com vs Zillow numbers (vs MLSListings vs Redfin vs the others...). It's not like any of these are ground truth. How much are the discrepancies, where are they and why?

Also, is zipcode the lowest level of granularity (presumably not)? Do any of these services have finer granularity?

Or just get an injunction.

Seems like an uphill argument as long as the data isn't behind a login and you're not DDoSing the site.

https://news.ycombinator.com/item?id=31075396

> they’re not able to keep up with the load

Zillow can absolutely keep up with the load. Your proxies are being blocked by the WAFs that Zillow utilizes, usually PerimeterX. Also, based on your other comments, you have no idea how to architect a web application that can handle the trickle of HN traffic, and I'd wager any problems are more likely on your side.

I assumed the "they're" referred to the "rotating proxies" not "zillow".

Is correct. The proxy service… it’s slower on the responses from the logs

I mean, I think I have some idea. The problem is more on the affordability side. I had to scale up the containers in my cluster to cope with the increase in 502s. I live with the fear of a high AWS bill. Then there is the question of the proxy rotators data limit being breached, which happened! I topped it up for now. Now I’d rather spend time asking Zillow to allow me to exist rather than trying to architect a more efficient proxy mechanism.

This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response from Zillow/Proxies requires nearly zero cpu on your servers and it should be a background/async process. There’s only ~40k zip codes and only ~10k that remotely matter for even very small towns.

Find a shared proxy provider that doesn’t limit bandwidth. It’s usually about $0.50/mo per IP. Not that any of this really matters, you’ll still get blocked by Zillow. It’s not an IP limit l anyway, it’s a TLS fingerprinting or JS-based fingerprinting issue that is getting you blocked.

I'm using Puppeteer with Chromium to mimic a browser with the headers changed to look like a real user. The proxy service only handles the data transport. Running a browser cluster at scale costs me CPU, and memory (My peak yesterday on HN was 40-100 requests a second). Direct GET / POST requests to Zillow didn't work. I'm not sure $5 per month VM will cut it. Do you have the details on a provider that will give me a $5/m deal for this setup ?

Check out residential proxy services. Some of them have apps users install, others may be more... nefarious. They're cheap and you can rotate through them. Some may have clients that are a pain to work with, some may be SOCKS access.

https://www.webscrapingapi.com/top-residential-proxy-provide... obviously SEO'd page, but it has useful info.

Reply on news.ycombinator.com