Hacker News Reader

Comment by nhggfu | Hacker News Reader

someone got tweak we can use for puppeteer to achieve this effect? [kinda LOL @ asking people who SCRAPE DATA for free to pay for an API, no?]

View on HN

There's actually an (admittedly small) data scraping industry.

Let's say you are Wal-Mart, and you'd like to know which of the products you sell are available cheaper at Target. Or which neighbourhoods they deliver to that you don't, or which stores have longer opening hours than yours, or whatever.

You can't legally exchange data with Target directly, that would risk making you an illegal price-fixing cartel. You can legally visit their website, but you don't feel like matching up 10,000+ products.

Instead, you call up a business intelligence firm who has already scraped your site and theirs and matched the products up. They'll send you a CSV, for a price.

You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode.

edit: Firefox Nightly because it gives you much deeper access to Firefox internals.

interesting. would love to see some examples / code

I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients.

That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering).

[0] https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html

What are some sites that have good bot detection? I'd be interested in just seeing how my scraper holds up against them.

Kasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.

Reminded me the the Kasada VM reverse-engineering article: https://news.ycombinator.com/item?id=34285747

Reply on news.ycombinator.com