Hacker News Reader

> Headless browser scraping is between 10x and 100x more resource intensive, even if you carefully block requests and cache resources.

Instead of setting up some kind of partnership with our vendors, where they just send us information or provide an API, we scrape their websites.

The old version ran in a hour, using one thread on one machine. Downloaded PDF's and extracted the values.

The new version is Selenium based, uses 20 cores, 300GB of memory, and takes all night to run. It does the same thing, but from inside of a browser.

As a bonus, the 'web scrapers' were blamed for every performance issue we had for a long time.

On the topic of JSON and whatnot that reminded me of this awesome post[1] about looking within heap snapshots for entire data structures where they may have a lot of nicely structured data within it not readily available from more direct URL calls.

[1] https://www.adriancooney.ie/blog/web-scraping-via-javascript...

(Previous comments about it from 2022 here: https://news.ycombinator.com/item?id=31205139)

It also depends on what the goal is. If you need to extract some data from a specific site (as opposed to obtaining faithful representations of pages from arbitrary sites like a search engine might need to), then SPAs might be the easiest targets of all. Just do some inspection beforehand to find out where the SPA is loading the data from, very often the easiest thing is to query the API directly.

Sometimes the API will need some form of session tokens found in HTTP meta tags, form inputs, or cookies. Sometimes they have some CORS checks in place that are very easy to bypass from a server-side environment by spoofing the Origin header.

I do a lot of scraping at my day job, and I agree the times we need to use a headless browser are very rare. Vast majority of the time you can either find an API or JSON endpoint, or scrape the JS contained in the returned SSR document to get what you want.

Headless browsers are expensive, but if you needed them, I suspect there are easy wins to be had on the performance front.

For example, you can probably skip all layout/rendering work and simply return faked values whenever javascript tries to read back a canvas it just drew into, or tries to read the computed width of some element. The vast majority of those things won't prevent the page loading.

Which way to go will depend on your use-case and what websites you want to scrape.

For occasional web crawling, headless browser is great as it's easy to set up and you're almost guaranteed to get the data you need.

For frequent or large-scale crawling, it's a different story. Surely you can implement something with just HTTP library, but you'll need to test it throughly and make some research before hand. That said, most scraped, content-heavy websites use either static HTML or SSR, in which case you can use HTTP no problem.

Depends on what you want with the data I guess. From a search engine point of view, the SPA case isn't that relevant, since SPAs can't reliably be linked to and in general tend to not be very stable and it's overall difficult to figure out how to enumerate and traverse their views.

I think a good middle ground might be to do a first pass with a "stupid" crawler, and then re-visit the sites where you were blocked or that just contained a bunch of javascript with a headless browser.

The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content.

If Google can crawl it, then you can too. And while Google doesn't use a headless browser (or at least I assume they don't) they absolutely do execute javascript before loading the content of the page. And they execute the click event handlers on every link/button and when we use "history.pushState()" to change the URL Google considers that a new page.

You're just going to get a loading spinner with no content if you do a dumb crawl (I disagree with that and think we should be running a headless browser server side to execute javascript and generate the initial page content for all our pages... but so far management hasn't prioritised that change... instead they just keep telling us to make our client side javascript run faster... imagine if there was no javascript to execute at all? At least none before first contentfull paint)

> The SPA's we create can be reliably linked to (the current URL changes as the user moves around, even though the page hasn't reloaded) and they are "stable" because our business would go bankrupt if Google couldn't crawl our content.

This is true for some SPAs, but not all SPAs, and there's not really any way of telling which is which.

I don't personally attempt to crawl SPAs because it's not the sort of content I want to index.

I have a pet theory that there are two forms of the web: the document web and the application web. SPAs have some very attractive properties for the application web but complicate/break the document web.

That being said, with sites like HN, Reddit, LinkedIn, Twitter, news outlets, etc. the lines between “document” and “application” get blurred. In some ways they’ve built a micro-application that hosts documents. Content can be user submitted in-browser. Content can be “engaged with” in browser. Some handle this blurring better than others. HN is an example IMO of getting it right where nearly everything that should be addressable (like comments) can be linked to. Others not so much.

(As an aside, I love marginalia!)

For application websites like the ones you listed, you'd typically end up building a special integration for crawling against their API or data dumps. This is also true for github, stackoverflow, and even document:y websites like wikipedia.

It's simply not feasible to treat them as any other website if you wanna index their data.

Site owners adding browser fingerprinting explicitly is vanishingly rare however site owners who sit behind cloud flare who very likely fingerprint browsers is very common.

The issues you list can probably be split into 2 main issues

- The host has a tendency to block unknown user agents, or user agents that claim to be a browser but are not

- Anything that requires client side rendering

I'd suppose both problems are more pertinent in 2023 than they were in 2012.

At web scale, the issue appears on at what point would you be required to use a headless browser when not using one in the first place e.g. if React is included/referenced. Perhaps some simple fingerprinting of JS files would do, IMO in reality the line is very blurry, so you either do or don't.

I've always wondered why not to use request interceptors, get the html/json/xml/whatever url and then just call them.

If you need cookies/headers, you can always open the browser, log in and then make the same requests in the console, instead of waiting for the browser to load and scrape the UI (by xpath, etc.)

sounds weird going in circles: - SPA call some URL - SPA use the response data to populate the UI - You scrape the UI

instead of just calling the url inside the browser? am i missing something?

Is there like a custom performance based headless browser

How are they scaled?