Web Scraping: Data for Everyone | Hacker News Reader

Web Scraping: Data for Everyone

omarkama.li

84 pointsomneity25 comments

Are there any existing or developing solutions for visual web scraping? It feels like the only way to capture data from emerging WEI and canvas-powered websites will soon be screen readers designed for accessibility and automated GUI interaction.

You could also explore approaches like what puppeteer-heap-snapshot does and extract the data you're interested in from the in memory Javascript objects that the canvas powered website uses to generate its contents.

You mean something like octoparse.com or serpapi.com?

octoparse extracts data from DOM elements. It can't OCR or visually process a dynamic page. The page has to have a structured and predictable HTML DOM.

Unless I'm missing something?

And serpapi is just for google searches?

Optical Character Recognition unfortunately.

> canvas-powered websites

Is this really happening right now? Are people really rendering their entire websites into an opaque canvas? Decades of moving the web platform forward and it has come to this?

If it's a flutter web app, yes.

Reminds me of flash, kinda

Rare, but I was curious to know what the state of the art was. It's kind of like the "analog hole" in audio and video recording.

Makes sense to me.

Decades of bloated, backwards-compatible hacks in CSS, HTML, and Javascript makes a purpose-built, clean, bloat-free UI system feels pretty good actually.

This is what happens when you move the web "forward" for advertisers and exploitive data brokers instead of users.

Putting everything inside a canvas is exactly what advertisers and data brokers want. It basically turns your browser into a television. It destroys structure which significantly complicates ad blocking. Might also be illegal due to accessibility laws.

You're probably looking for RPA / Surface automation solutions. This used to be all the rage a few years back, supporting various "digital transformation" initiatives in large Enterprise.

UIPath [0] is a main contender in this space.

[0]: https://www.uipath.com/

UIPath doesn't have a visual web scraping module or automation, from what I have seen. You might be able to build a custom one and execute it with their orchestration/automation tools, but I don't see that any support for that out of the box?

UIPath[0] and many others do. It's actually super common, both needed to support desktop apps built using custom UI toolkits (from memory, xaml and others actually give you something DOM-like to hook into, but others like javafx don't) and it's also useful since a lot of enterprise software runs within VDI/VNC/... so all you have (at a low level) is a "screenshot" and the ability to click on coordinates or trigger keyboard events. [1] is what surface automation looks like in practice.

[0]: https://www.uipath.com/product/ai-computer-vision-for-rpa

[1]: https://images.ctfassets.net/5965pury2lcm/4hwXQpRTQkIIAy2q8k...

An interesting development is AI bots that can act as human agents and visit a site and extract information. I imagine these are only needed when there are barriers to use all other solutions, otherwise it’s a bit overlillt

And barriers there are... Lots of websites only allow the Google bot and no one else.

Search engines scrape data. Why not simply mandate that they have to make that data available to all?

I do not believe that Google would share anything if it was not required by law.

I heard there are other public community driven solutions, not necessarily search engines. For example marginalia search engine provides some domain data, but I think it was distributed as some sort of binary info.

I have developed myself an engine to scrap data, which is using beautiful soup. I export my data to a github repository, where it is available. All domains receive a page rating, based on their content. I believe that all pages should provide scrap-able title, description and a valid status code.

We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code.

Most of what I make available are just sample data for testing the search engine, it's a tiny fraction of the entire crawl set for purely logistical reasons. I don't have the bandwidth to upload datasets on the order of terabytes.

Though I've begun distributing dumps of the domain and link database if anyone is interested:

https://downloads.marginalia.nu/exports/

Might be persuaded to share the the document metadata databases as well in some form, as that's only order of 90 GB. Would contain titles and summaries and a few other flags for each URL.

> We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code.

Well there's already the WARC format for that.

As someone who has only heard about web scraping but has had no idea about how it works this has been a really interesting read! Very easy to understand. Thanks!

Hey, your comment made my day! This is exactly why I write. Thanks!

You have unlocked the key to making the front page of YC, post about web scraping.

That's an interesting observation indeed.

Reply on news.ycombinator.com