Hacker News Reader

Comment by afandian | Hacker News Reader

One thing I'm interested in is scraping those annoying sites that require JavaScript execution. More and more webpages are requiring JS even to display anything beyond a blank page. These sites self-select themselves for exclusion in scraping like this.

I've looked into Headless Chrome, but I'd be interested to see a 'scraping framework' level abstraction for those sites.

View on HN

If you're targeting a specific site (as opposed to blindly spidering multiple), all JavaScript is actually better for scraping, in my experience. JavaScript apps communicate with an API 99% of the time, so to scrape them you can just replicate the API requests. And as a bonus you'll get nicely formatted JSON responses; no need to parse fragile HTML.

Similarly, I've found that most sites worth scraping also have a mobile app, which you can run through a MITM proxy, and then simply write a scraper to call the API endpoints directly.

This is absolutely true. This is why SPAs are awesome to scrape

You might have missed puppeteer[1], if you don't mind writing Javascript it seems to provide a simple interface for scraping.

[1] https://github.com/GoogleChrome/puppeteer

Scraping JS-only sites is also possible without a headless browser, but requires a bit more debugging of the internal structure of these sites. Most of JS-only websites have API endpoints with JSON responses, which can make scraping more reliable than parsing custom (and sometimes invalid) HTML. The drawback of headless browser based scraping is that it requires significant amount of cpu time and memory compared to "static" scraping frameworks.

As mentioned, Puppeteer was used in this project, which is Chrome based.

I've also used Selenium (via Python -- I used BeautifulSoup to parse the resulting HTML) in the past for precisely for the reasons you stated. Selenium uses "web drivers", which lets you start other headless browsers as well (Firefox, Opera, IE, etc.).

http://www.seleniumhq.org/

http://selenium-python.readthedocs.io/

All it took was a couple of lines of Python code...

"More and more webpages are requiring JS even to display anything beyond a blank page."

Can you provide some example webpages so we can take a look?

Also, I agree with @asciimoo's point about endpoints. One could make an argument that compared to the website design of the 1990's and 2000's, retrieving structured data from websites is actually getting easier not more difficult. I recall one period of time where the trend was to design websites entirely in Macromedia/Adobe Flash.

Here is what is missing from this project (and many others like it): when providing software that performs text processing one needs to not only provide example code but also example output. This enables a user to quickly compare her current text-processing solution with the software being provided without having to install, review and run the unfamiliar software.

For example, without some sample output she cannot test, e.g., whether her current solution produces the same output faster or using less code.

With a minute's searching, here's a webpage that's meaningless without JavaScript:

https://blogger.googleblog.com/

It's arguably a page of text not a web app that needs user interaction.

I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons.

And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.

The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript.

Blogger has a feed. Here is one way to retrieve it, in two steps.

     1. get TargetBlogID
     2. retrieve data
     optional:
     3. format data for viewing

     Example:

     1.
     x=$(exec curl https://blogger.googleblog.com \
     |exec sed '
     s/\\046/\&/g;
     s/\\46/\&/g;
     s/\\075/=/g;
     s/\\75/=/g;
     /targetBlogID/!d;
     s/.*targetBlogID=//;
     s/&.*//;
     '); 

     2.
     curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 

     3. 
     exec sed '
     # ^M is "\r"
     s/^[0-9a-f]*^M//; 
     s/&lt;/</g;
     s/&gt;/>/g;
     s/&amp;/\&/g;
     s/&quot;/\"/g;
     1i\
     <br><br>
     
     s/<name>/<br><br>name &/g;
     s/<uri>/<br>uri &/g;
     s/<generator>/<br>generator &/g;
     s/Blogger//;
     s/<id>/<br>id &/g;
     s/<published>/<br>published &/g;
     s/<email>/<br>email &/g;
     s/<title type=.text.>/<br><br>&/g;
     s/<openSearch:totalResults>/<br>total results &/g;
     s/<openSearch:startIndex>/<br>start index &/g;
     s/<openSearch:itemsPerPage>/<br>items per page &/g;
     s/<updated>/<br>updated &/g;
     s/<thr:total>/<br>thr:total &/g;
     s/<\/feed>/&<br><br><br>/;
     s/^M*/<br>/;
     ' y.htm \
     |exec tr -cd '\12\40-\176'

It's almost like you asked that question to bait someone into giving you an excuse to show "just how easy it is" with some cli hacking.

Just 3 easy steps

Feedly, Netflix, Google Maps, as of this article:

https://sonniesedge.co.uk/blog/a-day-without-javascript

I remember that article.

I do not use any of those three websites so I would be curious to know some specific examples of what the desired structured data looks like.

If for Netflix the desired data is only DVD summaries, I recall that these were easily retrievable without Javascript not too long ago.

I would argue that it's reasonable for these examples to require JS. My complaint above was the accessibility text-only sites.

I would agree. That article had a very poor choice of websites, IMO. The web is quite usable without Javascript IME.

I know of a site where all the data is embedded inline ("var Person = {firstName: 'Fred', lastName: 'Smith' }") then rendered to HTML with JS. No endpoints, Javascript not JSON, complete pain in the ass to extract without rendering in a headless browser first.

Isn't this better? Why do you need it render when they're giving you basically JSON of the data (might need to quote the keys in strict parsers), just extract that.

The short answer is that, yes, the only parser available to me is that strict. If I'm going to jump languages, I might as well go the whole hog and jump to Puppeteer.

> Can you provide some example webpages so we can take a look?

learnopengl.com

curl -d 'content_name=Heading' https://example.com/content_load.php?

curl -d 'content_name=Heading/Subheading' https://example.com/content_load.php?

  s/example/learnopengl/
  s/Heading/Introduction/

Wix. Everything with Wix.[1]

[1] http://www.wix.com/stunningwebsites/400

I looked at some exemplary Wix sites since I rarely ever encounter these.

The ones they identify as "stunning" seem to be mostly devoid of text and are mainly images and videos.

As others in this thread have noted is common on today's web, they use json endpoints. Some sites also have an optional XML feed activated at /feed.xml.

Viewing the images and videos is possible without any use of Javascript. Simply extract the image and video urls and their descriptions from the json.

Below I only demonstrate how to extract the text, as html. One could view this html in a browser, without Javascript.

Random wix site: www.cricketcanadakids.com

     curl -o x.htm https://www.cricketcanadakids.com

     # produce a listing of the page titles and json endpoint urls
     # in addition to a title, each page has a "pageID"
     # beginning on 3rd line is CSV e.g. for database: 
     # ID, title1, title2, url
     # last line of CSV is "master" json
     x=https://static.wixstatic.com/sites/; 
     exec awk '{gsub(/:{/,/"\n"{/);print}' \
     |exec sed -n '
      s/{/\
      &/g;
      s/\],/\
      &/;
      /<title>/p;
      /mainPage/p;
     ' \
     |exec sed '
     /var publicModel/d;
     /.*\"baseUrl\"/d;
     /{filename}/d;
     s/<title>//;s/<\/title>//;
     /ExternalBaseUrl/{s/.*Url\":\"//;s/\".*/\
     \
     \"ID\",\"title1\",\"title2\",\"url\"/;};

     s/\\//g;
     s/{\"pageId\":\"//;
     3!s/\",\"/,/g;
     s/title\"://;
     s/,pageUriSEO//;
     s/:\"/,/;
     s>pageJsonFileName\":\">'"$x"'>;
     s/\"}.*/.z/;
     s/\],\"mainPageId\",//;
     s>masterPageJsonFileName\":\">master,master,'"$x"'>;
     /topology\":\[/s/json.*/json.z/;
     ' x.htm

Random page from the site: coach-profiles. JSON url taken from CSV above.

     curl -4o y.json https://static.wixstatic.com/sites/7f1cbe_7131eb80aa297a10c02a08c8ffbc3ef6_122.json.z

     # produce simple html version of page
     # text-only, no images or videos
     x=$(exec echo b|exec tr b '\34');
     exec tr -cd '\12\40-\176' < y.json \
     |exec sed 's/{/'"$x"'&/g;s/ *//;' \
     |exec tr -d '\12' \
     |exec tr  '\34' '\12' \
     |exec sed '1d;$d' \
     |exec sed '
     s/</\
     &/g;
     ' \
     |exec sed 's/\\["tn]//g;s/\",\".*//;/{/d;'

Improve the CSV:

   # add to end
   \
   |exec tr '\15' '\12' \
   |exec awk 'NR==1{a=$0}NR==3{b=$0}NR==5{print}NR>=6{print "\42"a"\42,"b","$0}' 

   # update CSV header line 
     \"sitetitle\",\"url\",\"pageID\",\"pagetitle1\",\"pagetitle2\",\"jsonurl\"/;};

Probably would be better to change "jsonurl" to "jsonfile" and just save the filename. The location (URL) of this file may be in several locations or it may change over time.

You'd need to parse and run the JavaScript, including a virtual DOM. You'd also need to support any browser events that the page depends on. It's conceptually simple, and getting support for 80-95% of cases is probably easily doable, but I imagine a long tail of cases where your browser emulator isn't quite compatible in an important way. I know Google and others do this to index the web, but in not sure about any existing open source projects.

I released Page.REST(https://page.rest#prerender) couple of weeks ago. It will pre-render JS based sites - so you can then extract rendered content using CSS selectors.

Ditto: https://www.prerender.cloud/docs/api

  // URL to screenshot
  service.prerender.cloud/screenshot/https://www.google.com/

  // URL to pdf
  service.prerender.cloud/pdf/https://www.google.com/

  // URL to html (prerender)
  service.prerender.cloud/https://www.google.com/

You could look into PhantomJS. Also effectively a headless browser, but it's scriptable.

http://phantomjs.org

I started with that. Last time I looked, the maintainer had stepped down because of Chrome Headless.

And it's not a "scraping framework" any more than Chrome Headless is.

edit: Headless Chrome has very rich scripting integration with e.g. https://github.com/webfolderio/cdp4j

yes, this would be capable of waiting, but it doesn't have the abstraction built in so it looks like it would have the same problem I often run into with scapers - you go through page to get links to follow/markup to act on and then have to wait a reasonable amount of time / catch some browser based event to determine if you need to look at content one more time before continuing.

Reply on news.ycombinator.com