Hacker News Reader

>Sorry, can't kimonify

>According that web site's data protection policy, we were unable to kimonify that particular page.

Sigh... Oh well... Back to scraping.

What page were you trying to hit? We'll check it out

Pages buried in here: https://fannin4.wcjc.edu/

The course catalog is public, so no login is needed. I want to scrape various data related to courses, to populate forms automatically and such.

Yeah, of course it won't work with HTTPS sites. They'd have to proxy those HTTPS sites and perform a MITM just to do it.

As opposed to HTTP where they proxy it and MITM it? I don't understand the objection.

I kind of just assumed that's what they were doing.

HTTPS is definitely a problem for proxy servers unless you the proxy server rewrites all the URLs in the html pages loaded as well as all the URLs of the Ajax calls to point back to the proxy server.

It may be to their advantage to come up with a solution for this, given the popularity of https these days.

Reply on news.ycombinator.com