Hacker News Reader

Comment by andai | Hacker News Reader

In uni I built a simple web scraper in JavaScript. It just ran in the browser. It would fetch a page, extract the links, then fetch those pages.

You could watch it run in realtime and it would build out a tree of nested links as it crawled. It was a lot of fun to watch! (More fun than CLI based crawlers for sure.)

The only issue I had was not being able to fetch pages from the front end due to CORS, so I just added a "proxy" to my server in 1 line of PHP. There, now it's secure? ;)

View on HN

CORS protects the browser's knowledge of a user on your website from other websites. It's a big failing of the protocol and/or communications about it that people think it offers any more security guarantees than that.

>There, now it's secure? ;)

CORS achieved its objective here, because now those requests were coming from your PHP server rather than the user's browser.

If you know what a user agent is, let alone how to change it, CORS is not meant for you.

Its guide rails to help the tech illiterate not get hacked. It raises the bar on what gets through. It’s not going to stop a determined attacker, but will catch enough to make a dent. Defense in depth and all that.

CORS (or rather the same origin policy, of which CORS is an explicit server-side opt-out) is not a generic security improvement, it solves a very specific problem: (Code on) website A being able to make requests to website B with the cookies of B (often implying user login state/authentication at B) and read the response.

In a (possibly better) parallel universe, cross-site requests just don't send cookies or other ambient authentication state like that by default, and we wouldn't need CORS.

It sounds like you need to go back to school because you’re entirely

Edit: Guys! I'm talking about cool scraper running in browser (has that ever been done before?), and everyone's hyperfixating on bloody CORS.

Well I guess I failed the Show Don't Tell eh :) lost the code when I left uni, I might have to make another one...

Because the Single Origin Policy is a client side security measure. It's to protect the user from random malicious JS out on the web. If you are writing server side code you're outside the scope of who the security model is trying to protect.

That's rater like "gnu.org" which blocks you when you're using a slightly older browser. But when you change your user agent to "curl" it magically starts working. Or the German news site "spiegel.de" which also blocks old browsers from accessing the site entirely, unless you change the user agent to "bingbot" (or some other random bot from their whitelist). *insert a facepalm emoji here*

They probably do that to keep out the dumbest scrapers and bots. They often present themselves as out of date Chrome browsers.

Nailed it. I get a zillion bit hits claiming to be Safari for iOS 8 and the like.

Sorry if you’re trying to visit my site on such a device.

Reply on news.ycombinator.com