Hacker News Reader

Comment by mmathias | Hacker News Reader

That's odd. UrlRoulette does some sanity checking on the content of the URL. Thank you for this bug report!

https://cryptowat.ch fails too. seems to fail at very basic URL parsing.

it fails at a lot of other basic things: I was unable to post anything because my firefox 52 is not supported to get a recaptcha challenge.

  Please upgrade to a supported browser to get a reCAPTCHA challenge.

  Alternatively if you think you are getting this page in   
  error, please check your internet connection and reload.

I could reload to infinity, this would not fix this broken website or change the useless error message.

You test the URL with a HEAD method that often 404. You should try with GET instead.

A server should respond with the same headers to a HEAD request as to a GET one.

SHOULD and DOES are not the same thing.

Returning 405 'Method Not Allowed' on a HEAD isn't unreasonable, and that is the issue we are seeing.

EDIT: did say GET on prior edit.

> Returning 405 'Method Not Allowed' on a GET isn't unreasonable

I think you meant "on a HEAD".

er yes, my bad. Fixed

They should, but many good sites just don't, for whatever reason. Even ones you know are run by smart people:

  $ curl -I -X HEAD https://redis.io/
  HTTP/1.1 404 Not Found

For this app, being more forgiving is a better idea.

https://www.stumbleupon.com had a similar issue.

Fails for Amazon URLs, too (And no, I didn't include a referral link in there.)

Amazon rejects obvious non real browser user agents. This app tries to verify urls before putting them in the queue. I think they may have more anti scraper measures as well.

I also had it fail for www.autodraw.com

Reply on news.ycombinator.com