Hacker News Reader

Comment by gkya | Hacker News Reader

When I was recently writing a Python script to ping the urls in places.sqlite to check for link rot, I had to change the urllib's useragent to mimick a modern browser, because otherwise lots of web sites redirect me around or return errors (even 404).

View on HN

Yep, one bored day I was playing with the oneom.tk API from python and it took a long while to figure out why I could get a page in the browser but the python API fetch wasn't working. Until I set User-Agent to "Mozilla/5.0" that is.

Why anyone thinks it's a good idea not to send API data to a python User-Agent is kind of beyond me but who knows...can't really complain since they provide data for free and I really was just playing around without actually needing to do anything productive.

Actually found a bug in urllib3 (that I probably should get around to reporting) -- page sends http unless you set Accept to 'application/json' which urllib3 doesn't let you do.

You should be able to add headers to your request. I set my user agent as such:

  import urllib.request as rq
  req = rq.Request(url, headers={'User-Agent': useragent})
  request = rq.urlopen(req, timeout=timeout)

You should be able to set Accept in a similar fashion (tho IDK if urllib mandates a certain value for that header).

Yeah, it works in urllib but not in urllib3.

IIRC 'User-Agent' wasn't the problem but trying to set 'Accept' is what failed (and even more strangely they only do User-Agent filtering on API calls but not for http). I started to dig into the code but since I got it to work in urllib I left it as Someone Else's Problem.

Oh, alright then. I didn't know urllib3 was a different thing. I'm recently returning to python after some years, still catching up with all the new things.

urllib3 lead maintainer here. I wasn't able to reproduce your bug, at least with the main development branch, when I passed `Accept` as part of the `headers` dict. If you could file an issue with a reproduction case, I'd definitely appreciate it.

Seems like the problem is due to the way they filter on User-Agent and not with urllib3, sorry.

Eh, maybe...

http = urllib3.PoolManager()

r = http.request('GET', "https://oneom.tk/data/config", headers={'Accept': 'application/json', 'User-Agent' : "Mozilla/5.0"})

r.headers['Content-Type'] 'application/json'

r = http.request('GET', "https://oneom.tk/data/config", headers={'Accept': 'application/json'})

r.headers['Content-Type'] 'text/html; charset=UTF-8'

It looks like this API in particular expects all requests to have a User-Agent header, and we don't set one by default. Setting any user agent appears to work; the issue isn't with the `Accept` header.

You can reproduce with curl with the following command:

    curl -v 'https://oneom.tk/data/config' -H 'Accept: application/json' -H 'User-Agent:'

That'll nullify the default curl user agent and should produce the same results you were seeing with urllib3.

That makes sense, I kind of figured they were doing something screwy. I'm guessing a blacklist on certain User-Agent settings.

When I was messing with it I could get it to work with curl and (eventually) urllib but urllib3 was no bueno until I tried to reproduce the problem and just copied the header with the User-Agent field from the urllib code.

Reply on news.ycombinator.com