Advanced usage of Python requests: timeouts, retries, hooks | Hacker News Reader

Advanced usage of Python requests: timeouts, retries, hooks

findwork.dev

123 pointsdhxt11 comments

I just recently dealt with a timeout issue in requests: I was setting the timeout to 1 but it took 10 seconds for the request to fail. It turns out that the timeout does not apply to the name resolver and the name lookup was timing out. So the default system-level timeout was 10 seconds.

Thank you for this brief reference. I'm always going back to my old projects to remind myself how I did things with requests. I didn't realize some of things existed.

You're welcome. I thought I'd aggregate all the useful tricks I'd found while working on my last project. It's a collection of snippets I had found in the docs and in Github issues.

The post contains the most important snippets, but you might find the advanced section of the requests docs [1] useful and the requests_toolbelt [2].

[1] https://requests.readthedocs.io/en/master/user/advanced/

[2] https://toolbelt.readthedocs.io/en/latest/

I will definitely be diving into the advanced section in the Requests docs. Thanks!

I wish that requests had a default timeout instead of defaulting to waiting forever.

It's very easy for devs to forget to think about timeouts, but inevitably that ends in a cascading production outage.

I'm not sure if you noticed, but the post contains a section on how to set default timeouts.

https://findwork.dev/blog/advanced-usage-python-requests-tim...

Right. I'm saying the library should set a default timeout by default, which you could then change if needed.

Of course, whatever it chose would inevitably be wrong for many scenarios, it would be less wrong than defaulting to waiting forever.

I absolutely agree. According to this comment [1] they're planning to introduce it in v3.

[1] https://github.com/psf/requests/issues/3070#issuecomment-328...

If I enable logging of retries: logging.basicConfig(level=logging.DEBUG)

I dont see any retries on response 200: ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='domain.com', port=443): Read timed out."))

Is there a way to also retry on ChunkedEncodingError?

That sentry responses decorator is absolutely beautiful

Fantastic reference. Thanks for sharing!

Reply on news.ycombinator.com