On incomplete HTTP reads and the requests library in Python

This is one of the most frustrating issues I've had with the requests/urllib3 libraries. I have no idea what I'm supposed to do with half of an HTML page or a PDF or a PNG. And even if it did, I certainly don't know how I'm supposed to handle those things when I don't get an indication that I'm likely dealing with an incomplete response. I can construct a use case where I might want this behavior (maybe a FLIF image), but it feels like such a huge outlier to me, that this being the default behavior of requests, which caters to making the common cases easy, has always felt a bit odd.

I don't begrudge the authors for this at all. After all, these are people donating their time to work on libraries solving hard problems for free that I can then use as part of my job. There is nothing that has stopped me from forking these libraries and changing the behavior myself, other than a lack of time and that, while I dislike this behavior, its only is a problem somewhat infrequently. All that being said, I really wish the response to this issue has been something other than what feels like a language-lawerly reading of the spec (requests isn't an "agent" and so that sentence in the spec doesn't apply) and the theory that someone _might_ be able to do something with an incomplete response _some_ of the time and as such, the vastly more common case should be made much more complicated.

But, anyway, I'm glad to hear that this issue is being addressed and I thank the authors for their work!

I've written an HTTP service that offers "tail -f" as a service. When you GET a resource with a Range: bytes=0- (or some other starting offset with unspecified end offset) you get chunked transfer-encoding that doesn't terminate until the file is unlinked or renamed out of the way.

This is incredibly handy, both for the usual things one might want to tail -f (log files), and as a cheap-but-very-functional pub/sub system.

One thing I've learned is to watch out for silly proxies (client or reverse) that want to read the whole thing and then re-serve it as definite-length (i.e., not chunked, with Content-Length) or which impose a timeout on the transfer.

HTTP needs a way to say that a chunked encoding is of indefinite length, though arguably the Range: header in my case ought to be all the hint the proxies (and libraries!) need.

As for the compression and resulting discrepancy of length: if you are doing transparent HTTP compression, you are supposed to send Transfer-Encoding: gzip (which implies chunked mode and that Content-Length shoul not be present and should be ignored by UA). Content-Encoding: gzip means that the resource itself is gzip compressed (which means that the UA should decompress it for displaying to user, but should save it to disk compressed, IE and Safari gets this wrong)

>Secondly, if we throw an exception we irrevocably destroy the data we read. It becomes impossible to access. This means that situations where the user might want to ‘muddle through’, taking as much of the data as they were able to read and keeping hold of it, becomes a little bit harder.

This is an awful justification. They should do this by raising an exception and attaching the response data to it, so the user who wants to "muddle through" can catch the exception and "muddle through" explicitly.

Regardless, this is a clear violation of https://en.wikipedia.org/wiki/Fail-fast and disappointing behavior from a package that prides itself on clear, obvious API behavior.

I am confused about the test used here. The `requests` docs very plainly state that `Response.ok` only checks the status code. Looking into the codebase proves that as well. Is there a status code for "I am going to send bad-length content"?

This might be a UX problem and not a technical one. The author seems to think that `Response.ok` should be an all-purpose check.