Hacker News Reader

Comment by pkulak | Hacker News Reader

And if you set your own content length header, most http servers will respect it and not chunk. That way, you can stream a 4-gig file that you know the size of per the metadata. This makes downloading nicer because browsers and such will then show a progress bar and time estimate.

However, you better be right! I just found a bug in some really old code that was gzipping every response when it was appropriate (ie, asked for, textual, etc). But it was ignoring the content-length header! So, if it was set manually, it would then be wrong after compression. That caused insidious bugs for years. The fix, obviously, was to just delete that manual header if the stream was going to be compressed.

View on HN

  > That caused insidious bugs for years.

A lot of people here could probably benefit professionally from hearing about what the bugs were. Knowing what to identify in the future could be really helpful. Thanks.

Well, for years our logs would fill up with these nasty warnings of "connection dropped", something like that. Naturally, you think that's just some badly configured client, mobile connection, something. But then why would it be configured to log that as a warning (or it may have even triggered an error). I think that was because when payloads are small the compression overhead makes them larger, which means the Content-Length is too small and clients would be terminating the connection early. And getting garbage or truncated responses. Ouch!

Insidious was the right word. Thank you!

Did you see if you could turn this into HTTP Request Smuggling? Or something else with security impact?

Sounds like a powerful bug you have, potentially.

To me it sounds like the server handled the request just fine and reused the header (which was wrong.) The client then had the problem of a wrong response.

If you say to read 1000 bytes from memory and then pass a 900 bytes long array, that's a security bug that can cause crash, corrupt data, and leaking stuff that shouldn't have leaked.

The size of the buffer and how many bytes are written have nothing intrinsically linked to what the header says. It's a bug sure but does not mean there's any security issue on the server.

It will likely generate corrupt files on the client as well.

Not very. The system might allocate that length ahead of time (I've seen that option in torrent clients and iirc ftp systems) but, latest by the time a FIN comes in, it'll know the file is finished and can truncate it. If finished-early downloads are not implemented despite it doing preallocation, that's still not a security bug

if a FIN comes, the client will mark the file as partially downloaded.

but it might not come, since decades http sends more than one file per connection, so it might just get the beginning of the next reply, write that and the next reply will be corrupt as well.

It's a bug for sure, but I think whether it's a security issue could depend on the language. If the callee is able to determine the length of the array, it can just return an error instead of a potential buffer overrun.

In this case, the 1000 bytes aren't being read from memory, they're being read from a socket. If you try to over-read from a socket the worst you'll get is a blocking call or an error (depending what mode you're in).

maybe response based trickery then? :D What happens to the response after that one, are the first 100 bytes cut of, or what?

I'm pretty sure something like this can cause some form of HTTP desync in a loadbalancer/proxy setup.

Reply on news.ycombinator.com