How do HTTP servers figure out Content-Length?
aarol.dev
aarol.dev
In most HTTP server implementations from other languages I've worked with I recall having to either:
- explicitly define the Content-Length up-front (clients then usually don't like it if you send too little and servers don't like it if you send too much)
- have a single "write" operation with an object where the Content-Length can be figured out quite easily
- turn on chunking myself and handle the chunk writing myself
I don't recall having seen the kind of automatic chunking described in the article before (and I'm not too sure whether I'm a fan of it).
This approach makes sense from the API standpoint because the caller generally has no idea whether the chunked encoding is necessary, or even its very existence. Honestly that's less confusing than what express.js does to the middleware function: `app.get("/", (req, res) => { ... })` and `app.get("/", (req, res, next) => { ... })` behave differently because it tries to infer the presence of `next` by probing `Function.prototype.length`.
But one of the if not THE major exception is this : buffering and flushing works automagically and a lot of PHP dev end up massively blindsinded by it at some point
PS: with the rise of modern PHP and it's high quality object based framework, this become less and less true
PS2: I am not in ANY way saying anything good or bad or superior or inferior about any dev here, just a difference in approach
But yeah the moment you ended up wanting to do anything advanced, you were doing your own buffer on top of that anyway, or disabling it and going raw.
I can't say I miss those days ! Or this platform for that matter.
PHP 75.8%
Ruby 6.0%
....This feels like a completely random swipe at an unrelated feature of a JavaScript framework, and I'm not even sure that it's an accurate swipe.
The entire point of Function.length (slight nit: Function.prototype.length is different and is always zero) is to check the arity of the function [0]. There's no "tries to": if your middleware function accepts three arguments then it will have a length of 3.
Aside from that, I've also done a bunch of digging and can't find any evidence that they're doing [1]. Do you have a source for the claim that this is what they're doing?
[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
[1] https://github.com/search?q=repo%3Aexpressjs%2Fexpress%20%22...
I'm surprised to see that it's now gone too! The exact commit is [1], which happened before Express.js 4.7, and you can search for the variable name `arity` in any previous versions to see what I was talking. It seems that my memory was slightly off as well, my bad. The correct description would be that older versions of Express.js used to distinguish "error" callbacks from normal router callbacks by their arities, so `(req, res)` and `(req, res, next)` would have been thankfully okay, while any extra argument added by an accident will effectively disable that callback without any indication. It was a very good reason for me to be surprised and annoyed at that time.
[1] https://github.com/expressjs/express/commit/76e8bfa1dcb7b293...
[0] https://github.com/pillarjs/router/blob/2e7fb67ad1b0c1cd2d9e...
If you want to go that route, it's not Function.length, either—which is different and is always 1 (barring potential future spec changes that change the arity of the Function global).
Entirely unrelated, but the older I get, the more it seems like exposing the things under ".prototype" as parts of the object was probably a mistake. If I'm not mistaken, that is reflection, and it feels like JS reaches for reflection much more often than other languages. I think in part because it's a native part of the object rather than a reflection library, so it feels like less of an anti-pattern.
I never really thought about this, but it does explain how optional arguments without a default value work in Typescript. How very strange of a language decision.
> To be clear, distinguishing different types based on arity would have been okay if JS was statically typed or `Function` exposed more thorough information about its signature.
I actually like this less in a system with better typing. I don't personally think it's a good tradeoff to dramatically increase the complexity of the types just to avoid having a separate method to register a chunked handler. It would make more sense to me to have "app.get()" and "app.getChunked()", or some kind of closure that converts a chunked handler to something app.get() will allow, like "app.get(chunked((req, res, next) => {}))".
The typing effectively becomes part of the control flow of the application, which is something I tend to prefer avoiding. Data modelling should model the domain, code should implement business logic. Having data modelling impact business logic feels like some kind of recursive anti-pattern, but I'm not quite clever enough to figure out why it makes me feel that way.
We had a small router/firewall thing at a previous company that had a web interface, but for some reason its Content-Length header had an off-by-one error. IIRC Chrome handled this okay (once the connection was closed it would display the content) while Firefox would hang waiting for that one extra byte that never came.
However, you better be right! I just found a bug in some really old code that was gzipping every response when it was appropriate (ie, asked for, textual, etc). But it was ignoring the content-length header! So, if it was set manually, it would then be wrong after compression. That caused insidious bugs for years. The fix, obviously, was to just delete that manual header if the stream was going to be compressed.
Sounds like a powerful bug you have, potentially.
but it might not come, since decades http sends more than one file per connection, so it might just get the beginning of the next reply, write that and the next reply will be corrupt as well.
I'm pretty sure something like this can cause some form of HTTP desync in a loadbalancer/proxy setup.
> That caused insidious bugs for years.
https://www.oreilly.com/library/view/high-performance-web/97...
Optimizing per-packet really improves things but has gotten very difficult with SSL and now QUIC. I'm not sure Google ever got the front page down to a single packet (would love a reference!) but it definitely paid very close attention to every byte and details of TCP performance.
I was curious but the most recent data I could find was from 2017 when there was a mix of CDNs at initcwnd=10 and initcwnd>10:
https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-p...
Currently Linux still follows RFC6928 and defaults to initcwnd=10:
https://github.com/torvalds/linux/blob/v6.11/include/net/tcp...
(pre-ecmascript versions of JS not investigated)
EcmaScript 1(1997) = JavaScript 1.1 - missing many ES3 features (see below), of which exceptions are the unrecoverable thing.
EcmaScript 2(1998) - minimal changes, mostly deprecations and clarifications of intent, reserve Java keywords
EcmaScript 3(1999) - exceptions, regexes, switch, do-while, instanceof, undefined, strict equality, encodeURI* instead of escape, JSON, several methods on Object/String/Array/Date
EcmaScript 4(2003) - does not exist due to committee implosion
EcmaScript 5(2009) - strict mode, getters/setters, remove reservations of many Java keywords, add reservation for let/yield, debugger, many static functions of Object, Array.isArray, many Array methods, String().trim method, Date.now, Date().toISOString, Date().toJSON
EcmaScript 5.1(2011) - I did not notice any changes compared to ES5, likely just wording changes. This is the first one that's available in HTML rather than just PDF.
EcmaScript 6(2015) - classes, let/const, symbols, modules (in theory; it's $CURRENTYEAR and there are still major problems with them in practice), and all sorts of things (not listed)
EcmaScript 11(2020) - bigint, globalThis
There are still pre-ES6 implementations in the wild (not for the general web obviously) ... from what I've seen they're mostly ES5, sometimes with a few easy ES6 features added.
This is why in 2024 you still must use XmlHttpRequest instead of fetch() when progress reporting is needed. fetch() cannot do progress reporting on compressed streams.
2. You can iterate through the uncompressed response bytes with a ReadableStream.
Please explain how would you produce a progress percentage from these?
Not sure about you, but to me "XmlHttpRequest" in my request handling code feels less dirty than "x-amz-meta-". But to each their own I guess.
Check CloudFlare, Akamai, ...
Or we can keep using XmlHttpRequest.
Tough choice.
If you need progress on a text file, then don't compress it while downloading. Text files that are small won't really need progress or compression.
If you're sending a large amount of data that can be compressed, then zip it before sending it, download-with-progress (without http compression), and then unzip the file in the browser and do what you need with the contents.
I'm sure there are probably other ways to handle it too.
Yes, there is. The simplest solution is called XmlHttpRequest, which has a "progress" event that correctly reports the response body bytes.
You can even make a wrapper for XmlHttpRequest and call it "myFetch()" if you insist.
Also in 2018, some fun where when downloading a file, browsers report bytes written to disk vs content-length, which is wildly out when you factor in gzip https://x.com/jaffathecake/status/996720156905820160
It may be better now but a huge number of libraries and frameworks would either include the terminating NULL byte in the count but not send it, or not include the terminator in the count but include it in the stream.
https://notes.benheater.com/books/web/page/multipart-forms-a...
Buffering can be appropriate for small responses; or at least convenient. But for bigger responses this can be error prone. If you do this right, you serve the first byte of the response to the user before you read the last byte from wherever you are reading (database, file system, S3, etc.). If you do it wrong, you might run out of memory. Or your user's request times out before you are ready to respond.
This is a thing that's gotten harder with non-blocking frameworks. Spring Boot in particular can be a PITA on this front if you use it with non-blocking IO. I had some fun figuring that out some years ago. Using Kotlin makes it slightly easier to deal with low level Spring internals (fluxes and what not).
Sometimes the right answer is that it's too expensive to figure out the content length, or a content hash. Whatever you do, you need to send the headers with that information before you send anything else. And if you need to read everything before you can calculate that information and send it, your choices are buffering or omitting that information.
This is the #1 most common mistake made by a "web framework".
Before $YOU jump up with a list of exceptions, it slowly gets better over time, and it has been getting better for a while, and there are many frameworks in the world, so the list that get it right is quite long. But there's still a lot of frameworks out there that assume this, that consider streaming to be the "exception" rather than non-streaming being a special case of streaming, and I still see new people make this mistake with some frequency, so the list of frameworks that still incorporate this mistake into their very core is also quite long.
My favorite is when I see a new framework sit on top of something like Go that properly streams, and it actively wrecks the underlying streaming capability to turn an HTTP response into a string.
Streaming properly is harder in the short term, but writing a framework where all responses are strings becomes harder in the long term. You eventually hit the wall where that is no longer feasible, but then, fixing it becomes very difficult.
Simply not sending a content-length is often the right answer. In an API situation, whatever negative consequences there are are fairly muted. The real problem I encounter a lot is when I'm streaming out some response from some DB query and I encounter a situation that I would have yielded a 500-type response for after I've already streamed out some content. It can be helpful to specify in your API that you may both emit content and an error and users need to check both. For instance, in the common case of dumping JSON, you can spec a top-level {"results": [...], "error": ...} as your return type, stream out a "results", but if a later error occurs, still return an "error" later. Arguably suboptimal, but requiring all errors to be known up front in a streaming situation is impossible, so... suboptimal wins over impossible.
e.g I drafted this a long time ago, because if you generate something live and send it in a streaming fashion, well you can't have progress reporting since you don't know the final size in bytes, even though server side you know how far you're into generating.
This was used for multiple things like generating CSV exports from a bunch of RDBM records, or compressed tarballs from a set of files, or a bunch of other silly things like generating sequences (Fibonacci, random integers, whatever...), that could take "a while" (as in, enough to be friendly and report progress).
https://github.com/lloeki/http-chunked-progress/blob/master/...
To me the more interesting question is how web server receive an incoming request. You want to be able to read the whole thing into a single buffer, but you don't know how long its going to be until you actually read some of it. I learned recently that libc has a way to "peek" at some data without removing it from the recv buffer..... I'm curious if this is ever used to optimize the receive process?
It's not. Like, hell no. That is so complex. Multiplexing, underlying TCP specifications, Server Push, Stream prioritization (vs priorization !), encryption (ALPN or NPN ?), extension like HSTS, CORS, WebDav or HLS, ...
It's a great protocol, nowhere near simple.
> Basically, it’s a text file that has some specific rules to make parsing it easier.
Nope, since HTTP/2 that is just a textual representation, not the real "on the wire" protocol. HTTP/2 is 10 now.
HTTP/1.0 is simple. HTTP/1.1 is undoubtedly more complex but manageable.
The statement that HTTP is simple is just not true. Even if Go makes it look easy.
So when the article say "All HTTP requests look something like this", that's false, that is not a big deal but it spread that idea that HTTP is easy and it's not.
The whole section on cache is "reality based," and it's only gotten worse as the years have moved on.
Anyway, back in the day Content-Length was one of the fields you were never supposed to trust. There's really no reason to trust it now, but I suppose you can use it as a hint to see amount of buffer you're supposed to allocate. But of course, the content length may exceed that length, which would mean that if you did it incorrectly you'd copy the incoming request data past the end of the buffer.
So even today, don't trust Content-Length.
Websocket is a different protocol that is started up via HTTP.
You can absolutely assume that http 1.1 will work on basically anything; websockets are more finicky even now, and certainly were back in the day.
Whaaaaa??? We should eliminate these non-JS filled nonsense immediately!
For example, some websocket servers don't pass back errors to the client (AWS). That makes it quite difficult to, say, retry on the client side.
Chunked encoding is used by video players - so you can request X bytes of a video file. That means you don't have to download the whole file, and if the user closes the video you didn't waste bandwidth. There are likely more uses of it.
Just a nitpick, but what you describe here is byte range requests. They can be used with or without chunked encoding, which is a separate thing.