Curl says bye bye to pipelining | Hacker News Reader

Curl says bye bye to pipelining

daniel.haxx.se

74 pointsokket18 comments

Maybe the headline should be edited to "Curl says bye bye to HTTP/1.1 Pipelining" to prevent giving people scanning the HN headlines a heart attack. :-)

Yeah, this title is a bit "click-baity"

I had to read a many comments to see how this could be interpreted as alarming. I guess I assume more context of HM readers.

Note, this is about HTTP/1.1 Pipelining being removed not *nix Pipeline.

   HTTP/1.1 Pipelining was badly supported by curl for a long time in the sense that we had a series of known bugs and it was a fragile feature without enough tests.

Thanks for that! I got a real fright!

*nix pipes are part of POSIX and the OS/Shell implements them, if the app takes input and produces output it can be a component of a pipeline.

In short, the Curl developers couldn't remove pipes without crippling the app.

Which is exactly why the prospect of removing pipelining was so surprising!

Everything you say is correct, and may even be a word to the wise, as it were -- but there are always new command line apps made by people who haven't been thoroughly exposed to the true Unix/Linux/Posix way, and need to have that kind of thing explained. [1] [2] [3]

Which is to say that your comments are not self-evident to everyone, by any means.

Example: less(1) was eventually fitted with the -R flag to allow highlighting to be passed by pipe, to make it friendly to piping from highlight-producing apps rather than its default highlight-unfriendly stripping of input controls/escapes.

Benign intentions with regard to files sometimes require a little more thought to be simultaneously friendly to pipes, except for the simplest of plain text apps.

(I'm not trying to make a big point, but I just rediscovered 'less -R' out of necessity today. So, I was pondering some of these issues)

Worth a re-read for everyone (and I worry that these things may be in the process of being forgotten):

[1] https://en.wikipedia.org/wiki/Unix_philosophy

[2] https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond's... and the book:

[3] https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming

[4] Raymond didn't invent any of this, but he explained it very nicely, IMHO.

Edit: P.S. the above is somewhat meandering, and is sure to be criticized because simple equivalence of piped and file input is the most important thing, which I agree with -- but I was mixing that with a secondary point because of what I was doing today. Sorry. "I did not have time to make this letter shorter" etc.

Good call, given that it's not enabled in modern browsers, and is unreliably supported in proxies and servers.

It's for the best that it becomes widely treated as a non-feature of HTTP/1.1.

Still somewhat baffling that it's apparently so buggy across servers and proxies that it was for the best to disable it across browsers. In comparison to http2 multiplexing it seems trivial to get right, yet posts like this one suggest that http2 works fine everywhere and in every regard, and http1.x doesn't.

It's because real life http2 makes encryption mandatory, so that middle boxes can't fiddle with the stream. Thanks to that everything arrives as it was intended on the server

Why isn't pipelining part of TCP/IP, so the OS can handle it instead of the application?

Because TCP was designed as a single-stream protocol. There's SCTP which operates with messages rather than streams and can send those messages simultaneously.

Sounds interesting. Is that implemented in kernels? And why wasn't HTTP1.1+ built on top of that?

Yes, SCTP is implemented in kernels.

Pretty much any modern OS supports SCTP. HTTP wasn't built off it because HTTP came long before SCTP, but you won't find many protocols built off SCTP anyway, because it's poorly supported by middle-boxes and firewalls. Just look at any cloud firewall for example, you can open TCP or UDP, but not SCP or any of the "lesser known" protocols. It's currently in an unfortunate chicken and egg problem, where crappy firewalls don't support it because it's rarely used, and it's rarely used because crappy firewalls don't support it, but are everywhere on the public internet.

HTTP3 actually does propose changing the transport protocol, but to QUIC, which is a transport over UDP.

SCTP can be tunneled over UDP - which WebRTC does.

It didn’t get the right traction. It isn’t built into windows, it was even a second class citizen on Linux not that long ago requiring lksctp to be installed (maybe it still does).

> Is that implemented in kernels?

Yes, but it's largely unusable because it's not supported by most home network routers, and isn't supported at all on Windows or macOS.

> And why wasn't HTTP1.1+ built on top of that?

Because it didn't exist yet. HTTP 1.1 was standardized in 1997, and SCTP development didn't start until around 2000.

Reply on news.ycombinator.com