Pipelines – a guided tour of the new IO API in .NET | Hacker News Reader

Pipelines – a guided tour of the new IO API in .NET

blog.marcgravell.com

286 pointsbenaadams26 comments

As Tychus Findlay might say, "It's about damned time."

The Stream API has to be one of the ugliest warts in .NET, and it's been around since the beginning. I am so glad Microsoft finally addressed it. Every time I use Stream, I am reminded of how badly the Stream API violates SOLID principles.

I like how Microsoft approached the problem from below by introducing new types like Span<T> and Memory<T>, then the Pipes API. It shows that the future of the platform is in good hands.

First job out of school my coworker was talking about how nice dotnet Stream API was. I had a Stream. Streams have Position. A GZipStream is a Stream. A GZipStream does not have position. Countless hours of Java style "OOP" indoctrination flashed before my eyes and I realized that even something as big as Dotnet could do such a thing. It made me feel like a kid that just realized their parents aren't infallible.

I dunno, I'd prefer a consistent Stream interface that just sometimes doesn't have a position, and wraps other Streams with little fuss, rather than have to change gears into Source/Sink lexicon.

I see where you're coming from, and hey, the thing works. Not everything is perfect, but the Stream API could definitely be better.

I agree it would be nice to have a consistent interface. The problem is it's neither consistent nor an interface, it's an inconsistent hierarchy of subclasses. There's always fuss because everyone is going to get bitten by it at some point. It's just a poor abstraction, and it leaves a bad taste in your mouth as a library (well, framework) consumer.

Microsoft promised me that they gave me a thing that had certain features and capabilities using industry standard terminology and knowledge. There's a contract. If you go out for a car loan that says "0% APR for 12 months!" You expect that to be the first 12 months, not "0% APR only on days divisible by 7 during leap years, up to a total of 12 * 30 days over the course of the next millennium, otherwise 7.5%"

Thank you for eloquently summarizing my "violates SOLID principles" comment. Rigid OOP class hierarchies suck, in general.

It's probably one of the best out there, for likely no other reason than it being included in the box. I still remember back in .Net 1.0, Googling how to load a PNG to GDI from memory and being being met with "just load it from a MemoryStream." At the time this was profound (maybe not so much for Haskell coder, but we didn't all have the fortune of growing up with that). A few years later I was developing something that used STARTTLS (XMPP) and actually starting that TLS stream was just a matter of wrapping the current stream in a SslStream and swapping it out. It's very consistent.

Contrast this with, say, C where there is nothing. Encryption would look vastly different to compression, but both are a similar abstraction: transformations. Now if you wanted to load that result into a device bitmap, you'd likely have to dump the entire stream into memory on onto disk - depending on what different pattern the other library you are using thinks about streams.

Go did a much better job out of the box, but you'd have a hard time convincing me that .Net streams weren't an inspiration there; especially given the nomenclature used.

Smalltalk, CLU, Modula-3, Oberon, Eiffel, C++, Java, Delphi already had the streams concept.

Plenty of stuff for inspiration for.NET devs.

> The Stream API has to be one of the ugliest warts in .NET, and it's been around since the beginning.

While true, at least it was less ugly than Java, and that was the initial design goal.

I agree though. Lots of things have changed and been further improved on since then, and maybe it was about time these parts got fixed up too.

Maybe the absence of lambdas, async/await and 'using' in the initial release be a reason for this?

A lot of these features evolved over time, so the fundamental building blocks might have been missing.

I am not sure....

Right, it was a good API for its time, especially compared to alternatives, most particularly Java's Streams API at the time.

That said, the number of projects I worked on where this new API would have made things so much easier is staggering. It is a shame we can't just fax this API back to our past selves.

The other big building block this uses is Span<T>/Memory<T> which is recent as in this year. Span<T>'s the sort of huge paradigm shift that likely would never have occurred to early .NET design, but starts to seem obvious in retrospect.

While I agree in total (e.g., how Microsoft approached the problem is great) I do not agree that the Stream API is that ugly. Yes, it is not nice and has a lot of pitfalls, but it is one of the "lower level" APIs in .NET and certainly did its job. Now its (finally) time to move on.

The future here looks bright!

Credit where it is deserved. Span is a great idea, and building fixed sized streams on top of them is a great idea (I haven't read it, but I assume this is the thing they're going after).

I had some trouble using ReadOnlySequence in a library I am writing (mostly because of no documentation). After looking through how Kestrel uses it(they started using Pipelines in ASP.NET Core 2.1) I found their BufferReader[1] class. It made using ReadOnlySequence much easier.

Overall, pipelines have been really solid for me so far. Working out how to use it without documentation was a total pain though, hopefully they get that out soon. If anyone is interested I can throw the library up on GitHub if anyone wants some real world examples of using pipelines.

[1] https://github.com/aspnet/Common/blob/master/shared/Microsof...

https://www.codemag.com/Article/1807051/Introducing-.NET-Cor... might help with Readonlysequence too

David Fowler (The architect of this API) is brilliant! The Pipelines API turned out better than I ever expected.

His work rate is pretty impressive.

This is literally the most documentation on the internet for this API. AFAICT, though, it doesn't have compression or encryption stages, yet?

Since the entire design of Pipelines is 0-copy, I figure you would have to manually pull from one pipe, operate on it and push the resulting data into another pipe.

Im sure there is an example of connecting pipes together like this somewhere. Maybe in Kestrel or SignalR (Core)?

> Im sure there is an example of connection pipes together like this somewhere. Maybe in Kestrel or SignalR?

Last I checked, SignalR haven’t had a release in years, so I wouldn’t go there looking for news.

SignalR Core is what i meant.

It is alive an under active development on .NET Core

https://github.com/aspnet/SignalR

That’s definitely something different and you appear to be correct:

https://github.com/aspnet/SignalR/search?q=pipelines&unscope...

Yes, but assume you’re lazy: CompressionStream and EncryptionStream already exist.

If you're referring to [`CryptoStream`](https://msdn.microsoft.com/en-us/library/system.security.cry...), forget it. Using it securely (in particular against chosen ciphertext attacks) is nigh impossible.

I wonder which of that applies to SslStream class from the framework?

https://msdn.microsoft.com/en-us/library/system.net.security...

Part 2 here: https://blog.marcgravell.com/2018/07/pipe-dreams-part-2.html

Very nice post, I’m eagerly waiting for the next parts.

Reply on news.ycombinator.com