Pipeline-Oriented Programming [video] | Hacker News Reader

Pipeline-Oriented Programming [video]

youtube.com

61 pointsingve48 comments

Expressions (LINQ) are the most powerful feature available in C#. There isn't much you can't do with them if you get a bit clever. I can't imagine how many total hours of development time these abstractions have saved. I also find it relatively easy to explain these bits of code to non-developers. Performance is a common argument for not utilizing code like this, but modern .NET does all kinds of insane runtime optimization techniques to make sure things go fast (if they need to!).

We just recently started looking at using EFCore (instead of raw SQL via Dapper) and it is dawning on me that it was a strategic mistake to not further utilize the leverage that expressions can bring to the party.

If you want to see this taken to the max, then my library [1] has many monadic types (which is what LINQ is for) implemented. Including an implementation of Haskell Pipes [2], with some examples [3]

[1] https://github.com/louthy/language-ext

[2] https://louthy.github.io/language-ext/LanguageExt.Core/Effec...

[3] https://github.com/louthy/language-ext/blob/main/Samples/Eff...

For some reason your github link repeatedly crashes for me. Not sure whether it’s because im trying to view it from my iphone..

Crashes for me too on Safari/M1 MBP.

Github's client-side rendering "enhancements" strike again. Maybe I'll submit another cranky support ticket this weekend...

Android Chrome loads the page but scrolling is rather janky, so i suspect you may well be right.

Seems like you could do the same in Sql with CTEs or views, with the benefit of the query optimizer finding the fastest way to execute your pipeline on a large dataset.

> modern .NET does all kinds of insane runtime optimization techniques to make sure things go fast

Not as much of this applies to LINQ as we'd all wish. If you've got LINQ in a performance hotspot, the first thing to do is remove the LINQ. I love LINQ but plainly written loops are faster every time, and removing LINQ from profiled hotspots has always been a successful optimization for me in practice. Avoid it in performance hotspots. Once you've removed the LINQ, eliminate any inner-loop heap allocations and you're probably done optimizing. LINQ and heap allocation are the two major killers of performance in typical C# code.

Not sure why Scott Wlaschin needs to keep renaming existing concepts ('Monads' to 'Railway Oriented Programming' and 'Function Composition' to 'Pipeline Oriented Programming'), but I really don't think it helps shared understanding.

It feels, at worst, like the height of arrogance to assign new terms to existing concepts, as though you've just invented the concept. And, at best, a form of anti-intellectualism - to try and make functional concepts accessible to ex-OO devs. But it feels to me like all it does is muddy the waters.

Not everything needs to be oriented programming.

It might be purely marketing-oriented, or perhaps it’s just sales-funnel-oriented, who knows. And if anything unexpected pops up, look no further, the author can immediately come up with a new name for a decades old solution. Same happened to the “hexagonal,” “ports-and-adapters” and so on, which I noted mostly in the .NET community btw, correct me if that's wrong. Like you know every new consulting agency coming up with some new names for old stuff to stand out perhaps?

For what it's worth, it helped me a great deal in understanding.

Yeah. If the new term conveys the actual meaning better than the old term, then why not invent a new term when dealing with the new people? We didn't always fully understand what X was when we coined X as the term for it; sometimes better names are possible. Why not start to use them?

Or, probably more often, a new term for the same old thing is not better, serves no useful purpose, and just confuses.

Because The lisp community has been doing this for 50 years and has a different name for it.

While true, they have been doing it in their own silo while being smug toward everybody else.

People keep rediscovering and rensnming lisp features. I do think he's talking more about composition as a design concept, which is good for people who have only heard shout OO their entire career.

But yeah, it was weird hearing about this "new" idea I've been using for 30 years.

I worry that lazy evaluation will wind up as "filter based programming" next year.

But if it puts it in a form python or c# programmers can readily consume, it's probably okay.

Also nice to see someone doing something with f#.

Spatial Computing is another such ploy to differentiate at the surface level.

While I tend to agree with your critique I also appreciate that a different lens can be more accessible or less held back by historical baggage.

A perfect example of this was back when I had my breakthrough with Scheme and Lisp.

I always struggled with fundamental terminology used in this family of languages, particularly lambda. Lambda as a term does carry historical baggage. The term wasn’t picked out of thin air. And casually looking at Lisp and Scheme code, if you don’t know that baggage, lambda is (at least to me) an opaque term.

But I luckily stumbled upon a book “Simply Scheme”. And the first thing they did was redefine a bunch of terms. No more CAR, CDR, CADR. Instead you got things like FIRST, REST. SECOND. Importantly, they redefined LAMBDA to FUNCTION.

Well, Shazam. Ding, ding, ding. This was an “aha” moment for me. Boy did a lot of stuff suddenly fell into place.

So, simplifying the vocabulary can be very helpful for introducing gateway concepts.

He's not insisting that people already familiar with functional programming concepts not use the terminology they have historically used, he's using concrete metaphors as a teaching strategy.

Monads/Railway seems one step more than Pipeline in concept. The latter is merely connecting outputs to inputs (though I didn't get to the end of the video). Monads/Railway also handle other aspects such as errors (or side effects etc).

[It is very 'branding' and a disservice to not to mention the popular terms.]

I do see them all as the style I think and write in.

> It feels, at worst, like the height of arrogance to assign new terms to existing concepts, as though you've just invented the concept.

Hilariously I’ve done exactly that for the exact opposite reason.

I have in the past “come up” with a programming paradigm that perfectly satisfied a need and when I went to tune it up for general use I realized it was similar to a concept I was unfamiliar with, or struggled to grasp previously or previously hated working with in the manner it was taught or interfaced.

~”This looks like a monad, maybe I’ll call this a monad, but I’m unsure what precisely is the mathematical proof supporting, or what the limits and boundaries of the concept of a monad are so to avoid someone mocking me saying ‘this is the worst monad implementation, it’s hardly even a monad, it’s like a low effort subset of what a monad is.’ I’ll just name it something else.”

> at best, a form of anti-intellectualism

This is an uncharitable viewpoint. Bringing FP concepts to more devs is a good thing. I don't see how teaching better ways is any form of anti-intellectualism.

Because (if the assertion is true) he eschews the terminology used by the FP community for his own wording in an attempt to pander to people who aren't (apparently) willing or able to learn the existing terminology. In my FP library [1] (targeting C# programmers) I regularly get people asking "does the library support Railway Oriented Programming?", which they (if they'd have been taught the terminology used by the FP community and not misled by Scott Wlaschin) wouldn't ask after 10 seconds of looking at the readme to see the word 'monad' all over it.

It's treating OO programmers, who are curious about FP, as though they can't handle anything unless it's got "oriented programming" after it. In my mind that's a form of anti-intellectualism.

I really don't buy that it's needed at all. And I feel I'm in a pretty good place to state that, language-ext has ~6K stars on GitHub and the package has been downloaded nearly 13 million times [2]. It shows that devs that use an OO-first language can and are able to learn FP terminology. I never pandered and I get many OO devs coming to the library. Yeah, sometimes they need some guidance, but I've never needed to invented new terminology.

OO engineers are just as smart as FP engineers, they've just worked in different paradigms and need to learn the lingo when switching - the sooner they do it the better able they are to communicate with others in the FP community.

[1] https://github.com/louthy/language-ext

[2] https://www.nuget.org/packages/LanguageExt.Core

https://en.wikipedia.org/wiki/Pipeline_(software)

It's not a renaming. Pipelines have been a thing in software for ages. Correlating differing (surface) systems to each other is very useful when it turns out that they're expressing the same "thing" but in different manners.

> Not sure why Scott Wlaschin needs to keep renaming existing concepts

That's because most programmers ignore you as soon as they hear the m-word (or Haskell), as all that useless in "the real world"(TM).

This what I hate most about the field. Everyone trying to look smart by using big words for stupidly obvious concepts. OO is especially guilty for this with all the design patterns.

Every so often on hacker news you get some article like this that tries to introduce a "new concept" that they didn't know has already existed for the longest time or you get some other person writing a blog post about some programming analogy how programming is like "theory building". I just roll my eyes. I really wonder why you only see programmers who like to think of themselves as "highly intelligent" and calling what they're doing "theory building" and why civil engineers or electrical engineers don't supply this endless vault of "analogy blog posts" for HN readers to self pleasure themselves to.

Pipelines are effective for expressing functional concepts but are not suitable for general programming. E.g writing merge sort or a chaining hash table as a pipeline won't work out well. The problem is state. There are dataflow programming languages like Lustre and Esterel that tried to incorporate state as back arrows in their pipelines. It didn't work out so well.

But most things most people do during the day is not general programming. it would benefit from having to use principles like this instead of using 'general programming' to do them in a way that's harder to understand, harder to debug etc.

> But most things most people do during the day is not general programming.

Well, in one sense, nobody ever writes a general program - they only write specific ones.

But if your claim is "most things most people do during the day could be pipelined", I'd like to see your supporting evidence, because I'm not at all sure that claim is true.

For example, as far as I understand, GUIs are very hard to pipeline. How much of the programming done in the average day is on GUIs?

> I'd like to see your supporting evidence, because I'm not at all sure that claim is true

I believe it is, but of course it's hard to prove; I have my own libs in whatever language I use which are piping libraries very much inspired by FP, LINQ, Haskell, APL/k and other such things. And most things I do, do fit. But yeah; how to show that; I can translate most things we write anyway. We write SaaS in the broadest sense.

> For example, as far as I understand, GUIs are very hard to pipeline.

Are they? Aren't we pulling data in reducers, transforming it to structures that can be shown in the templates and transforming again on input/click and sending somewhere?

It's all pipes, kind of. Sure, some things are not, but those things at least we don't encounter very often. And when they are pipes, other ways of processing are just plainly worse in every way imho.

I don't believe in creating methodologies for blub programmers which they nevertheless have to abandon to do "real" programming. 90% of the problems comes from 10% of the code so that's what you got to address. I.e. Give me a blabla-oriented programming paradigm that makes writing efficient merge sorts easier and I'm all ears, one that can't even realistically do it just makes me yawn.

Apparently you write merge sorts a lot; why and for what? Who writes low level code like that? I have and can but I don’t need it often. Seems you do? And if you do, then sure, it doesn’t fit the case.

"The problem is state" is precisely why I do my general programming with functions.

Isn't the call stack also a form of state?

It is, but not in the way that imperiative aficionados would use it.

There are no imperative programmers, who are interested in FP - but won't switch to it because they'd have to give up their pragmatic getCallStack() and setCallStack() methods.

There are some things it's useful for and some things it isn't. Go figure.

shameless plug: I maintain a small library to do functional pipes. You can write:

    (
        range(10)
        | Map(lambda x: x * 10)
        | Filter(lambda x: x % 2 == 0)
        | Reduce(lambda a, b: a + b)
    )

and more. https://tandav.github.io/pipe21/

most operators are basically oneliners: https://github.com/tandav/pipe21/blob/master/pipe21.py#L17

I also maintain a list of libraries which uses pipeline/chain syntax: https://tandav.github.io/pipe21/similar-tools/

This is very cool. I remember I got sucked into things like Ramda going down this functional programming rabbit hole :-)

https://ramdajs.com/

The first thing I did when I encountered js in the 90s was write map and reduce. After leaving Mozilla I published a npm package with my hacks. I was very happy to find someone else had done a much better job at it and was actually maintaining it.

also, have you seen lazyjs? Also very cool.

Slides:

https://fsharpforfunandprofit.com/pipeline/

Code:

https://github.com/swlaschin/pipeline_oriented_programming_t...

This seems to be a rare update to the site which hasn't seen new content in more than 3 years?

If anyone's interested to see the slide examples but in Factor, I wrote them out:

https://programming.dev/comment/6718327

Here's a bit:

  : add1 ( x -- x' ) 1 + ;
  : square ( x -- x' ) dup * ;
  : double ( x -- x' ) 2 * ;

  : demo1 ( -- n )
    5
    add1
    square
    double
  ;

Strange that Elixir pipes are not mentioned at all..

Why? The talk is about F#.

There's a community that has experience doing analysis, design and coding with functional composition as an abnstraction. It's useful to reference that community so the viewer can get more information and more examples on applying it.

Also, sooner or later people will notice difficulties in communicating exceptional events up and down the chain and they would be greatly served by how Erlang/Elixir handles errors and process trees. Even if you want to stay in c# or f#, there's still wisdom to be gained by looking at how things work in functional languages and applying what you can in non functional languages.

Sure, but the talk is in a very .NET heavy conference filled with C#-ers, talking about F#, from a guy whose entire internet persona is advocating F#.

It's not strange that he doesn't mention SML, Caml, OCaml, Miranda, Haskell etc given this context.

This is also how PowerShell works. It's incredibly effective for generating/fetching data and processing it.

Unless things have changed the pipelines in powershell are collectors, unlike fifos in unix. For small tasks it's fine, but when shifting large blocks of data it can be noticeable slower.

The pipelines pass one item at a time unless you deliberately make them into collectors.

From memory I was doing something like write-output $var within a loop. My assumption was that a single item was readable at write time by the right hand side of the pipe. It built up something like a gig in memory before it was readable, which wasn't what I was expecting.

What would have been the way to make that more like a unix pipe? I didn't get to the bottom of it as time didn't permit.

Reply on news.ycombinator.com