Pipelining looks nice until you have to debug it. And exception handling is also very difficult, because that means to add forks into your pipelines. Pipelines are only good for programming the happy path.
And for exceptions, why not solve it in the data model, and reify failures? Push it further downstream, let your pipeline's nodes handle "monadic" result values.
Point being, it's always a tradeoff, but you can usually lessen the pain more than you think.
And that's without mentioning that a lot of "pipelining" is pure sugar over the same code we're already writing.
Exception handing is only a problem in languages that use exceptions. Fortunately there are many modern alternatives in wide use that don't use exceptions.
I've encountered and used this pattern in Python, Ruby, Haskell, Rust, C#, and maybe some other languages. It often feels nice to write, but reading can easily become difficult -- especially in Haskell where obscure operators can contain a lot of magic.
Debugging them interactively can be equally problematic, depending on the tooling. I'd argue, it's commonly harder to debug a pipeline than the equivalent imperative code and, that in the best case it's equally hard.
Programming should be focused on the happy path. Much of the syntax in primitive languages concerning exceptions and other early returns is pure noise.
The old adage of not writing code so smart you can’t debug it applies here.
Pipelining runs contrary enough to standard imperative patterns. You don’t just need a new mindset to write code this way. You need to think differently about how you structure your code overall and you need different tools.
That’s not to say that doing things a different way isn’t great, but it does come with baggage that you need to be in a position to carry.
If you need to handle an unhappy path in a way that isn’t optimal for nested function calls then you shouldn’t be nesting your function calls. Pipelining doesn’t magically make things easier nor harder in that regard.
But if a particular sequence of function calls do suit nesting, then pipelining makes the code much more readable because you’re not mixing right-to-left syntax (function nests) with left-to-right syntax (ie you’re typical language syntax).
Crudely put, in C-like languages, pipelining is just as way of turning
fn(fn(fn()))
into this:
fn | fn | fn
What you’re looking at is loops defined inside lambda functions. Pipelining makes it much easier to use anonymous functions and lambdas. But it doesn’t magically solve the problem of complex loops.
It does not solve it magically, but it does give the programmer options to coalesce different paradigms into one single working implementation.
The crux of the “magic” with pipelining is chaining functions. How those functions are composed and what they do is a separate topic entirely.
To me it is awkward to describe but simple to understand. Lucky me I have no intention of describing it.
That’s why I used scare quotes around the term ;)
> To me it is awkward to describe but simple to understand.
It’s not awkward to describe though. It’s literally just syntactic sugar for chaining functions.
It’s probably one of the easiest programming concepts to describe.
From our conversation, I’m not sure you do understand it because you keep bringing other tangential topics into the fold. Granted I don’t think the article does a great job at explaining what pipelining is, but then I think it’s point was more to demonstrate cool syntactic tricks you can pull off when writing functions as a pipeline.
edit: just realised you aren't the same person who wrote the original comment claiming pipelining was about iteration. Apologies for getting you mixed together.
data.iter()
.filter(|w| w.alive)
.map(|w| w.id)
.collect()
is one loop, as opposed to
collect(map(filter(iter(data), |w| w.alive), |w| w.id),
which is three loops.
Did you notice four letters 'i', 't', 'e' and 'r' in the code, followed by two round brackets? They mean "iterator".
`Iter` is a method of `data`. And do you know what a method is? It's a function attached to an object. A FUNCTION. Pipelining is just syntactic sugar around chaining functions.
You even proved my point when you quoted the article:
collect(map(filter(iter(data), |w| w.alive), |w| w.id))
The article itself makes no mention of this affecting how the code is executed either. Instead, it talks about code readability.
In fact the article further proves my point when it says:
> You can, of course, just assign the result of every filter and map call to a helper variable, and I will (begrudgingly) acknowledge that that works, and is significantly better than trying to do absurd levels of nesting.
What it means by this is something like the following:
list = iter(data)
list = map(channel, |w| w.toWingding())
list = filter(list, |w| w.alive)
list = map(list, |w| w.id)
result = collect(list)
+ Wikipedia: https://en.wikipedia.org/wiki/Pipeline_(computing) (no mention of iteration, just chaining functions and processes)
+ JavaScript proposal: https://www.geeksforgeeks.org/javascript-pipeline-operator/ (it's very clear how pipelining works in this guide)
+ Pipeline macros in LISP: https://blog.fugue88.ws/archives/2022-03/Pipelines-in-Lisp (again, literally just talking about cleaner syntax for nested functions)
The reason the article focuses on map/reduce type functions is because it's a common idiom for nesting commands. In fact you'll be familiar with this in Bash:
cat largefile.txt | sort | uniq --count
In here, each command is a process but analogous to a function in general-purpose programming languages like Rust, LISP, Javascript, etc. Those UNIX processes might internally loop through the contents of STDIN as a LF-delimited list but that happens transparently to the pipeline. Bash, when piping each command to the next, doesn't know how each process will internally operate. And likewise, in general-purpose programming language world, pipelines in LISP, Rust, JavaScript (et al) don't know nor care how each function behaves internally with it's passed parameters just so long as the output data type is compatible with the data type of the next function -- and if it isn't, then that's an error in the code (ie compile-time error in Rust or runtime error in Javascript).
So to summerise, pipelining has nothing to do with iteration. It's just syntactic sugar to make nested functions easier to read. And if the examples seem to focus on map/reduce, it's just because that's a common set of functions you'd want to chain and which are particularly ugly to read in nested form. ie they're an example of functions called in a pipeline, not the reason pipelines exist nor proof that pipelines themselves have any internal logic around iteration.
Pipelines are about iteration, of course. And they do have internal logic around iteration.
cat largefile.txt | sort | uniq --count
is an excellent example. While cat and count iterate on each character sequentially, sort and uniq require buffering and allocating additional structures.
Iteration is about looping and if the pipeline in the above example was some kind of secret source for iteration then the commands above would fork multiple times, but they don’t.
Rust chains everything because of this. It's often unpleasant (see: all the Rust GUI toolkits).