The JavaScript Pipeline Operator | Hacker News Reader

The JavaScript Pipeline Operator

yanis.blog

43 pointsyanis_t59 comments

You don’t need lodash or a new operator for the example provided:

  departments
    .flatMap(d => d.employees)
    .map(e => e.salary)
    .reduce((p, v) => Math.max(p, v), 0)

That said, the pipe operator is nice because it allows similar chaining patterns on subjects that are not arrays, as shown in the proposal’s README: https://github.com/tc39/proposal-pipeline-operator/blob/mast...

That's all well and good ( increasing readability ) but the problem remains that each step has to finish before the next step can begin... sometimes the entire dataset won't fit into memory/machine/whatever...

More useful, IMHO, would be a way to EASILY compose a true pipeline:

  const
    _pipe = (a, b) => (arg) => b(a(arg)),
    pipe = (...ops) => ops.reduce(_pipe)

...but have the behavior work like unix pipes ( a stream ), nodeJS supports this concept at it's most basic level using the pipe() abstraction, although you have to supply methods which handle being pipe'd to, and from... an example:

  const crypto = require('crypto');
  // ...
  fs.createReadStream(file)
    .pipe(zlib.createGzip())
    .pipe(crypto.createCipher('aes192', 'a_secret'))
    .pipe(reportProgress)
    .pipe(fs.createWriteStream(file + '.zz'))
    .on('finish', () => console.log('Done'));

*ripped from: [source](https://medium.freecodecamp.org/node-js-streams-everything-y...)

Imagine reading a 100gb json file line-by-line via ajax on the client, and feeding into the pipeline of transformative methods -- iteratively introduce data in one end of the pipe, and gathering the results at the other end, and creating some visualization like a graph or whatever... without ever having to have the entire thing in memory at once...

Have you seen https://github.com/labs42io/itiriri? It does lazy queries on iterables, like IEnumerable from C#.

    import { query } from 'itiriri';

    function* fibonacci() {
       let [a, b] = [0, 1];

      while (true) {
        yield a;
        [a, b] = [b, a + b];
      }
    }

    // Finding first 3 Fibonacci numbers that contain 42
    const result = query(fibonacci())
      .filter(x => x.toString().indexOf('42') !== -1)
      .take(3);

    for (const e of result) {
      console.log(e);
    }

    // outputs: 514229, 267914296, 7778742049

I wrote a lib that does this too. It's been a while, but using generators tended to be way slower than just using arrays, except in the most obvious cases (array of 1000000 elements, only take 5, no sorting involved, etc). Maybe that's changed. It's been a while since I've checked.

Realistically, there's always a few functions I want to call on my data that aren't part of the prototype of the thing I'm transforming. And then this pipe operator is called for.

Good point. Is flatMap already supported by browsers or is it a proposal?

Both flat[1] and flatMap[2] are unsupported in Edge, IE and Node.

1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

2. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Still a proposal and experimental, stage 3.

If you want to see how the operator "feels" in practice, I recommend trying out F#[0] or Elixir[1]. Hint: it's pretty great!

[0]: https://www.microsoft.com/net/languages/fsharp

[1]: https://elixir-lang.org/

ES6 was needed, but I think js should hit the breaks for new language features for a while.

I disagree. ES6 made JS much more palatable but it's still behind most modern languages in terms of number of features and in velocity of new features getting introduced. Javascript has actually been extremely slow to get new features compared to other languages.

Since ES6 (3 years ago), the only major, mainstream feature that has been added to JS has been async functions, which are awesome, but in that time Python (a language of similar age and global use) has gotten an optional type system, async/await, and _hundreds_ of standard library modules, classes, and functions.

In my opinion, JS badly needs more functions in it's standard library, which should move through the proposal policy quickly. However, common sense standard library functions like flatMap have been in the pipeline for years are still not in the final stage. We may not get flatMap, import(), trimStart, or trimEnd in ES2019, they are all stalled in Stage 3. There's no good arguments not to have these in the standard library, but we still have to wait years.

In addition to that, JS could really use some language features that make it's core paradigms, functional and object oriented, simpler, and have been in other languages for years. The pipeline operator would make functional programming much easier, and decorators would make object oriented programming easier. These are, in my opinion, common sense features since they are so well received in other langauges, but they get stalled out since they are seen as "bloating" the syntax, or making javascript "too opinionated on how to do something". Meanwhile, people are compiling full fledged functional languages (Reason, Elm, Purescript) to Javascript, because Javascript isn't keeping up with them on features. It could, but the TC39 review process is so slow that it's not worth waiting on. For those reasons, in my opinion, we should really be releasing new language features much quicker.

> In my opinion, JS badly needs more functions in it's standard library, which should move through the proposal policy quickly. However, common sense standard library functions like flatMap have been in the pipeline for years are still not in the final stage. We may not get flatMap, import(), trimStart, or trimEnd in ES2019, they are all stalled in Stage 3. There's no good arguments not to have these in the standard library, but we still have to wait years.

This is one of the arguments for the pipeline operator, as well. Right now browsers are inherently quite afraid of adding things to Object.prototype, Array.prototype, etc, because doing so breaks things. There are too many ancient JS libraries that patch whatever they want into Object.prototype/Array.prototype, such as MooTools, that threaten to break if the prototypes themselves change from what is expected. This is how flatMap() got stuck into "SmooshGate" and the intentionally absurd proposal that it should be named smooshMap(), because if Browsers add Array.prototype.flatMap it does break versions of MooTools still out in the wild.

The pipeline operator is one option (among several proposed options) to move the standard library forward without compromising backward compatibility. Maybe we can't have nice things like `myArray.flatMap()` because it breaks backwards compatibility, but maybe something like `myArray|>flatMap()` is an acceptable compromise.

> I disagree. ES6 made JS much more palatable but it's still behind most modern languages in terms of number of features and in velocity of new features getting introduced. Javascript has actually been extremely slow to get new features compared to other languages.

JS has been adding features at a tremendous rate. Several major features land every year. Adding something like async generators is a huge change that affects large parts of the language.

If the Go, Ruby, or Python guys want to add a feature, they just code up a proposal and add it if the maintainers like it. JS has 4 major implementations and a dozen or so other reasonably popular implementations. Adding a feature in a way that works with all of them (without breaking 25 years worth of applications) is hard.

Reason, Elm, and Purescript are not good comparisons. The features they add (especially the type system) aren't likely to ever be JS features (if they are even possible in the language).

You say "Several major features land every year.", but you don't back that up.

In 2018 the major features we got object rest/spread and asynchronous interation. Rest/spread is a big feature that I forgot to mention, but async iteration is only useful in a few situations, I'd call it a minor feature.

In 2017 we got async/await, and shared memory/atomics. Only a few fringe code bases will use shared/memory/atomics, so I'd call async/await the only major feature.

In 2016 we got 0 major features. The only things in the entire spec (an entire year of language progress) was Array.includes and an exponent operator (). Both of those are very clearly "minor" features.

So in the last 3 years, I'd say we've had 2 or 3 major, mainstream features, that's very far from "several major features every year". Especially 2016 where there were 0 major features.

In 2016, browsers were still working on implementing es2015 (I'm still waiting for proper tail calls on non-Safari engines).

Atomics is a huge feature (especially in the amount of work required). getOwnPropertyDescriptors is also a big addition.

Async iteration is a very big deal that can potentially affect things like reading files in node. Object spread seems big, but is actually far more simple as it is a syntactic special case of Object.assign(). In contrast, the far reaching repercussions of adding async iteration and the implementation are both large things. Given the level of optimization for JS regex, adding a bunch of new features there is also a big job. Promise finally is also a big update (though combining map and flatmap in promises automatically is the biggest issue with them).

This year, there's integers, working imports, and big class upgrades in the works. The list of major proposals is rapidly shrinking.

Is there another major language adding that many big features in the past three years? Keep in mind that most stage 3 proposals already have at least one implementation already done.

Velocity of new features being added is not a metric of quality.

I tend to agree with you. The thing is that most people would agree but at the time everyone wants different features. Me personally I even think we could be better without classes and all that "pretend you're writing Java" nonsense

> I even think we could be better without classes

Absolutely. Classes are also a big part of why there's still a lot of bikeshedding for some new features (class properties, decorators, private fields, etc).

Classes are familiar to most developers these days, but they're an absurdly complicated construct. The current implementation in javascript is "maximally minimal", in that the TC39 pushed the minimum they could that was still kind of sortoff useful to get the wheel rolling...but it's still not good enough. They'll need several more updates before they're truly useful. And even when that's all there, they still won't offer much that wasn't already doable, arguably better if less familiar.

At the end of the day, ES6 classes were designed back in the days where OOP was getting big in JavaScript. It took so long to release that by the time they were, the field had changed a lot.

I tend to prefer more utility/function approaches as much as possible. The exception to sometimes you need to carry context in a way often more simple (in terms of source complexity) to use class syntax. Not in every case, but some.

I see this idea that adding classes to JS was a bad thing a lot, but even though I don't ever use classes in my codebase, I don't know why it would make sense to remove them before adding new features, or to stall new features because they exist and I don't like them.

In my opinion, we should be moving forward with features that make functional paradigms easier, such as the pipeline operator, regardless of the fact that JS has object oriented features. There's no reason to stall new features just because different people want different features, at this moment we're pleasing no one with the slow release cycle.

I'd rather that JavaScript as a community take a break from the framework wars. The amount of gained productivity occurring from new frameworks being created is miniscule, and in some ways it detracting. Meanwhile, other language communities have settled on a few frameworks and have allowed them to mature. Out of the gate, React already spawned a bunch of clones that claim to do things better. I'd bet good money that in 3 years React will be tossed aside like AngularJS for the next thing announced by the predominant tech company of the day. Hey, maybe one of Elon Musk's companies makes a JavaScript framework.

I'd rather that JavaScript continue to add features so that it remains relevant. I like its ubiquity and the the fact that it's both a server and browser language, and would hate to see it get completely railroaded by other language runtimes that can be compiled to WASM.

I wont agree or disagree yet, but why?

Because it's easy to add features, but impossible to take them away.

Been using this in LiveScript for awhile (ls also provides it in the opposite direction <| ). Very useful for chaining functions in a logical order and without unnecessary () nesting. Absolutely needs to be paired with partial application though for multiple argument functions.

Yeah, partial application is also a stage 1 proposal now. But for now I guess you could also do with auto-curring functions (https://ramdajs.com/ or something like that)

FWIW, several language communities call this "the Thrush combinator"[0]. Armed with that term, searching the web for background/info/theory on this topic will be easier.

0 - https://debasishg.blogspot.com/2009/09/thrush-combinator-in-...

Thanks! Sounds a bit too scientific to my ear but good to know

It is what it is, and what it is in JavaScript is this:

  Object.prototype.pipeTo = function (functor) {
    return functor (this);
    };

  function inc (input) { return input + 1; }

  // this will write out "2" to the console log
  new Number (1).pipeTo (inc).pipeTo (console.log);

HTH

Interesting. This proposal would for the first time allow any javascript to be written in just 5 characters:

  []+|>

Down from the 6 that jsfuck requires:

  []+!()

> "Expressions Math.sqrt(64) and 64 |> Math.sqrt are equivalent. The difference is in how we read it. With pipeline operator data flows from left to the right, and thus making it much more comprehensible"

Ugh, no, that's a very subjective statement, 64 |> Math.sqrt is not "much more comprehensible" assuming especially for beginners.

I agree that this example isn't convincing, I also prefer the "vanilla" Math.sqrt(64).

It's more convincing (IMO) when chaining calls, eg:

Math.sqrt(Math.ceil(Math.sin(64))) VS. 64 |> Math.sin |> Math.ceil |> Math.sqrt

I used to feel that way until I got comfortable with Elixir and its pipe operator. I think without prior knowledge the latter, which is interpreted left-to-right, is more intuitive: "take some data, do x, do y, do z"

>The difference is in how we read it. With pipeline operator data flows from left to the right, and thus making it much more comprehensible without the need to introduce extra variables.

Why you need operator? Can't it be done with a function?

    pipe(64,Math.sqrt)

And if you have 10 of them in a row then you have: pipe(pipe(pipe(pipe(pipe(pipe(pipe(pipe(pipe(pipe(64,sqrt),add),divide),subtract,....)

I don't know about you, but I wouldn't want to debug that.

Javascript functions can have variable number of arguments.

    function pipe(...args) { }

    pipe(64,sqrt,add,divide,subtract)

Your example falls apart when making multiple function calls.

Not quite sure what you mean. But how do you chain multiple functions?

With the way syntax for function calls works, the innermost thing is executed first, and you end up reading stuff from right to left:

   third(z, second(y, first(x)))

With the pipeline operator (and assuming a language that allows partial application), the order in which things execute matches the order in which you read it:

  first x |> second y |> third z

The benefits start getting more noticeable when you're talking about stuff like manipulating sets of data. For example:

  myList
  |> filter (x -> x.color == 'red')
  |> sortBy (x -> x.count)
  |> first

compared to these sorts of pyramids of doom:

  first(
    sortBy(
      x -> x.count,
      filter(
        x -> x.color == 'red',
        myList
      )
    )
  )

Please explain.

64 |> Math.sqrt becomes pipe(64, Math.sqrt)

64 |> Math.sqrt |> Math.sin becomes pipe(pipe(64, Math.sqrt), Math.sin)

Assuming pipe was smart and took varags we can make this better.

64 |> Math.sqrt |> Math.sin becomes pipe(64, Math.sqrt, Math.sin)

I guess you could make pipe variadic:

    function pipe(val, ...funcs) {
      for (let f of funcs) val = f(val);
    }

EDIT: seems like I got ninja'd by an edit :)

A pipe operator compiles directly into the AST and completely disappears before execution.

    64 |> Math.sqrt |> Math.sin
    Compiles in the AST to
    Math.sin(Math.sqrt(64))

    //curries variables away from functions
    var realPipe = (a, ...args) => (...initParams) => {
      var acc = a(...initParams);
      for (var i = 0; i < args.length; ++i) {
        acc = args[i](acc)
      }
      return acc;
    }

    //if first parameter is a function, curry to delay execution
    //if first param is not a function, curry, but call immediately
    var pipe = (a, ...args) => 
      (typeof a === "function")
        ? realPipe(a, ...args)
        : realPipe(...args)(a)

I'd note that this pipe implementation also bleeds its abstraction. JS has first-class functions, but this will never treat a function as data. You must be aware of how it works so if you are passing a function (for example, a getter function), you must manually curry it afterward.

This complication is why most pipe implementations don't actually mess with that data attribute part (they just skip to the realPipe implementation), but since the pipe operator can handle it, I figured I'd try to as well to show the issues.

I see no reason our function form couldn't do the same magic behind the scenes. Really no different from a lot of C compilers in this respect and with the bonus of an absolutely trivial polyfill.

JavaScript argument list evaluation is from left to right, right?

   pipe(64,Math.sqrt,Math.sin)

and

   pipe(pipe(64, Math.sqrt), Math.sin)

are equivalent.

If you just define a simple function

pipe(val, func) => func(val)

then they wouldn't be equivalent but you absolutely could define pipe so they would be.

The whole point appears to be avoiding nesting parentheses.

Piping is relatively easy to write a function for. Here [0] is a package that I wrote that has a few trivial functions (including pipe) for things such as this.

0: https://www.npmjs.com/package/afpf

He was whining about lodash but still used it till the end.

His lodash and pipeline operator version looked more noisy than the lodash only version.

My only real criticism, is it would be nice if it supported generators/promises as an alternative to for-await syntax.

IxJS [1] already has pipeable functions for working with AsyncIterable (the underlying interface for the proposed for-await-of syntax).

[1] https://github.com/ReactiveX/IxJS

rxjs is already available and allows to perform chained transformations by using wide variety of methods.

rxjs encourages a pipe() function approach as mentioned in other threads. It would still benefit from a pipe operator replacing that function, as the operator version would be easier to type (in Typescript and Flow) and to some of us aesthetically cleaner. The pipe operator that this article is hoping gets adopted is just a generic version of the rxjs pipe() function.

Good point. I've heard about it but never could quite understand if it's useful enough. Can you share some public repos where it's being used?

I've been converting our eCommerce site to using it. It's OK. Way too complicated for the average developer, it's full of jargon that is not usually present in JS/web development. The use cases they have on their site do not reflect how it is actually used, I spent a lot of time wondering how to translate their examples of clicks and timers into my problems of API calls and route changes.

The more I think about how I use it, the more I realize I'm using it to clean up the leaky abstractions from our backend. If our backend was better, promises would meet my needs fully - get the data, stick it in the views. But no. I have to do all these bullshit transformations and call multiple APIs because apparently Java Spring apps are EXTREMELY DIFFICULT to develop...

the official web-site is pretty informative and has a ton of samples..https://rxjs-dev.firebaseapp.com/api - take a look at operators section for example. as for public repos - nowadays any modern front-end app that is built with Angular is using rxjs - it's de factor standard for data processing in angular world.

Thanks sir. Will take a look

You could just use ramda https://ramdajs.com/docs/#pipe

Not quite as seamless and it would be nice to have |> but ultimatles introduces a lot of complexity.

The only good reason I could see is if there is a performance benefit a JavaScript JIT could take advantage of by having the |> operator?

Honestly, I don't think so. More likely this will be just a syntactical sugar.

Performance benefits likely depend on which side of the partial application / currying coin the operator ends up landing on. Some of the partial application proposals would potentially reduce the needs for currying in the language, which would reduce the needs for a lot of variable closures in some libraries.

The biggest potential benefit is to typing (in Typescript or Flow), which potentially can provide potentially much better or at least much simpler typing and type inferencing with an operator versus many of the alternatives (such as variadic pipe() functions).

Reply on news.ycombinator.com