Elixir-style pipelines in Ruby | Hacker News Reader

Elixir-style pipelines in Ruby

gregnavis.com

63 pointsgregnavis16 comments

The beauty (and horror) of Ruby is that you can do almost anything with it. I think this is a really interesting and clever use of the "can do anything" aspect of Ruby, although I think I'd prefer not to run into it in a production app.

Still, it's really cool to see how far we can push/mold the language to accomplish different tasks and patterns.

I've had a look at threading/piping operators in a few languages (list below). I'd say that Racket has the best one I've used. I love that you can specify a '_' for the hole which the result from the previous operation will fill. Julia's threading macro is surprisingly brittle, only letting you chain single-argument functions unless you want to use anonymous functions with one bound variable and the rest free.

+ Haskell :: https://www.schoolofhaskell.com/user/Gabriel439/Pipes%20tuto...

+ Racket :: https://docs.racket-lang.org/threading/index.html

+ Clojure :: https://clojure.org/guides/threading_macros

+ Julia :: https://syl1.gitbook.io/julia-language-a-concise-tutorial/us...

+ R :: https://r4ds.had.co.nz/pipes.html

+ F# :: https://fsharp.github.io/fsharp-core-docs/reference/fsharp-c... — only works with unary functions.

+ Hack :: https://docs.hhvm.com/hack/expressions-and-operators/pipe — works with any functions, uses $$ for the pipelined value.

OCaml also has a similar thing (@@) but I can't find a good reference page.

OCaml :: https://v2.ocaml.org/api/Stdlib.html#1_Compositionoperators, it has `|>` for (unary) pipelines and `@@`.

> Julia's threading macro is surprisingly brittle, only letting you chain single-argument functions

Julia's threading/piping is surprisingly limited, but the phrasing here (combined with the link) could make things confusing to a Julia beginner. So to make things clear:

* The linked page is talking about an external Julia package which provides a macro. And that macro lets you use the `_` syntax similar to what you describe Racket as having.

* Julia's default inbuilt piping syntax is the one with the single-argument limitation, and that's an operator, not a macro.

There's been a lot of discussion about bringing the `_` syntax or something like it to the base language, but there seem to be implementation difficulties. [1]

> +R :: https://r4ds.had.co.nz/pipes.html

This page is talking about magrittr piping (which is probably still the most popular), but base R also got inbuilt piping syntax with version 4.1. It automatically passes the piped-in value as the first argument to the subsequent function. [2] [3]

[1] https://github.com/JuliaLang/julia/pull/24990

[2] https://www.r-bloggers.com/2021/05/the-new-r-pipe/

[3] https://michaelbarrowman.co.uk/post/the-new-base-pipe/

> I'd say that Racket has the best one I've used. I love that you can specify a '_' for the hole which the result from the previous operation will fill.

Clojure also has a placeholder variant, the as-> form. Quoting the example the Clojure doc page (https://clojure.org/guides/threading_macros):

  (as-> [:foo :bar] v
     (map name v)
     (first v)
     (.substring v 1))
  
  ;; => "oo"

Check out this proof of concept gem to perform pipe operations in Ruby using block expressions:

https://github.com/lendinghome/pipe_operator#-pipe_operator

  "https://api.github.com/repos/ruby/ruby".pipe do
    URI.parse
    Net::HTTP.get
    JSON.parse.fetch("stargazers_count")
    yield_self { |n| "Ruby has #{n} stars" }
    Kernel.puts
  end
  #=> Ruby has 15120 stars

I like using .then for chaining stuff, like this:

    sig { params(obj: T.untyped).void }
    def write(obj)
      obj
        .then { Array(_1) }
        .then { @wrapper.pack(_1) }
        .then { @deflate.deflate(_1, Zlib::SYNC_FLUSH) }
        .then { @body.write(_1) }
    end

While not as thorough solution as mentioned here in comments, UFCS goes a long way in this direction. In Next Generation Shell, I've designed the methods so that the first parameter is something that is likely to come from "the pipeline". Hence mylist.filter(...).map(...) just work. Combined with multiple dispatch and the fact that methods don't belong to a particular type/class, it allows creating user-defined methods with same convention to work with any existing and new types/classes.

UFCS - https://en.m.wikipedia.org/wiki/Uniform_Function_Call_Syntax

Do the steps of the pipeline implementation in Ruby run concurrently?

I once did an Elixir course and really liked the pipelines. I continued implementing pipelines with a Scheme macro, but not concurrently.

Elixir pipelines are not concurrent, unless you use something that allows that (flow and probably stream behind the scenes)

Not naively: this kind of pipelining is partial function application, not coordinated parallelism (the way Unix pipeliens are).

Does anyone know any documentation on how UNIX achieves parallelism in pipelines, especially modern implementations? Is each stage of a piped command just another spun off process?

Yes, it's just that: each process in the pipeline is spawned, and then blocks as it waits for input from the previous step in the pipeline.

I think a better way to think about it is that the UNIX model requires parallelism: without parallelism, each stage in the pipeline would need to fully buffer its intermediate state before forwarding it to the next stage.

> without parallelism, each stage in the pipeline would need to fully buffer its intermediate state before forwarding it to the next stage.

That's how DOS implemented the pipe syntax I believe.

> Is each stage of a piped command just another spun off process?

Yes. With, as the name says, a pipe inbetween.

Reply on news.ycombinator.com