Comment by dhosek | Hacker News Reader

Comment by dhosek | Hacker News Reader

On the pipability front of software, I think a lot of that happens through APIs now, either directly linked or through various network APIs. The gotcha is that few APIs can handle the kind of asynchronicity that a pipe can. With pipes I can so something like:

    oaktree | nutgrinder | cookiemaker | cookieeater

and as soon as each nut is ready from the oaktree, it can be consumed by the nutgrinder and it can send nut flour to the cookiemaker as its ready and when the cookiemaker has enough nut flour to make a cookie, it can send a cookie to the cookieeater. Four processes, acting in parallel¹ with little friction to make it happen.

Now, try to architect this in your favorite programming language.² I did see where Google proposed something (almost a decade ago) for C++ to allow this https://isocpp.org/files/papers/n3534.html although I don’t know where it stands now. I’ve been thinking along something similar for finl for each of its stages so that, e.g., parsing and tokenization doesn’t need to be complete to start formatting, but it would be nice if there were a simple language facility to allow this.

REST APIs also fail at this insofar as you need to wait for a whole request to complete before you get a response rather than being able to send the request and get each chunk of data back as appropriate. We seem to have forgotten so much that we knew thirty years ago.

⸻

1. Although not necessarily. IIRC, good old fashioned DOS pipes would wait until each process was done before opening the next application in the chain.

2. Assuming, of course, that shell scripting isn’t your favorite programming language.

>REST APIs also fail at this insofar as you need to wait for a whole request to complete before you get a response

This isn't a restriction from REST. Nothing stops you from implementing an API that starts responding before your request is finished.

The main problem I imagine is that you do not have a real way to deal with errors; you need to reply with 200 OK before you send anything else. There's also no separate channel for errors analogous to stderr.

> Now, try to architect this in your favorite programming language.

Isn't this trivial in java with streams? Or in C# with linq? Maybe I'm missing something.

Or in C++20 with range adaptors.

  for (const auto &nut: oaktree
       | std::views::transform(nutgrinder)
       | std::views::transform(cookiemaker)
       | std::views::transform(cookieeater))
  {
    // do something
  }

...or in Python with generators/iterators. Or in Lua with coroutines. Or in Rust with iterators...

I guess most higher-ish languages at some point gain support/framework for something like this. But the point stands: there's something both simple and powerful with pipeline abstraction. I think of pipeline as a way of thought. Something that bridges the gap between "user" and "programmer". It's no wonder some of that "way of thinking" was implemented into programming languages (with obvious and non-obvious implications, not limited to memory requirements).

These are all syntax though (except perhaps Lua’s coroutines—I don’t know enough to say about those). It’s not the syntax that’s the wonderful part about pipes, it’s the parallelism. See my other post with some Java code in this thread.

The individual steps in a java stream don’t run in separate threads/processes. So, for example. given the following code:

        IntStream.of(1,2,3,4,5,6,7,8,9)
                .peek(System.out::println)
                .boxed()
                .map(val -> {
                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                    return "a" + val;
                })
                .peek(System.out::println)
                .map(val -> {
                    try {
                        Thread.sleep(3000);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                    return "b" + val;
                })
                .forEach(System.out::println);

the forEach essentially acts as the driver for each step so that you’ll see everything happening in sequence. The equivalent shell script version of this would output in the order (approximately) of 1, 2, 3, 4, 5, 6, 7, 8, 9, a1, a2, a3, ab1, a4, a5, a6, ab2, a7, a8, a9, ab3, ab4, ab5, ab6, ab7, ab8, ab9 and finish in roughly 28 seconds instead of the 36 seconds that the streams version does. Streams are syntactic sugar and do not provide parallelism on their own.¹

⸻

1. And don’t get me started on the hot mess that’s the Java parallelStream

But don't pipes also block when the buffer (~1-4kb) is filled until the output can be consumed? So as I understand it's still sequential in the bash version, only there's a bit of buffering at play.

Tangential but just curious, do people make cookies out of acorns? Beside squirrel people that is.

Native Americans as well as others have used acorns for flour and such https://en.m.wikipedia.org/wiki/Acorn

Go handles this very cleanly with channels and goroutines.

I literally read this chapter in The Go Programming Language last night after posting this and thought to myself, well, shit, there it is.

Reply on news.ycombinator.com