Hacker News Reader

Comment by mkj | Hacker News Reader

You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.

View on HN

Actually yes should use vmsplice().

And pv on the other side of the pipe should use splice().

Now that would be a complete zero-copy I/O path, purely limited by the CPU, not by memory bandwidth. It would benchmark at hundreds of GB/s :)

> And pv on the other side of the pipe should use splice().

Good catch, I just noticed that it does.

Someone posted an implementation which uses vmsplice on the reddit thread:

https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

His code can further be optimized by having only 1 large element in the iovec (vs. many 2-byte "y\n" elements).

from what i can see it's already in a large iovec buffer

Correct, I misread the code. However bumping the pipe size buffer bumps the speed from 28 to 74 GB/s on my Skylake CPU (i5-6500).

Edit: I optimized further and now get 123 GB/s See https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).

On the other hand the current code is perfectly portable, tee(2) is a linux syscall.

Still gets piped and hits same performance.

It's 2x times faster for a quick test here. Only copying on the read side?

Out of interest: Could you please post the code?

https://pastebin.com/jrcJbjU4

I've realised straight tee() is actually wrong - it works fine for piping to something but "zcyes > /dev/null" won't work, it needs vmsplice() like mrb said.

  core2duo e8400, 4.8 kernel, coreutils 8.25
  > yes | dd bs=1M count=10000 of=/dev/null iflag=fullblock
  10485760000 bytes (10 GB, 9.8 GiB) copied, 4.1901 s, 2.5 GB/s
  > ./zcyes | dd bs=1M count=10000 of=/dev/null iflag=fullblock
  10485760000 bytes (10 GB, 9.8 GiB) copied, 1.8542 s, 5.7 GB/s

Reply on news.ycombinator.com