How is GNU `yes` so fast? | Hacker News Reader

How is GNU `yes` so fast?

reddit.com

Speed vs readability:

https://github.com/coreutils/coreutils/blob/master/src/yes.c

https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

Normally I think readability is more important than speed. But in this particular case, I think GNU is doing the right thing optimizing the code to the limit.

This is the beautiful part of Unix: small tools that do only one thing well. Programs following this philosophy are very good abstractions. They do one very well defined thing so you can use them without having to understand how they work. I have used Unix for years and I've never felt the need to read the source code for `yes`. And because they do a very small thing, even if you need to read them, the overhead of optimization is not that much, for example, the optimized GNU yes is just under 100 LOC if you remove comments and help boilerplate. Yes, it's longer than the BSD version, but it's just a matter of minutes to understands what it does.

I totally disagree. Nobody will ever want to use `yes` at 10 GB/s. They will want it to be reliable, and this sort of over-optimisation increases the risk of bugs.

I'm having a hard time imagining a case where even a grossly complex 100,000 line implementation of yes(1) couldn't be trivially proven to be correct.

Absolutely, I once trivially proved a 1000,000 line implementation of return 0 to be correct. I don't know why all comments are bothered by how much overkill this yes implementation is. Maybe they don't hold 100,000 PhDs like we do, am I right?

I agree with the general point that you could prove such a simple program correct relatively easily but that does still have a cost, which is always a concern in an open-source project. You still need someone to step up and do that work and continue to verify it periodically in the future – if that code is doing complicated things with buffering, that opens up possible odd bugs due to stdlib, gcc, maybe even kernel behaviour changes which might not affect simpler programs.

Not a huge bit of work to be sure but for a non-commercial project you might have trouble finding a volunteer who cares about that tedium.

Easy! Write a program that does a brute force check of Goldbach's conjecture on all integers. For each integer that passes the check, print a line. If you can prove this correct (or incorrect) you'll probably win a Fields medal.

The ability to get 10 GB/s dummy date into a pipe from the command line could come in handy at some point, for stress testing or something. I am not sure if it is over-optimized. (And even with the optimizations the risk of bugs should be very small.)

Of course you could just pipe from /dev/zero which will easily do 10GB/s on every machine.

Just tried it,

    cat /dev/zero|pv >/dev/null

gives me roughly 2/3 the speed of yes. (5 GB/s vs 7 GB/s) Plus yes gives you an arbitrary string, instead of only zeroes.

That's because you're using cat and a pipe. Try this instead:

  pv > /dev/null < /dev/zero

For anyone who is like me and finds it uncomfortable that things are now out of order, note that you can still put the input redirection in front:

  </dev/zero pv >/dev/null

Or even this:

pv </dev/zero >/dev/null

which is a common way of doing it (for any command with any inputs and outputs, not just the above ones), i.e.:

command < input_source > output_dest

All three pv command invocation variants, the one here and the two above, work. And it becomes more clear why they are the same, when you know that the redirections are done by the shell + kernel, and the command (such as pv) does not even know about it. So in all three cases, it does not see the redirections, because they are already done by the time the command starts running (with its stdin and stdout redirected to / from those respective sources). And that is precisely why the command works the same whether it is reading from the keyboard or a file or pipe, and whether it is writing to the screen or a file or pipe.

Tobik already wrote it

but here's todays Useless Use of Cat Award http://porkmail.org/era/unix/award.html

Well, sometimes using cat is faster, as I discovered to my surprise recently:

https://news.ycombinator.com/item?id=14414610

But in that case one could probably say that Gnu awk is just not very good at input handling (as Mawk doesn't appear to benefit from an extra "cat").

I believe /dev/zero writes data one byte at a time; that's likely the reason why.

[edit] That's actually inaccurate (and badly expressed), see comments below.

/dev/zero doesn't "write" anything in the sense that yes writes, since it's a character device and not a program. The Linux kernel's implementation of /dev/zero does not write one byte at a time.

You're right of course; and actually I believe the kernel will simply provide as many bytes as the read() requested; so the speed should mostly depend on how you access /dev/zero. IE, the user above was using cat and I think with dd and a proper block size it'd be much faster.

I was under the impression that cat automatically used a sane size for reading. Now that I think of it I cannot think of a source, other that to point out my own anecdotal experience.

When I was writing raspbian images to SD cards to use on a raspabery Cat and DD took within a few seconds of each other on an operation longer then a minute. Since then I have been using cat where I could, but I didn't think to right down the numbers though.

For something somewhat related, see parts of this thread:

https://news.ycombinator.com/item?id=14414610

Note that cat+gnu awk was faster than just gnu awk - but mawk was faster still (reading a not entirely small file).

And in a similar vein of gp comparing Gnu and Openbsd, note that openbsd cat is a little more convoluted than the simplest possible implementation (at least to my eyes):

https://github.com/openbsd/src/blob/master/bin/cat/cat.c

https://github.com/coreutils/coreutils/blob/master/src/cat.c

(That is, Gnu "cat" and OpenBSD "cat" are less different than Gnu "yes" and OpenBSD "yes").

yes will give repeated data though, not just zeros, seems like it's more useful here - as others have pointed out, also uses less syscalls than /dev/zero.

I've used 'yes' many times to generate huge amounts of data quickly. Back then, it never had the small string optimisation, but you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/urandom or even /dev/zero

I'm glad it runs fast, and I hope that all OS utilities are optimised (and tested, of course!) instead of making their source code pretty. The fact is, most people want to use programs, not read them.

> you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/zero

That really doesn't make any sense, /dev/zero should be at least as fast as yes.

/dev/zero should be at least as fast as yes

I agree, all I remember is that when I tried it, /dev/zero sometimes sucked performance-wise. I can't recall the exact circumstances as it was some time ago, and could have been on any of Linux/FreeBSD/SunOS/HP-UX/IRIX - perhaps it was the fastest common way at the time?

On a recent x64 Linux, /dev/zero seems plenty fast enough now:

  $ dd bs=8k count=819200 if=/dev/zero of=/dev/null
  819200+0 records in
  819200+0 records out
  6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.331137 s, 20.3 GB/s

  $ yes | dd bs=8k count=819200 of=/dev/null
  819200+0 records in
  819200+0 records out
  6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.959551 s, 7.0 GB/s

No need for "dd", let "pv" get the data from /dev/zero directly:

$ pv < /dev/zero > /dev/null [ 16GiB/s]

But the version of yes using vmsplice() is even faster than that on my machine.

What's the line to test `yes` with `pv`?

    pv < /usr/bin/yes > /dev/null

doesn't seem to work properly. FWIW I get 330MiB/s vs 8.4GiB/s for /dev/zero.

[Incidentally first I've heard of pv but I've known about dd for a decade or two].

Maybe this?

    yes | pv > /dev/null

That's not quite apples-to-apples though:

cat /dev/zero | pv > /dev/null

uses a pipe like the 'yes' line above, but runs substantially slower on my computer than the redirect-only version.

< and > are for file redirection, yes is a binary so you want to pipe its stdout into pv:

    yes | pv > /dev/null

I know this works, but how come we can see the output of pv when it is redirected to /dev/null? Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(

You're redirecting stdout to /dev/null and pv is writing to stderr. If you use &> instead, stderr and stdout will both be redirected to /dev/null and you will see no output at all.

> I know this works, but how come we can see the output of pv when it is redirected to /dev/null?

From pv's man page:

> Its standard input will be passed through to its standard output and progress will be shown on standard error.

> Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(

The Windows/DOS command line has the same concepts[0], though it's probably used less often: by default a process has 3 FDs for STDIN (0), STDOUT (1) and STDERR (2).

At a shell

* you can feed a file to STDIN via <$FILE (so `</dev/zero pv` will feed the data of /dev/zero to pv), the 0 is optional

* you can pipe an output to a command (other | command) or the output of a command to an other (command | other)

* you can redirect the STDOUT or STDERR to files via 1> and 2> (the "1" is optional so 1> and > are the same thing) (you can redirect both independently)

* you can "merge" either output stream to the other by redirecting them to &FD so 1>&2 will redirect STDOUT (1) to STDERR (2) and 2>&1 will redirect STDERR (2) to STDOUT (1), you can combine that with a regular redirection so e.g. `command >foo 2>&1 ` or with a pipe (`command 2>&1 | other`)

And you can actually create more FDs to use in your pipelines[1] though I don't remember ever seeing

[0] https://support.microsoft.com/en-us/help/110930/redirecting-...

[1] http://tldp.org/LDP/abs/html/io-redirection.html

So how did that redirect even work; should we be doing a `mknod` to make a "yes" device to make the comparison work (can we, does it help other than in my naive imagination).

The reason that redirect worked is because it was using the contents of the yes program instead of its output.

> The fact is, most people want to use programs, not read them.

I want to use safe programs, and programs with readable code are more likely to be properly audited.

Audited UNIX tools - does such a thing exist?

openbsd.org

Sounds like you want to stick to something with a non-GNU userspace, apparently.

If you want to maximize readability and simplicity then writing `yes` in C is a bad choice. It is much easier, cleaner and shorter to write it in python or just use the shell which would normally be using the `yes` in the first place. Since `yes` is used in shells and builtins can be considered very reliable, here is a implementation as a shell function:

    yes(){ while :; do echo "${1:-y}"; done; }

Python:

    import sys 
    while True:
      if len(sys.argv) > 1:
        print(sys.argv[1])
      else:
        print("y")

And if you don't need the features of `yes` and only need a loop that prints 'y', then there really is hard to beat the simplicity of:

  while:; do echo "y"; done

Is it really "easier, cleaner and shorter to write it in python"? Did you look at the OpenBSD implementation?

https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c

It's essentially line for line identical to your python code...

The C example has three includes, two conditionals, two loops, and one function definition. The python example has a single include, conditional, and loop.

For readability purpose it is easier to go through each lines of the python program than the OpenBSD C code. Its not massively different, but its distinguishable enough that I would choose the python version if I wanted to maximize readability, minimize syntax requirement and did not want to use shell script.

The Shell function is in my view the superior choice if the audience is a programmer than know the shell script syntax. It is just a single loop and is written in the environment that the program is intended to be used in. The only drawback there is the speed.

Most of what makes the C program bigger comes from the fact that the C program does more. Your python example doesn't call pledge(). Remove that from the C program and it drops to one include, one conditional, and two loops. Further, counting the two loops against C doesn't make any sense: it's entirely up to the programmer whether to have a conditional containing two loops, or a loop containing a conditional. Both languages could naturally do it either way.

> The python example has a single include, conditional, and loop.

... and python. Don't forget to count python.

Exactly. That's the reason I write stuff in C instead of my favorite interpreted language, Ruby. When you write something in C, that's it. No large interpreter plus runtime needed.

This is innificient. Why do the argv check in every iteration of the while loop? It's not going to change between iterations.

> Nobody will ever want to use `yes` at 10 GB/s

Just because you say so?

If the speed of yes is bounded by memory speed, doing anything useful will almost certainly consume that data at a far lower rate. Putting it on a disk, pushing it over a network, etc. will almost always be slower than yes is able to generate data.

The typical use case is piping it into another running program. Maybe someone wants to do that really quickly rather than putting it on disk or pushing it over a network.

'yes' to a file is how I sometimes benchmark disk speed. It should have fewer system calls than reading from /dev/zero and then writing.

I actually checked just now, and it looks like you'd be making twice as many system calls with /dev/zero compared to generating the data locally:

    strace dd count=1 bs=512 if=/dev/zero of=test
    open("/dev/zero", O_RDONLY)             = 3
    dup2(3, 0)                              = 0
    close(3)                                = 0
    lseek(0, 0, SEEK_CUR)                   = 0
    open("test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
    dup2(3, 1)                              = 1
    close(3)                                = 0
    read(0,"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
    write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
    close(0)                                = 0
    close(1)

While 'strace yes > test2' is just a constant stream of write() calls.

The difference matters if you're benchmarking e.g. some new SSD compared to a tmpfs on a machine with 100+ GB of RAM. It's always better if the tools have less overhead, because the comparison is more meaningful.

Also consider that it can be faster to write to a local network than to disk. I've never done it, but I imagine that the kernel's not going to want to deal with your /dev/zero calls if it's spending all of its time writing to a 10GB switch. I can imagine some very specialized storage servers that could spend most of their time writing from memory buffers to a network switch, or if you're troubleshooting a slowdown in the networking itself.

When I started this comment, I didn't think you were measuring what you thought you were measuring with those straces. 'strace yes > test2' only watches 'yes', not '> test2' (which is handled by the shell). Here's what your command outputs:

    $ strace yes > /tmp/test2
    [various initialization steps]
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [...]

To measure everything, I started up a whole new shell, expecting to see a 'write(1, ...)', a 'read(1)', and a 'write(f, ...)':

    $ strace -f sh -c 'yes > /tmp/test2'
    [various initialization steps]
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [pid 15839] write(1, "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n"..., 8192) = 8192
    [...]

How does this possibly work? File descriptor 1 is supposed to be the terminal, not a file! Of course, the magic of file redirection:

    open("/tmp/test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 # Open the file, get FD 3
    fcntl(1, F_DUPFD, 10)                   = 10 # Copy FD 1 (the terminal, STDOUT) to FD 10 temporarily
    close(1)                                = 0  # Close original FD 1
    fcntl(10, F_SETFD, FD_CLOEXEC)          = 0  # Close FD 10 (the terminal, copy of STDOUT) when exec() is called
    dup2(3, 1)                              = 1  # Copy FD 3 (the file) to FD 1 (STDOUT)
    close(3)                                = 0  # Close FD 3 (the file's original descrptor)

I had the impression that you thought "yes" pipes the output to shell, and then the shell writes to disk. That's incorrect. " > file" means redirecting to a file, and therefore all write system calls actually write to disk.

That's exactly what I thought. Now, had you asked me how redirection worked I would have said "the redirection operators cause the shell to attach a file to the descriptor," but I'd never actually thought through the implications of that in terms of what syscalls get made, and the output of strace presented a rather visceral demonstration of the implications of this clever bit of design.

Hey, 640K is enough for anyone!

Unix userland tools have evolved over decades to be as efficient as possible because they have historically underpinned everything the operating system does. The faster each tool works, the faster the other tools that depend on them work. If increased efficiency results in a bug, that bug can then be fixed, making it a net gain for system stability.

It's not about running 'yes' at 10GB/s, it's about less overhead to do a simple job. If this version of yes is 100x faster, that implies it using 1% of a cpu to do the same work that would otherwise occupy 100% of a cpu. This leaves more of the machine to do what is likely to be the intended task.

It's not really optimized to the limit – or perhaps it is, but then the limit is fairly easy to reach.

When I saw this item here I reached for the terminal and wrote a simple (and non-compliant) alternative that simply initialises 8150 bytes of "y\n" and then loops a write forever. I understand that it is not a fully standard-compliant yes, and that maybe GNU yes is indeed fast, but that awfully simple program that takes all of 10 lines (everything included) and took me all of a minute to write performs just as well as far as pv is concerned.

(I eventually completed a feature complete yes but I still think that simply not using `puts` is hardly optimising to the limit.)

One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written.

The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput.

It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case.

This is the first revision:

https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46...

(joke) They copied the API of the main() function... :-)

And here is the madness: :)

https://github.com/coreutils/coreutils/commit/35217221c211f3...

And the motivation: it's used as test inputs. Not sure I agree with completely destroying the readability of a perfectly readable file - but ok.

The motivation, as far as I see, was only that it "may be used."

But see the post of belorn where he argues that the error handling seems to be good.

Is the date on that accurate, did that happen in 2016? Kinda hurts the theory that they were trying to avoid Unix similarity.

It says: Mar 9, 2015

One point in favor of this simple version is that it's immediately obvious that it doesn't do the same thing as the OpenBSD version. In OpenBSD `yes a b c` will only print "a" while in GNU it prints "a b c". I did not catch that when I was reading the more complicated modern version.

Came here to post this, so instead I'll just back you up with a source:

https://www.gnu.org/prep/standards/standards.html#Reading-No...

"Add a programming language for extensibility and write part of the program in that language" wow, that's just asking for trouble...

Yeah, I mean, who even uses emacs?

Youngster!

Remember that "yes" is the bottom end of complexity here, not the top. As the utility grows larger that stops looking like "asking for trouble" and starts looking like "often the only sensible solution". Expecting people to extend programs in C is often "asking for trouble", after all!

(Lately I've been really tempted to pull a Cato and start terminating my every HN post with "C delenda est." https://en.wikipedia.org/wiki/Carthago_delenda_est)

Never go full Cato.

That's the LISP programming philosophy of the creator and host of this web forum

Does writing it in a different language suffice?

Depends how different.

At the risk of sounding like a copyright newbie, shouldn't that be covered by just doing a 'clean room' implementation? As long as you can verifiably prove that you didn't copy the source, it should fall under general use (as there's really only one way of doing such a thing), right? Much like Apple can't patent rectangles, although they tried.

Even if you eventually "win" you already lost when plausible litigation began.

This is why Linux and GNU has won.

It's just a trade off. For utilities whose behaviour doesn't change we're happy to improve the speed.

IOW, Can't leave anything alone.

You're always welcome to port old, slow utilities forward. That's the beauty of open source!

This work certainly has value but it is frustrating to keep up with everything being changed. And if changes affect me do the fashions and attitudes of those making the changes have synchronicity with the way that I use computers?

In the old days I think we would have left yes in c because the compiler will build it on any platform.

My first experience with Linux is porting land.c to one of the commercial Unix. Now I first met TCP/IP on a 3B2 and later met systems sold by SMI and DEC and SCO and we all mostly constructed packets the same way and a guy called Stevens had written some nice books about this that everyone had. I think I recall the commercial and free BSD also did things the usual way. But whoever figured out this interesting phenomenon land.c demonstrated happened to be a Linux user and this platform had some different ideas about it. I saw rewriting the relevant parts as an annoying, menial task.

And it's also a source of many bugs (in the past and likely in the future as well). In most use cases today I'd rather have a slightly slower userland which is easily read (and audited) then one which compromises quality for speed in edge cases.

According to the article/experiment you don't have to bork up the code much, just copy your stuff into a big buffer before you start printing it (and print using write(2) instead of puts)

This works easily for the default case, which prints "y\n" (two bytes), which is likely to divide BUFSIZ. To handle the general case with the same efficiency, you have to have a buffer size that is a multiple of both BUFSIZ and the length of the string to be printed. It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).

> It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).

Why would it be slower to do a single, say, 8190 bytes write instead of 2730 3-byte writes?

Generally writes not aligned to cache line are slightly slower on most common architectures, vastly slower on others. (Such as many MIPS)

Small write calls themselves incur a considerable syscall overhead.

I am wondering why the GNU version uses

atexit (close_stdout);

Aren't all streams closed at exit? And why closing stdout, anyway?

It's to make sure the buffer is flushed. Streams are closed, but not explicitly flushed at exit. Stdout only because it's the only filehandle with a buffer in yes.

In normal use, GNU `yes` does unbuffered IO on stdout. However, it does use buffered IO for --help and --version messages; it sets atexit(close_stdout) to cover both of those cases at once, rather than handling them both separately.

Yeah I got a surprise when I wanted to see how strlen was implemented:

https://github.com/lattera/glibc/blob/master/string/strlen.c

If you're on amd64 that's the wrong file:

https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86...

The pcmpeqb instruction is from SSE 4, it compares 16 bytes per op

I wonder how much of BSD is written in this canonical style.

I think unix v6 was mostly as clear. Linux suffered the real world penalty. But maybe BSD managed to keep it's source poetic.

In general I love reading the code base of BSD-systems, NetBSD in particular is really beautiful and easy to follow.

I generally look at some BSD sources when I want to know how some unix tools are working, they are always much much more readable than the GNU equivalent.

Mah, how many times do you read a program and how many times do you execute it?

A nitpick, but I notice that the BSD implementation do not catch errors when printing. In theory it could get EINTR and only write partial amounts of argv[1], especially if the argument string is really long and the program runs for a extended amount of time. The GNU version do catch EINTR.

Naturally it could be that the C function puts used by most people in OpenBSD is implemented with built in EINTR catching loop, or that OpenBSD do not interrupts writes.

You'd have to check the various POSIX standards to be sure - and have to then verify that the OS/libc actually follows them, but you can pretty much rely on every libc's I/O wrapper functions to handle interrupted system calls or incomplete writes. I've never seen any code check the return value of a printf() to verify that all the characters were printed.

As you say, the GNU versions definitely handle EINTR - the linux man page for puts() just says it returns a non-negative value on success, it's not even specified whether it returns the number of bytes written or not.

`puts` is an "stdio" function, not a system call. It won't EINTR, it correctly resumes if the underlying write() EINTRs. If it does get an error, it will return EOF, then you'll have to call `ferror()` to find out which error.

But the point stands; there's no error handling in the OpenBSD version. But that could be considered a design decision; the OpenBSD version never gives up on writing data until you kill it; the GNU version bails at the first error.

I think the GNU version is surprisingly readable for what it does. I have seen code that is a lot less readable than this and doesn't have the performance benefits described in the article.

And for a third example, Busybox's implementation:

https://git.busybox.net/busybox/tree/coreutils/yes.c

The /r/programming discussion of this is interesting [1].

Someone does a Go version and gets the same speed as GNU yes. Someone else tries several languages. This person got the same speed in luajit, and faster in m4 and php. Ruby and perl about 10% slower, python2 about 10% slower still, and python3 about half that. The code is given for all of these, and subsequent comments improved python3 about 50% from his results, but still not up to python2.

[1] https://www.reddit.com/r/programming/comments/6gxf02/how_is_...

m4 is orders of magnitude slower (notice the different unit).

If you write the Python script using bytes (not Unicode), then Python 3 is faster than Python 2, at least for me. Python 3 is 20 % slower than GNU yes and Python 2 35 % slower than GNU yes.

e: I think in your last sentence you are comparing results from different computers.

This[1] comment seems directly opposite from what you wrote, with Python 3.4 (8.76GiB/s) even outrunning Python 2.7 (8.55 GiB/s).

(haven't run the tests myself though)

edit: haven't noticed the comment by dom0.

[1] https://www.reddit.com/r/programming/comments/6gxf02/comment...

I had to smile, as this thread is a microcosm of programming language stereotypes: with the python programmers tweaking code to get that extra 10% (nobody mentioned pypy..), someone trying to get javascript to work in a long running program, of course a rust implementation that isn't working well just yet (but we're all rooting for it.. set that compiler flag and rerun).. One comment about perl (turns into a one liner..). Nobody bothering to redo the Lua, Ruby and PHP. And for fun somebody throws down some fortran.

what is going on with the m4 code?

One night for fun, I wrote a fizzbuzz (without actual printing) in almost all of the languages mentionned here (except fortran), and benchmarked it with 10000 executions each.

And? Don't leave us hanging! Who wins at FizzBuzz? Also, surely 10k is just getting started?

10k was just sufficient to get actual differences. Of course I went crazy and did something like 10M, but waiting 10 minutes to get the same relative differences didn't bring anything.

Remember this is without any printing: C is fast, if I remember correctly it was something like 0.015 or 0.03s while at the other end of the spectrum I had to wait 0.1s for JavaScript. C results varied from simple to double because of the OS I guess because it was so quick.

It isn't a really significant benchmark though, for any language. It was one of those nights. Ah, and I remember a large difference between go run and go install.

> It isn't a really significant benchmark though, for any language

Sure, but it's reassuring to know that the universe is still sane, C is still faster than Javascript.

Thanks for expanding on your comment!

Found the old sources, reran the benchmark for 100 times each.

    c    0.062s
    f90  0.080s
    go   0.108s
    awk  0.116s
    lua  0.117s
    rs   0.140s
    sh   0.434s
    php  0.886s
    scm  1.323s
    py   2.965s
    rb   3.738s
    js   5.411s

My memory was bad, I also added fortran for the occasion.

Thanks for digging those up!

Pretty sure none of your implementations will beat the FizzBuzzEnterpriseEdition [1].

[1]: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...

That link is always marked as already visited and yet I always forget it exists.

this is hilarious on so many levels...

Within this thread someone pointed out https://github.com/cgati/yes/blob/master/src/main.rs. This rust version gives me 7.81GiB/s versus GNU's 7.54GiB/s.

Yeah. Rust came through (woot!), though thats some verbose low level code fu with cows?

(use std::borrow::Cow;) ?!

(I don't know rust, its on my list.. Especially with cow borrowing! They clearly are my people).

I noticed they added a comparison of Rust to gnu yes on the thread. They only got: 2.17GiB/s but on a slower machine.. (2.04 for the GNU)

Cow means "clone on write". It's a generic wrapper for types that have both a borrowed version and an owned version. For example, `Cow<'a, str>` can hold either an `&'a str` (a borrowed string) or a `String` (an owned string), and a `Cow<'a, [u8]>` can hold a `&'a [u8]` (a slice of bytes) or a `Vec<u8>` (an owned vector of bytes).

In that Rust program, Cow is being used here so if the user provides an argument it ends up with an owned vector that contains that argument plus a newline, otherwise it ends up with a borrowed byte slice that contains "y\n". That said, there's not really a whole lot of point to using Cow here since allocation is a one-time cost and nobody cares if yes starts up a few microseconds slower, but the usage of Cow is limited to just a couple of lines so it's not really complicating anything (the actual write() call just ends up taking a &[u8] byte slice, so the usage of Cow is strictly limited to main()).

tine, tiny note: it means Clone on write, not Copy, and it's String not Str.

Oops, typo. I've updated my comment accordingly, thanks.

Also, I didn't realize it was Clone-on-write. Interesting, and the documentation does confirm this. I say "interesting" because the actual operation involved is `to_owned()`, not `clone()`, seeing as how `clone()` on a `&str` just gives you the same `&str` back.

Yeah, conceptually it's closer to Clone even if it's not literally Clone.

Actually the second python version is about on par, however it's running on a slower system. He got 6.78GiB/s from GNU yes and 6.76GiB/s with python3. Some one later compares it with python2 which in fact is slower with this codebase.

    > $ python2.7 yes.py | pv > /dev/null
    > ... [7.22GiB/s] ...
    > $ python3.4 yes.py | pv >/dev/null
    > ... [8.76GiB/s] ..

I was under the impression Python was magnitudes slower than compiled native languages, even when you aren't abusing Pythonic features that are computationally expensive like list comprehensions. And this isn't even in Pypy where this loop would be JIT'd and then run natively anyway, unless cpython does some kind of JIT? Is is just that the buffer is so large that the kernel handler is taking so much of a % of the runtime that the inefficiencies of the language stop mattering?

Assuming that ‘yes‘ is primarily limited by memory performance and kernel calls, the language runtime should have little influence.

Actually I got perl beating all others by far. Easy, because you can control the buffer size.

darwin with BSD + GNU yes, vs some scripting langs:

    $ /usr/bin/yes | pv > /dev/null
    ^C00MiB 0:00:04 [23.9MiB/s] [     <=>                                                              
    $ /opt/local/libexec/gnubin/yes | pv > /dev/null
    ^C77GiB 0:00:05 [ 644MiB/s] [      <=>
    
    $ bash -c 'while true; do echo "y"; done' | pv > /dev/null
    ^C38KiB 0:00:03 [ 233KiB/s] [
    $ node -e 'while (true) { console.log("y"); }' | pv > /dev/null
    ^C57MiB 0:00:03 [ 582KiB/s] [

    $ perl6 -e 'loop { print "y\n" x (1024*8) }' | pv > /dev/null
    ^C295MiB 0:00:02 [ 139MiB/s] [    <=>
    $ ruby -e 'while true do puts("y"*1024*8) end' | pv > /dev/null
    ^C14GiB 0:00:09 [ 827MiB/s] [
    $ python2.7 -c 'while True: print "y\n" * (1024*8),' | pv > /dev/null
    ^C.3GiB 0:00:10 [1.73GiB/s] [             <=>
    $ python3.6 -c 'while True: print("y\n" * (1024*8)),' | pv > /dev/null
    ^C73GiB 0:00:05 [1.79GiB/s] [      <=>
    $ perl -C0 -E 'print "y\n" x (1024*8) while 1' | pv > /dev/null
    ^C.6GiB 0:00:08 [1.84GiB/s] [          <=>

The recent commit that sped up GNU yes has a summary of the perf measurements

https://github.com/coreutils/coreutils/commit/3521722

If anyone, like me, is wondering what "yes" is used for. You can use to pipe "y" to commands that require interactivity, so if you just want to say "y" to all the inputs, you can use "yes" to do this:

  yes | rm -r large_directory

  yes | fsck /dev/foo

I sometimes do

   yes "" | cat -n

to generate a sequence number.

Try `seq 1 n`

euske may be using one of the operating systems (one such being mentioned on this very page) where there is no seq utility.

    JdeBP ~ $seq
    ksh: seq: not found
    JdeBP ~ $

Of course, on the one mentioned there is jot.

And on bash you can simply use {1..n}:

    $ echo {1..5}
    1 2 3 4 5

If you have Bash 4.0 or above, you can include a step size:

    $ echo {1..10..2}
    1 3 5 7 9

I know its only an example but why

   yes | rm -r <>

and not

    rm -rf <>

I think rm -rf would also suppress errors in addition to confirmation messages, so presumably if you want a "soft" force...

No it doesn't.

  > rm -rf foo/bar
  rm: foo/bar: Permission denied

If foo/bar does not exist, it should be silent. Via http://pubs.opengroup.org/onlinepubs/007904975/utilities/rm.... on the -f option:

> [...] Do not write diagnostic messages or modify the exit status in the case of nonexistent operands. [...]

Yes, but that's the only error it suppresses, and that's because "not existing" isn't really an error for a tool that's trying to delete something. In my example foo/bar existed but I didn't have permissions to delete it, and as you'll see it printed out that error.

I believe "yes" has become less useful over the years, with flags like force and quiet solves this now.

On my quad-core MacBook, I've used "yes" in four different terminals as a way to quickly test a 100% CPU load.

Yes (so to speak), you can also do 'yes &' multiple times to create lots of background load (and then 'pkill yes' before anything melts...)

"yes d | mail" is an excellent way to get rid of failed cron job mails.

Why wouldn't it be `mail | yes d` if you're trying to answer "d" to all of the input requests of mail?

Because you want all the d's to be sent to mail...

Back when I worked at the Genius Bar at Apple Stores I saw a customer come in and talk to a 'Genius' about their MacBook being "slow". After a quick bit of troubleshooting, he just opened up 4 terminal windows an ran yes in all of them, and did some hand wavy explanation about diagnostics.

I'm curious why he did it. To impress people who work at the apple store?

OK, may be I misunderstood, who ran yes, the customer or the genius?

I just read this on Wikipedia (https://en.wikipedia.org/wiki/Yes_(Unix)#Uses):

> In 2006, the yes command received publicity for being a means to test whether or not a user's MacBook is affected by the Intermittent Shutdown Syndrome. By running the yes command twice via Terminal under Mac OS X, users were able to max out their computer's CPU, and thus see if the failure was heat related.

Ahh interesting, maybe he did have a legitimate reason for it.

I always assumed it was busywork to make the customer feel better.

Now if I could just prove to Apple that my computer randomly shuts down all the time... problem is, it doesn't appear to be heat related.

You know the system keeps logs, right?

Open console.app and look at the timestamp of log files when the crash occurred.

Mine just freezes, the last 0.3 second of sound is just repeated until it reboots by itself.

For reference, I had that exact same issue on my MBA - went on for years. Many times while watching youtube or doing something Garageband related.

At the time I had 10.11 on it, but that OS had itself been upgraded from 10.10.

When 10.12 came out, I decided to install fresh - so I backed everything up and installed onto a new SSD drive. The issue has now gone away completely....

Funny, I've seen that multiple times on my homebuilt Fedora system.

Hm, my old Dellbuntu laptop does the same. It's been dropped a few times in its 10 years of service, and now sometimes will shut down if I tap it too brusquely, regardless of the temperature. The BIOS will report it overheated, and i don't know what's going on.

Found him/her https://news.ycombinator.com/item?id=14544793

the limit isn't the processor, it's how fast memory is. With DDR3-1600, it should be 11.97 GiB/s (12.8 GB/s)

I don't understand this reasoning. Why is it being limited to main memory speed? Surely the yes program, the fragments of the OS being used, and the program reading the data, all fit within the L2 cache?

In general, only the CPU itself sees the L2 cache. Anything you see on another device (screen, disk, NIC etc) has been flushed out of cache.

Sure, but this is a pipe between two tiny processes, hopefully with very little else being run on the computer at the time (otherwise all bets are off for any benchmarking). There's no kind of 'real' I/O going on (in terms of stuff like screen, disk, NIC, and so on)

There's no reason that the L2 cache needed to be flushed at any point - the caches are all dealing with physical memory, rather than virtualised address space, so the fact that there are two processes here shouldn't stop the caching from working.

I am probably way behind the current state of the CPU judging by the downvotes I got so if you are saying there is no reason and the data can be written into a device without leaving the CPU I will just concede my ignorance.

Don't fret about the downvotes, these magic internet points aren't redeemable anywhere :)

It's completely possible for the data to not (all) leave the CPU. If the caches are large enough, then the pages full of "y\n" will be still resident in the cache when the next iteration of the program overwrites the same pages again. Then the CPU has no need to send the original page out to main memory.

If you did for(;;) buffer[(i++)%size] = 'y'; then you'd be correct. However, you do i/o. And the 'y's appear at the i/o driver and have to be made visible for the device it's driving, which can be anything, including a process on another CPU core in another socket. If they remained in the issuing CPU cache, I fail to see how the destination device could possibly see them. There are some devices which can snoop the cache (like SoC GPUs and other CPU sockets on some archs) but the snooping is much slower than memory bus. Writing to memory is the only way which a) guarantees the data is available elsewhere and b) is the fastest.

It is NOT limited to the external RAM speed and the best proof is that it actually uses over 40GB/s of memory bandwidth.

For each byte of data passing through pv:

1. the byte is read from yes memory to CPU registers by the write syscall handler in the kernel

2. the byte is written to kernel's internal buffer associated with the pipe

3. the byte is read back in the read syscall called by pv

4. the byte is written to a buffer in pv memory

5. and thas's the end because write syscall executed by pv on /dev/null very likely doesn't actually bother reading the submitted buffer at all

edit: Actually it might only be 20GB/s because on Linux pv seems to use the splice syscall to transfer data from stdin to stdout without copying to userspace.

This is also the reason why further "optimization" of this program in assembly was a fool's errand: the bulk of CPU load is in the kernel.

All good valid points, but I'm still a bit surprised that the limit is not higher, I thought that the L2 cache was over an order of magnitude faster than main memory (plus, as someone pointed out in the reddit thread, the peak memory performance should really be double the quoted 12GB/s due to dual channel memory).

The actual throughput then, once you include OS copying, is either 2 or 4 times the quoted speed (depending on splice usage), so we're either at main memory theoretical speeds, or double main memory speeds. Intuitively, I'd still have expected that it should be a larger multiple.

(A quick search can't find me any reliable Intel L1/L2 cache speeds/multiples to quote, so I admit this comment is more speculation than it should be!)

yes and pv processes are not scheduled on the same CPU core, so different L2 cache.

I wonder if `taskset -c1 yes | taskset -c1 pv > /dev/null` would significantly change the throughput.

    $ yes |pv > /dev/null
    46.6GiB 0:00:05 [9.33GiB/s]

    $ taskset 1 yes |taskset 1 pv > /dev/null
    32.9GiB 0:00:05 [6.58GiB/s]

    $ taskset 1 yes |taskset 2 pv > /dev/null
    45.7GiB 0:00:05 [9.13GiB/s]

    $ taskset 1 yes |taskset 4 pv > /dev/null
    45.7GiB 0:00:05 [9.18GiB/s]

Very rough numbers - the 9.13/9.33 difference flip-flopped when I ran the commands again. Binding both processes to the same core is definitely a performance hit though. There might be some gain through a shared cache, but it's lost more through lack of parallelism.

I tried 2/4 as not sure how 'real' cores vs 'hyperthread' cores are numbered. These numbers are from a i7-7700k.

How do you know that the dataset fits in L2 ?

Assuming pv uses splice(), there is one only copy in the workload: copy_from_user() from fixed source buffer to some kernel allocated page, then those pages are spliced to /dev/null.

If the pages are not "recycled" (through LRU scheme for allocation), the destination changes every time and the L2 cache is constantly trashed.

I only learned of pv from this article so I can't speak much about its buffering. I would guess that the kernel would try to re-use recent freed pages to minimise cache thrashing. But anyway, on the 'yes' side, the program isn't re-allocating its 8kb buffer after every write(), so there's a lot of data being re-read from the same memory location.

L2 cache and copying "y" bytes have very little to do with this; I suspect if you could produce high-granularity timings it would almost all be in the syscall overhead.

See eg. https://stackoverflow.com/questions/23599074/system-calls-ov... who benchmarked it at ~638ns per "read" call.

(Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)

Yes they do.

A read() syscall takes longer than a getpid() syscall because read() has more work to do, it actually does a data copy of len bytes, which takes some time (and will be faster/slower if data is cache hot)

What we call the "syscall overhead" is what happens before and after the actual data copy, switching between user and kernel mode.

You make that overhead negligible by calling read() with a large size.

(Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)

Snap!

IIRC, ZWS used shared memory to mirror the results of the time() syscall across processes, to save a few nanoseconds on some operating systems :) That was before Linux and other OSs used techniques like the vsyscall/VDSO mentioned in the stackoverflow discussion...

As another point in your favor, one of the commentors reached 123 GB/s modifying both yes and pv.

https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.

Still gets piped and hits same performance.

It's 2x times faster for a quick test here. Only copying on the read side?

Out of interest: Could you please post the code?

https://pastebin.com/jrcJbjU4

I've realised straight tee() is actually wrong - it works fine for piping to something but "zcyes > /dev/null" won't work, it needs vmsplice() like mrb said.

  core2duo e8400, 4.8 kernel, coreutils 8.25
  > yes | dd bs=1M count=10000 of=/dev/null iflag=fullblock
  10485760000 bytes (10 GB, 9.8 GiB) copied, 4.1901 s, 2.5 GB/s
  > ./zcyes | dd bs=1M count=10000 of=/dev/null iflag=fullblock
  10485760000 bytes (10 GB, 9.8 GiB) copied, 1.8542 s, 5.7 GB/s

Actually yes should use vmsplice().

And pv on the other side of the pipe should use splice().

Now that would be a complete zero-copy I/O path, purely limited by the CPU, not by memory bandwidth. It would benchmark at hundreds of GB/s :)

> And pv on the other side of the pipe should use splice().

Good catch, I just noticed that it does.

Someone posted an implementation which uses vmsplice on the reddit thread:

https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

His code can further be optimized by having only 1 large element in the iovec (vs. many 2-byte "y\n" elements).

from what i can see it's already in a large iovec buffer

Correct, I misread the code. However bumping the pipe size buffer bumps the speed from 28 to 74 GB/s on my Skylake CPU (i5-6500).

Edit: I optimized further and now get 123 GB/s See https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).

On the other hand the current code is perfectly portable, tee(2) is a linux syscall.

FreeBSD's yes has just been updated because of this.

https://github.com/freebsd/freebsd/commit/1d61762ca37c20ab6f...

It's about twice as fast as GNU yes now on my FreeBSD system here.

Looks like that may drop some data if you get a short write, possible when writing to pipes etc.

Update: They fixed that issue with this follow up https://github.com/freebsd/freebsd/commit/2592fbb8

Let's not forget the most crossplatformest, purest `yes` of them all: https://www.npmjs.com/package/yes

    # /usr/local/bin/yes | pv > /dev/null
    11.5MiB 0:00:09 [1.02MiB/s] [                             <=>]
    
    # /usr/bin/yes | pv > /dev/null
    1.07GiB 0:00:09 [ 142MiB/s] [                             <=>]

JavaScript wins again!!

It's come to my attention that lower numbers are not better here. I have filed a bug, we'll get to the bottom of this shortly. I want to apologize to all our users, this issue does not reflect the values and principals we at Pure JavaScript `yes` hold dear https://github.com/Sequoia/yes/issues/3

I was not going to post this because hacker news has this ethic (?) of down voting anything that seen as not positive. Perhaps we should have discussion about that, I'm not sure that's a good thing but I'm not in charge here.

The top comment is:

"It's a shame they didn't finish their kernel, but at least they got yes working at 10GiB/s."

which as an OS guy, someone who has been working on Unix for 30+ years, as a guy who was friends with one the QNX kernel guys (they had perhaps the only widely used microkernel that actually delivered), that's hugely amusing and spot on. The GNU guys never really stepped up to being kernel people. Bitch at me all you want, they didn't get there. It's a funny comment, especially coming from reddit.

> hacker news has this ethic (?) of down voting anything that seen as not positive

We must not be reading the same Hacker News...

Anyway, the comment you're quoting is just a shallow jab that belittles the GNU developers' work without contributing anything new or meaningful. It's telling that you had to spend two paragraphs to justify cross-posting it here.

You say that and then immediately become an example of what hes talking about. This "shallow jab that contributes nothing new or meaningful" is, in some circles, known as a "joke." I'm continually frustrated by people who think that misinterpreting comments as harmful is a useful activity.

Not really -- I didn't downvote that comment. And I still dispute the notion that negativity is rare or always shunned on this board: to the contrary, it's so commonplace that an actual rule [0] had to be added to try to sway things in the other direction.

Jokes have their place, but bringing up the failure of Hurd in every GNU-related post is banal. And saying they "never really stepped up" to your level as a mighty kernel developer, as if the people who brought us glibc and coreutils lack an understanding of OS internals, just seems rude and curiously out of touch.

[0] https://news.ycombinator.com/item?id=9317916

So negativity here is not rare, when people don't like something they are fast to jump on it.

What I was trying to get at is this, if you care about your upvotes, hacker news promotes a sort of hive mind. Which is somewhat like "say only nice things unless you are clearly swatting down something that is obviously wrong".

Which is mostly fine, fantastic in fact. I'm fine with it, hacker news is really pleasant because of the (more or less ) quality of the posts and especially the quality of the comments. I'd much rather have it be this way than a free for all, those go bad pretty fast.

So I'm for the hive mind, I was just pointing out that you can't make jokes and be upvoted. The joke wasn't banal at all IMO. GNU has done a lot of good, I've been there since the beginning and paid attention along the way. They have also been pretty self serving with their choices, every project has to sign away their rights and then GNU takes full credit for the project even though they had nothing to do with it other than it being GNU troff for example. Given their tendency to take credit for stuff that they didn't do, and their claim that they can do an OS but clearly can't, that joke is funny as heck. If you don't get that, sorry, you haven't been paying attention.

() The quality of the posts in the area of programming, especially systems programming, is spotty. Some stuff is great, stuff I didn't know (there is a lot of that here and I'm very grateful for it, it's why I stick around), some stuff is meh, and then there is stuff like "wow, look at $OBVIOUS, isn't that cool?" that gets upvoted. That last one I just don't get, but whatever, the good stuff is good. The signal/noise ratio here is better than any other programmer oriented site I've found.

Years ago I read a similar experiment about max. CPU data flow. Guy was testing how much data can his CPU pass in a second. He was writing it in C, using some Linux optimization, optimizing code for CPU caches, using some magical C vectors that are optimized for such purpose. He got some help from someone working at Google. I tried to find that post but never succeeded. Does anyone here know it?

`yes` (with the backticks) is my favorite "bring the system to its knees right now" shell command.

Does not do that on modern Linux or especially -ck patch.

Good to know. I think the last time I tried it was on a rhel5 or rhel6 variant.

However, as avip found out below, it does still render OS X useless within less than a minute (at least on my 2015 MBP).

What does `yes` try to do?

I'm guessing it tries to read the output and return it. Since yes doesn't actually terminate it's going to generate a massive temporary variable to store it's continuous output.

This will attempt to open a child shell process with whatever output of "yes" is, interpreted as shell script.

But before that, parent shell has to buffer until EOF. With "yes" output being unbound, this means unbound buffering / memory growth. That is, until OOM killer take notice and shut it down.

Whole thing will probably take few seconds to a minute (depending on how much free RAM there is vs. how fast it is), peg a single CPU core in the process, and will recover cleanly (except for shell in question being terminated).

...unless system in question has single CPU and large+slow swap. Then yes, bring-it-to-its-knees.

It may also push everything in memory to swap in the process, what is the real speed killer.

Make sure to have a sane memory limit set in PAM and switch vm.swappiness to lower value than 60.

Single CPU is fine nowadays. It will be slowish if you ran it with no nice or priority.

RAM can be tweaked by preventing Linux heuristic overcommit via sysctl vm.overcommit_memory=2. (0 is only recommended if you do not run broken applications. It so happens that many JS and Java VMs are broken on memory pressure.)

Well thanks for killing my mac. People are trying to work here you know.

You took a command from a comment which said they use it to "bring their system to its knees", ran it, and then complained?

I thought that's what internet is for.

How? Only eats 2GiB and crashes bash with

    bash: xrealloc: cannot allocate 18446744071562067968 bytes

And the question is, do we need yes to be so optimized?

Not complaining, I like this kind of analysis

But it seems you won't be limited, in a shell script, by the speed you can push y's

That kind of question doesn't make sense for open source code.

Somebody wanted to optimize 'yes', so they did. There doesn't need to be a good reason, just like there doesn't need to be a good reason for a person to read a certain book or watch a certain movie, other than they want to do it.

The question was "do we need it? (for a specific use case)" rather than "should we do it?".

And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed

> And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed

That's not how open source works, though. There's not a group of people obligated to work on the GNU utilities. There's not a central project manager ordering people around telling them what changes to make.

Declaring that 'yes' is generally fast enough already doesn't imply that it was fast enough for the person who spent their time optimizing it. Somebody needed (or just wanted) 'yes' to be really fast, so they did the work and submitted the changes back.

> There's not a central project manager ordering people around telling them what changes to make.

No, but there is review and approval of patches (also the most frequent contributors know where the project is going and there is bug tracking)

If one just makes it faster without major benefits that patch is likely to be rejected

> If one just makes it faster without major benefits that patch is likely to be rejected

IF the project were overwhelmed with incoming patches, then I would agree that there's probably higher priority changes than optimizing 'yes'.

But I'm almost certain the GNU project is not being overwhelmed with changes to the core utilities. Everything else being equal (code quality, readability, test coverage, etc.) there's really no reason to reject a patch that demonstrably improves the project, even in some silly way like making 'yes' really fast.

It makes sense for any kind of code.

This is software engineering. Thinking about the tradeoffs and whether they are appropriate or warranted is bread and butter. It very much makes sense to ask whether all of this is merely optimizing for a benchmark at the expense of other factors that might turn out to be more important, such as long term maintainability, portability, behaviour in some common cases which are significantly unlike the benchmark, and so forth.

Several people have made some of these points here and in other discussions; and not only do they make sense, they are an important part of the discipline.

Is the optimized implementation documented well enough that other people coming to it anew can understand it in another 10 years' time? How many of the magic constants have to be tweaked as the operating system evolves? Is it clear to maintenance programmers how they need to be tweaked? Does optimizing for the benchmark skew the implementation too far away from one common case where one is only wanting a few "y"s (because, say, the program being told "y" only asks 17 questions) resulting in every invocation of the yes program rapidly generating KiBs or even MiBs of output, that lives in pipe buffers in kernel space unused and only to be thrown away? Does it make more sense to put buffer size optimizations in the C library where all can benefit? What is the benefit of optimizing for Linux-only system calls on GNU Hurd? Will we optimize yes for GNU Hurd, too, in the same codebase? How much conditional compilation will we end up with? If the C library is improved to do better in the future, is it a problem that the optimized for benchmark program no longer automatically gains from it? How much of a problem is it that a GNU program is now not portable to systems other than Linux, and is locked into one operating system?

And what about other benchmarks? Many have noted that this benchmark pretty much relies on the fact that the output of the yes program is simply thrown away, and no real work is being done by anything else on the system. What about a benchmark where it is? What about a benchmark where it is important that yes issues a system call per line, yielding the CPU to other processes so that they can process that line before yes bothers to print the next? What about a benchmark that measures kernel memory usage and treats lower as better? What about benchmarks that measure how small the yes program is; not just in terms of code but in terms of memory, I/O, and CPU usages; because the memory, I/O, and CPU should actually be given to more important programs than yes on the system that are actually the system's main function? What about low impact?

Yes, it's fun to focus narrowly and optimize yes so that it generates as much output as it can for one specific toy usage. But in engineering one has to ask and think about so much more.

This can be extended to any input given to the program, since yes is defined to take an argument from the command line to print out instead of "y". See https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

A detail you seem to be missing is that you are not limited to a shell script. The shell sets up the pipeline, but the members of the pipeline can be written in arbitrary languages and just have each stdout linked to the next processes stdin.

As a result you can process very large volumes of data and consume (not waste, consume) significant system resources to perform your processing.

I would just pre-allocate a static array of "y\n" of size BUFSIZ, write it out in a loop, and call it for the day, skipping the whole malloc and filling loop business.

Make the static array BUFSIZ * 1024 to trim the syscalls by a factor of 1000.

Real yes accepts a string to print (so you can have it spit out a full "yes" or "Y" rather than a hardcoded "y").

Have a few pre-written arrays for the common cases: "y", "Y", "n", "N", etc. Those are the fast cases (or the benchmark optimized cases, like, what Volkswagen did). Have another pre-allocated static array to fill in with other input.

> (or the benchmark optimized cases, like, what Volkswagen did)

This is amazing. Maybe we can turn this into a verb? "I Volkswagened the common cases with precalced buffers".

That's how you get sued. Especially because now Ford and GM are also in court, for having done the same as VW.

In the same vein as Adobe sues me for photoshopping a picture or Alphabet sues me for googling my exgirlfriends' names?

No, but you don’t wanna get sued for libel and slander. If VW can realistically show that it is industry-standard behaviour (which is pretty obvious), then they might even have a chance to win.

Either way, it’d be expensive for you.

I'll call you when they sue.

It's been said, "Truth is the best defense for libel". Reasonably held opinion would then be the second.

Realistically, not mentioning anyone or anything is the only complete defense. The truth still comes close.

I do not think Volkswagen will try to sue you for adding a new phrase to urban dictionary ;-)

Let me check that I have correctly understood the scenario you're anticipating.

1. Someone uses "volkswagen" as a verb meaning "cheat in benchmarks".

2. Volkswagen takes them to court, on the basis that it is slanderous or libellous to associate Volkswagen with cheating in benchmarks.

3. Counsel for the defence reminds the court that Volkswagen were in the news for a protracted period for a benchmark-rigging that saw them hit with a multi-billion-dollar fine, slashed a third off their stock price, saw their CEO resign, led to there being a "Volkswagen emissions scandal" page on Wikipedia, etc., etc., etc.

4. [THIS IS THE BIT I'D LIKE YOU TO FILL IN FOR ME.]

5. Volkswagen wins the case.

How exactly does step 4 go?

By volkswagen arguing that this is industry standard behaviour, and the entire media campaign is already libel and slander.

And they’d have a pretty good case with that, considering the original report all this media campaign was based on called out 6 carmakers, but the media (and the poster) only call out Volkswagen.

"To volkswagen" as in "optimize for a benchmark"? I'm so going to use this as soon as possible.

Incidentally, that word works even better in German, where every verb must end in "-en".

I am pretty sure software cheating on benchmarks precedes VW. Samsung a few years ago? Probably not the first either.

I think video card manufacturers were first to cheat benchmarks

'Never trust a benchmark you didn't Volkswagen yourself'

That's exactly what the OP did.

No, OP mallocs an 8k buffer then fills it as part of main.

See the "fifth iteration" done in assembly.

Why is it so slow(compared to the post) in the macbook air. Native yes runs at 26 MiB/s, and GNU yes at 620 MiB/s.

The CPU and RAM are slower and there is much more kernel overhead. Not to mention pipe buffer size is small on OS X.

Come on people, I ran GNU yes just there, it isn't as slow as OP says but it is a quarter of expected performance.

It isn't! yes runs at 7.2 GiB/s on my macbook air. Though I have Linux installed on it instead of Mac OS :)

Are you trolling? The commenter obviously meant the macOS version of "yes".

The VM subsystem is much slower. Also possibly a not-up-to-date GNU grep, on my (ancient) MBP I get 26MB "native", 780MB GNU and 2.8GB for TFA's C program (2.9 if I replace the malloc by a stack-allocated array, weirdly enough)

Probably alignment issues: stuff on the stack might be aligned by default, in contrast to malloc'd memory.

With that malloc overhead, I expect GNU yes to be slower when only a few bytes are read from it.

So, what's the distribution of #bytes read for runs of 'yes'? If we know that, is GNU 'yes' really faster than the simpler BSD versions?

Also, assuming this exercise still is somewhat worhtwhile, could startup time be decreased by creating a static buffer with a few thousand copies of "y\n"? What effect does that have on the size of the binary? I suspect it wouldn't get up much given that you can lose dynamic linking information (that may mean having to make a direct syscall, too).

Unless you are statically linking, one malloc doesn't significantly affect your startup time. When only a few y's are read, time is going to be dominated by ld.so, by a large margin.

> I expect GNU yes to be slower when only a few bytes are read from it

Wouldn't that be entirely negligible compared to starting the program in the first place?

Measurements are really noisy, but I seem to get significantly better numbers than that when I use fsplice() on a pre-generated few pages of file data instead.

Yes, splice can bypass the pipe buffer in some cases.

I thought this was a fascinating read but it left a serious question lingering in my mind, which is a little out-of-scope for the article, but I hope someone here can address.

Why did the GNU developers go to such lengths to optimize the yes program? It's a tiny, simple shell utility that is mostly used for allowing developers to lazily "y" there way through confirm prompts thrown out by other shell scripts.

is this a case of optimization "horniness" (for lack of a better word) taken to its most absurd extreme, or is there some use case where making the yes program very fast is actually important?

The stated use case for the perf improvement was "yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient."

Source: https://github.com/coreutils/coreutils/commit/35217221c211f3...

I've personally used it for generating repeating text and filling disks in system testing, so I appreciate it being faster at those tasks. I also sometimes use it as a signal generator for a hacky load generator, like so:

  yes | xargs -L 1 -P NUM_PROCESSES -I {} curl SOME_TARGET_URL > /dev/null

This doesn't benefit from being faster per se, but I appreciate it using less CPU since I want to give curl as much system resources as possible.

But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?

I would write() the buffer each time it gets enlarged, in order to improve startup speed.

Also: The reddit program has a bug if the size of the buffer is not a multiple of the input text size.

And it's increasing the buffer by incrementing one at a time, instead of copying the buffer to itself, reducing the number of loops needed (at cost of slightly more complicated math).

If your only overhead is filling an 8K buffer, I don't think your user is going to care. Taking one microsecond instead of one nanosecond doesn't matter all that much when you're going to lose way more than that in pipes, the kernel, the program you're piping it to, etc.

But what's the use case for a large volume of continuous output? It feels like we're optimizing for the wrong use case

Maybe filling a disk or flooding a network connection as in

    yes | ssh server "cat > /dev/null"

But yes take arguments so there might be more use cases:

    $ man yes
    NAME
           yes - output a string repeatedly until killed
    
    SYNOPSIS
           yes [STRING]...
           yes OPTION
    
    DESCRIPTION
           Repeatedly output a line with all specified STRING(s), or 'y'.

>But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?

If "only few yes's are needed" then the slowdown to produce them will be inconsequential, whether they still fill a buffer in this case or not.

> But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?

One could put a write in the memcpy loop, so that first it writes one copy of the string, then two, then 4, 8 (etc.), meaning time to first byte is short, but it quickly gets up to the eventual asymptotic speed with a full buffer.

( https://www.reddit.com/r/rust/comments/4wde08/optimising_yes... )

You could also have a static buffer full of yesses, it would increase the binary size by 8k, but GNU yes is already 40k on my system so...

That is mentioned elsewhere in this very thread, where it is also pointed out that that isn't a general solution (`yes` can print more than just y).

> The reddit program has a bug if the size of the buffer is not a multiple of the input text size.

The author appears to be aware of that:

> Even with the function overheads and additional bounds checks of GNU's yes

GNU yes is fast because it is coded with the assumption that it's not answering any real question, such as "can I combine this free code with a proprietary program?" or "Would you accept the following monstrous patch to GNU Coreutils /bin/true without a copyright assignment?"

I think we can do better.

How about a /proc/bin/yes for this? Like most /proc files, it would appear to be empty. Executing it would involve a fs/binfmt_proc.c file in the kernel source, which would be a handler for this sort of executable. That would get the job done entirely in the kernel.

man, I just spent like 8 minutes today writing a python script to use up all the disk space on some servers (part of ops readiness testing) when I could have just used this trick.

`yes` will help me on the "see what happens when something uses all the CPU and memory" test case. Thanks Reddit/HN!

dd can also be useful for that kind of thing since you can use a source like /dev/urandom to generate random bytes if you're trying to avoid compression and the adjustable block sizes can be optimized for the underlying storage system.

I did actually use dd to begin with but it didn't work as I wanted it too right away (can't remember why, probably something SUSE related) and by then I'd used up 6 mins of the 15 mins I gave myself for the job.

I also had to do the same test on windows, so python won

Did you notice PHP outperforms any other scripting languages? Some report that it event beats the GNU yes implementation.

After reading here so many unfair critics and pedantic dislike over PHP[1][2][3][4][5][6], I just want to say: STFU.

[1] https://news.ycombinator.com/item?id=12706136

[2] https://news.ycombinator.com/item?id=3825227

[3] https://news.ycombinator.com/item?id=3824881

[4] https://news.ycombinator.com/item?id=1823022

[5] https://news.ycombinator.com/item?id=1819517

[6] https://news.ycombinator.com/item?id=1819413

... Just to name a few.

It's almost funny that you try to refute, or rather, dismiss, criticism of PHP with "but it's fast", when none of what you cite even mentions that. I just want to say: STFU.

I don't dismiss criticism. I dismiss unfair critics. What I cite is full of that. And, btw, [6] implies PHP is slow, read on:

[quote]

It's hard to find a worse example than php, because when many "bad" languages fail in one or the other category, php seems to fail in most categories.

* Java has a really repetitive syntax, but it's libraries might make up for that.

* C++ with all it's features has bazillion ways to shoot yourself, but it's arguably fast and the syntax is bearable. ...

[/quote]

It doesn't in fact imply that PHP is slow. What it says about PHP (besides "php seems to fail in most categories" which doesn't imply any single specific failure because most != all) is this: "But php has both an ugly syntax, horrendous stdlib, and fame of security issues." You will notice that none of those specific complaints is about speed.

>>>Did you notice PHP outperforms any other scripting languages?

I am not sure what was the last time I had to choose a language by a single dimension. If we are talking about performance, scripting languages rarely shine. On the top of that, performance critical things in scripting languages are often implemented in a compiled language (like Pandas for example).

>> If we are talking about performance, scripting languages rarely shine

Except that, in the yes example, PHP shines by its performance.

Of course, you don't choose a language based on that on a single dimension, but speed is an important factor otherwise benchmarks would not exist.

while(1){write();} is about as far away as it gets from actual application code. This topic may be fun, but remains an utterly pointless micro-benchmark.

I was refering to http://benchmarksgame.alioth.debian.org/

PHP here is just calling c functions. The actual non-benchmark php people pay for is completely orthagonal to these results.

None of those comments are complaining about PHP being slow at repeatedly writing a string whose length is a multiple of the page size to standard output. That is a pretty niche case, and hardly refutes even the complaint that PHP is slow, much less all the complaints about PHP that aren't related to its speed.

FWIW, when I run the tests there's no substantial difference between Perl, LUA, or PHP.

I run the command `yes | pv > /dev/null` on my MacBook Pro, it's only 37m/s, is this normal? I am not familiar with the command.

From the OP:

>OS X just uses an old NetBSD version similar to OpenBSD's >NetBSD's is 139MiB/s, FreeBSD, OpenBSD, DragonFlyBSD have very similar code as NetBSD and are probably identical,

It really depends on how fast your memory and processor are. Everything there was benchmarked with an i7-4790 with DDR3-1600, and very very little running in the background.

Two orders of magnitude difference can't easily be blamed on CPU, in this case it depends on how optimized yes is. I have some older version of coreutils on one machine here and performance is similarly abysmal.

I am getting 26 MiB/s using native yes, and 620 MiB/s using GNU yes in macbook air.

Well now I know what `yes` does (And pv)

Why is he using backticks to quote "yes" in the title?

It is common in markdown formatting to indicate a command line with backquotes. It tells the markdown compiler to apply special css and possibly syntax highlighting. So, a lot of people who write about shell commands and setup tutorials, or use markdown on something like github will automatically identify it as a command.

clearly, we just need /dev/yes

Sounds like a reasonable feature request for systemd.

yes | write <USERNAME> "Don't you hate dialup connections?"

I think you mean yes "Don't you hate dialup connections?" | write username

The proprietary Oracle Solaris 11.2 yes really slowed down when they added DRM and Verified Boot support...

https://m.xkcd.com/619/

tl;dr someone who doesn't understand how i/o works gets a small insight into how memory and a cpu work and decides "Buffering is the secret" and "You can't out-optimize your hardware"

Can we have a new flag for posts by people who don't know what they're doing so I can skip them? I am serious.

Reply on news.ycombinator.com