A Little Story About the `yes` Unix Command

“GO.COM contained no program bytes at all – it was entirely empty. However, because GO.COM was empty, but still a valid program file as far as CP/M was concerned (it had a directory entry and file-name ending with .com), the CP/M loader, the part of the OS whose job it is to pull programs off disk and slap them into the TPA, would still load it!

So, how does this help? Well, using the scenario above:

• the user exited WordStar • the user ran DIR (or whatever else they needed) and at some future point would be ready to re-run Wordstar • the user now ‘loaded’ and ran GO.COM • the loader would load zero bytes of the GO.COM program off disk into the TPA – starting at address 0100h – and then jump to 0100h – to run the program it just loaded [GO.COM]! • result – it simply re-ran whatever was in the TPA when the user last exited to DOS – instantly [WordStar in this example]!

So, GO.COM, which consisted of zero bytes of code – and sold for £5 a copy is, I figure, the most profitable program ever written (as any other program will return mathematically fewer £s per byte than GO.COM did)!”

1. http://web.archive.org/web/20160304014157/http://peetm.com/b...

There needs to be a certain level of creative work to qualify for copyright, and an empty program that does nothing is rather unlikely to qualify.

The definite case on that subject must be John Cage's "4:33", which is 4 minutes and 33 seconds of silence. Its copyright has been upheld in court. Yes, the estate of John Cage sued somebody who "quoted" to much of his work of silence.

That sounds somewhat silly, yet it's well-reasoned: the infringing artist actually acknowledged Cage in the CD inset. And 4'33 isn't really only silence. Audio recordings feature the pianist sitting down, open and closing the piano cover, and the audience making various noises. It's somewhat "out there" obviously, but just the fact that it's rather well-known kind of shows that it did what it was supposed to do.

I love this.

The output of yes(1) is normally consumed by another program, not redirected to a file, so this is what the examples emulate.

  #ifndef _GNU_SOURCE
  #define _GNU_SOURCE 1
  #endif
  
  #include <errno.h>    /* ERANGE errno */
  #include <inttypes.h> /* strtoumax(3) */
  #include <limits.h>   /* LONG_MAX */
  #include <stdint.h>   /* uintmax_t */
  #include <stdio.h>    /* fflush(3) fprintf(3) fread(3) fwrite(3) tmpfile(3) */
  #include <stdlib.h>   /* EXIT_FAILURE */
  #include <string.h>   /* strlen(3) */
  
  #include <err.h>      /* err(3) errx(3) */
  #include <fcntl.h>    /* loff_t splice(2) */
  #include <sys/time.h> /* struct timeval gettimeofday(2) */
  #include <unistd.h>   /* _SC_PAGESIZE pread(2) sysconf(3) write(2) */
  
  #ifndef HAVE_SPLICE
  #define HAVE_SPLICE __linux__
  #endif
  
  #define MIN(a, b) (((a) < (b))? (a) : (b))
  #define MAX(a, b) (((a) > (b))? (a) : (b))
  
  #ifndef howmany
  #define howmany(x, y)  (((x) + ((y) - 1)) / (y))
  #endif
  
  #define UMAX_PREC (sizeof (uintmax_t) * CHAR_BIT) /* NB: assumes no padding */
  #define UMAX_HALF ((UMAX_PREC + 1) / 2)
  #define UMAX_LO(n) ((n) & ((UINTMAX_C(1) << UMAX_HALF) - 1))
  #define UMAX_HI(n) ((n) >> UMAX_HALF)
  
  static inline _Bool
  add_overflow(uintmax_t *r, const uintmax_t a, const uintmax_t b)
  {
          if (~a < b)
                  return 1;
          *r = a + b;
          return 0;
  }
  
  /*
   * Implement multiplication using a polynomial with four multiplications and
   * three additions, except we can optimize out some operations.
   */
  static inline _Bool
  mul_overflow(uintmax_t *_r, const uintmax_t _a, const uintmax_t _b)
  {
    uintmax_t a[2] = { UMAX_LO(_a), UMAX_HI(_a) };
    uintmax_t b[2] = { UMAX_LO(_b), UMAX_HI(_b) };
    uintmax_t r[2];
  
    /* if both are non-0, we'd always overflow */
    if (a[1] && b[1])
      return 1;
  
    /* either a[1] or b[1] must be 0 here, so no intermediate overflow */
    r[1] = (a[1] * b[0]) + (a[0] * b[1]);
  
    /* if the result has MSW bits set, we'd overflow */
    if (UMAX_HI(r[1]))
      return 1;
  
    r[0] = a[0] * b[0];
  
    return add_overflow(_r, r[0], r[1] << UMAX_HALF);
  }
  
  #if 0
  static uintmax_t
  add(uintmax_t a, uintmax_t b)
  {
    uintmax_t r;
    if (add_overflow(&r, a, b))
      errx(1, "arithmetic overflow (%ju * %ju)", a, b);
    return r;
  }
  #endif
  
  static uintmax_t
  mul(uintmax_t a, uintmax_t b)
  {
    uintmax_t r;
    if (mul_overflow(&r, a, b))
      errx(1, "arithmetic overflow (%ju * %ju)", a, b);
    return r;
  }
  
  static uintmax_t
  toumax(const char *p)
  {
    char *pe;
    uintmax_t n;
  
    errno = 0;
    if (UINTMAX_MAX == (n = strtoumax(p, &pe, 10)) && errno != 0)
      err(1, "%s", p);
    if (*pe != '\0' || p == pe)
      errx(1, "%s: invalid number", p);
  
    return n;
  }
  
  static uintmax_t
  gcd(uintmax_t a, uintmax_t b)
  {
    uintmax_t c;
  
    while (a != 0) {
      c = a;
      a = b % a;
      b = c;
    }
  
    return b;
  }
  
  static uintmax_t
  lcm(uintmax_t a, uintmax_t b)
  {
    return mul(a, b) / gcd(a, b);
  }
  
  static size_t
  xgetpagesize(void)
  {
    long n = sysconf(_SC_PAGESIZE);
    if (n <= 0)
      err(1, "sysconf");
    return n;
  }
  
  static double
  gtod(void)
  {
    struct timeval now;
    gettimeofday(&now, NULL);
    return now.tv_sec + (now.tv_usec / 1000000.0);
  }
  
  int
  main(int argc, char **argv)
  {
    _Bool cflag = 0, nflag = 0;
    uintmax_t count = UINTMAX_MAX, lines = 0, total = 0;
    int optc;
  
    while (-1 != (optc = getopt(argc, argv, "c:n:"))) {
      switch (optc) {
      case 'c':
        if (nflag)
          errx(1, "-c and -n mutually exclusive");
        count = toumax(optarg);
        cflag = 1;
        break;
                  case 'n':
                    if (cflag)
          errx(1, "-c and -n mutually exclusive");
        lines = toumax(optarg);
        nflag = 1;
        break;
      default:
        return EXIT_FAILURE;
      }
    }
    argc -= optind;
    argv += optind;
  
    const char *const line = (argc > 0)? argv[0] : "y";
    const size_t linesize = strlen(line) + 1;
    const size_t pagesize = xgetpagesize();
    const size_t blocksize = MAX(mul(pagesize, 16), lcm(linesize, pagesize));
    const uintmax_t filesize = mul(blocksize, howmany((1UL << 20), blocksize));
  
    if (nflag)
      count = mul(linesize, lines);
  
  //  fprintf(stderr, "linesize:  %zu\n", linesize);
  //  fprintf(stderr, "pagesize:  %zu\n", xgetpagesize());
  //  fprintf(stderr, "blocksize: %ju\n", blocksize);
  //  fprintf(stderr, "filesize:  %ju\n", filesize);
  
    FILE *fh;
    if (!(fh = tmpfile()))
      err(1, "tmpfile");
    for (size_t i = 0; i < filesize; i += linesize) {
      int n = fprintf(fh, "%s\n", line);
      if (n == -1)
        err(1, "fprintf");
      if ((size_t)n != linesize)
        errx(1, "wrote %d bytes, expected %zu", n, linesize);
    }
    if (0 != fflush(fh))
      err(1, "fflush");
  
    double begin = gtod();
  #if HAVE_SPLICE
    int fd = fileno(fh);
    _Bool eof = 0;
    do {
      loff_t p = 0;
      size_t r;
      ssize_t n;
  
      /* NB: no LOFF_MAX available */
      _Static_assert(sizeof p <= sizeof (long), "unexpected type for loff_t");
      if (filesize > LONG_MAX)
        errx(1, "filesize too large (%ju > %jd)", filesize, (intmax_t)LONG_MAX);
  
      while ((size_t)p < filesize && count >= blocksize && !eof) {
        r = MIN(blocksize, count);
        r -= r % blocksize;
        n = splice(fd, &p, STDOUT_FILENO, NULL, r, SPLICE_F_MOVE);
        if (n == -1)
          err(1, "splice");
        p += n;
        count -= n;
        total += n;
        eof = n == 0;
      }
    } while (!eof && (count || (!cflag && !nflag)));
  #endif
    {
      char buf[BUFSIZ], *p, *pe;
      size_t n;
  
      while (count) {
        rewind(fh);
        if (!(n = fread(buf, 1, MIN(sizeof buf, count), fh))) {
          if (ferror(fh))
            err(1, "fread");
          break;
        }
  
        p = buf;
        pe = &buf[n];
        while (p < pe) {
          if (!(n = fwrite(p, 1, (size_t)(pe - p), stdout)))
            err(1, "fwrite");
          p += n;
          count -= n;
          total += n;
        }
      }
    }
    if (0 != fflush(stdout))
      err(1, "fflush");
  
    double elapsed = gtod() - begin;
    fprintf(stderr, "%.3f GB/s (%zu bytes in %.3f seconds)\n", ((double)total / elapsed) / (1UL<<30), total, elapsed);
  
    return 0;
  }

Regarding portability, GNU tail uses the Linux-specific inotify(2) to respond faster to writes. Like alot of OSS, coreutils uses the BSD .d_type member extension[1] of struct dirent to avoid unnecessary stat() calls. There are many other more intrusive OS-specific details baked into coreutils, but often it's the nature of the problem--in many situations you're dependent on platform-specific details or features. For the most part, these nitty-gritty platform-specific details are far more intrusive in terms of code complexity than the performance optimizations.

[1] Missing from Solaris, and probably most other SysV derivatives.

  #include <sys/uio.h>
  
  #define str(x) #x
  #define expand(x) str(x)
  #define y1 y\n
  #define y2 expand(y1) expand(y1)
  #define y4 y2 y2
  #define y8 y4 y4
  #define y16 y8 y8
  #define y32 y16 y16
  #define y64 y32 y32
  #define y128 y64 y64
  #define y256 y128 y128
  #define y512 y256 y256
  #define y1024 y512 y512
  #define y2048 y1024 y1024

  //2048 "y\n" s
  static char __attribute__((aligned(4096))) str[] = y2048;
  static struct iovec v[] = {
  	{str, 4096},			//adjust number of these to taste
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  };
  
  
  int main(int argc, char** argv) {
  
  	while(1) {
  		//scatter-gather write (on linux there is basically no way for write to write an odd number of characters, except EOF, so we need not check for partial writes
  		(void)writev(1, v, sizeof(v) / sizeof(*v));
  	}
  	return 0;
  }

Does it have nothing to do with the fact that string-of-bytes is the default in Python 2, whereas string-of-characters is the default in Python 3? Or is that perhaps related to the explanation you gave? Forcing the byte interpretation, Python 3 is slightly faster than Python 2 for me. Forcing the character interpretation, Python 2 wins, but not by as much as before.

Bytes:

  while True:
      print(b'y')

  while True:
      print(u'y')

    import os
    ys = 'y\n' * 2048
    while True:
        os.write(1, ys)

(had to separate the asters to stop HN swallowing them)

Plain old `yes` gets a measly 33Mb/s!!

  perl -e 'print "y\n" while 1' | pv -r > /dev/null
  [3.37MB/s]
  perl -e 'print "y\n" x 2048 while 1' | pv -r > /dev/null
  [ 425MB/s]
  yes | pv -r > /dev/null
  [11.1MB/s]

Why the hell is it line-buffered when writing to a pipe? Yet another common sense “enhancement”?

If it's I/O bound why are they changing at all?

    #!/usr/bin/python
    import sys

    yes = "yes\n"
    print(sys.argv)
    if len(sys.argv) == 2:
        yes = sys.argv[1] + "\n"

    size = 4096
    buf = bytes(yes * size, 'utf-8')

    while True:
	sys.stdout.buffer.write(buf)

Why not just use /dev/zero or /dev/urandom for generating load?

I think most modern compilers will warn on encountering such short hand due to the common error of accidentally declaring integers when you meant to assign a value to a variable you forgot to declare.

    for p in `seq 1 $(sysctl -n hw.ncpu)`; do yes > /dev/null & done

  parallel -- yes yes yes yes >/dev/null

Similar looking model from same company: https://selinc.com/products/3355/

On the other side there are proper languages which do provide all safeties and beat rust or C++ in performance, such as pony. A proper type-system does help (compile-time guaranteed), but you could also add such features to the run-time system (e.g. a GC or safe threading with a single-writer system such as parrot). With rust you don't have any of it, you have to manually add locks or mutexes into your threaded code, and try to avoid unsafe.

If you rewrite this to use function calls instead of methods you end up with something like:

    (or (cadr (args)) "y")