Why does an extraneous build step make my Zig app 10x faster?

The latter is faster in actual CPU time, however note that TFA the measurement only starts with the program, it does not start with the start of the pipeline.

Because the compilation time overlaps with the pipes filling up, blocking on the pipe is mostly excluded from the measurement in the former case (by the time the program starts there’s enough data in the pipe that the program can slurp a bunch of it, especially reading it byte by byte), but included in the latter.

It depends on where the timing code is. If the timer starts after all the data has already been loaded, the time recorded will be lower (even if the total time for the whole process is higher).

I said idiomatic because that’s the form they’re going to encounter it in the wild, in library doc, in stack overflow answers.

I do t think rhe sort of low level bit banging you propose is a worthwhile use of a students time, given the vast amount they have to learn that won’t be immediately obsolete.

This is similar to the presentation in Software Tools, IIRC.

I beat benchmark after benchmark in 80s on disk I/O. I was amazed that nobody else figured out what my compiler was doing - using a 16K buffer for the floppy drive rather than 512 bytes.

I do not think the typical language tutorial is well served by trying to teach a condensed version of all of CS.

Thanks for the tip! I don't have experience with strace, and I'm wondering if I'm misunderstanding what you're saying.

I tried running your suggested command, and I got 2,800 lines of output like this:

    execve("/nix/store/vqvj60h076bhqj6977caz0pfxs6543nb-bash-5.2-p15/bin/sh", ["sh", "-c", "echo \"60016000526001601ff3\" | xx"...], 0x7fffffffcf38 /* 106 vars */) = 0
    brk(NULL)                               = 0x4ea000
    arch_prctl(0x3001 /* ARCH_??? */, 0x7fffffffcdd0) = -1 EINVAL (Invalid argument)
    access("/etc/ld-nix.so.preload", R_OK)  = -1 ENOENT (No such file or directory)
    openat(AT_FDCWD, "/nix/store/aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
    read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
    newfstatat(3, "", {st_mode=S_IFREG|0555, st_size=15688, ...}, AT_EMPTY_PATH) = 0
    mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7fc2000
    mmap(NULL, 16400, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ffff7fbd000
    mmap(0x7ffff7fbe000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7ffff7fbe000

Am I doing it wrong? Or is there more training involved before one could usefully integrate this into debugging? Because to me, the output is pretty inscrutable.

My biggest annoyance is when I get some tooling from some other team, and they're like "oh just extend this Python script". It'll operate on local files, using shell commands, in a non-reentrant way, with only customization from commenting out code. Maybe there's some argparse but you end up writing a program using their giant args as primitives.

Guys just write small programs and chain them. The wisdom of the ancients is continuously lost.

"The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”"

http://catb.org/~esr/writings/unix-koans/ten-thousand.html

What programming language do you use where there isn't performant hashing/compression algorithms implemented as libraries?

I think a main reason for this is that you can have a "good enough" working mental model of a process, that holds up to your typical use cases and even moderate scrutiny. It's often only once you run into a case where your mental model fails that you even think to challenge the assumptions it was built on - at least, this has been my experience.

Well, it could be implemented like this, it's just more cumbersome than "create N-1 anonymous pipes, fork N processes, wait for the last process to finish": at the very least you'll need to select() on the last unattached pipe, and when job control comes into the picture, you'd really would like the "setting up the pipeline" and "monitoring the pipeline's execution" parts to be disentangled.

Not even that they might be particularly slow to start in absolute terms, but just that they might be slow relative to how fast the previous stage starts cranking out some input for it.

(Since, as GP said, not an infinite buffer.)

Not exactly. Non-native PowerShell pipelines are executed in a single thread, but the steps are interleaved, not buffered. That is, each object is passed through the whole pipeline before the next object is processed. This is non-ideal for high-performance data processing (e.g. `cat`ing a 10GB file, searching through it and gzipping the output), but for 99% of daily commands, it does not make any difference.

cmd.exe uses standard OS pipes and behaves the same as UNIX shells, same as Powershell invoking native binaries.

> Though now I will break your mind as my mind was broken not a long time ago. Powershell, which is often said to be a better shell, works like that. It doesn't run things in parallel. I think the same is to be said about Windows cmd/batch, but don't cite me on that. That one thing makes Powershell insufficient to ever be a full replacement of a proper shell.

A Pipeline is PowerShell is definitely streaming unless you accidentally forces the output into a list/array at some point, e.g. try this for yourself (somewhere you can interrupt the script obviously as it's going to run forever)

    class InfiniteEnumerator : System.Collections.IEnumerator
    {
        hidden [ulong]$countMod2e64 = 0

        [object] get_Current()
        {
            return $this.countMod2e64
        }
        
        [bool] MoveNext() {
            $this.countMod2e64 += 1
            return $true
        }
        
        Reset() {
            $this.countMod2e64 = 0
        }

    }

    class InfiniteEnumerable : System.Collections.IEnumerable {
        InfiniteEnumerable() {}
        
        [System.Collections.IEnumerator] GetEnumerator() {
            return [InfiniteEnumerator]::new()
        }
    }

    [InfiniteEnumerable]::new() | ForEach-Object { Write-Host "Element number mod 2^64: $_" }

Whether it runs in parallel depends on the implementation of each side. Interpreted powershell code does not run in parallel unless you run it a job, use ForEach-Object -Parallel, or explicitly put it on another thread. But the data is not collected together before being sent from one step from the next.

The character "|" has been introduced in computers in the language NPL at IBM in December 1964 as a notation for bitwise OR, replacing ".OR.", which had been used by IBM in its previous programming language, "FORTRAN IV" (OR was between dots to distinguish it from identifiers, marking it as an operator).

The next year the experimental NPL (New Programming Language) has been rebranded as PL/I and it has become a commercial product of IBM.

Following PL/I, other programming languages have begun to use "&" and "|" for AND and OR, including the B language, the predecessor of C.

The pipe and its notation have been introduced in the Third Edition of UNIX (based on a proposal made by M. D. McIlroy), in 1972, so after the language B had been used for a few years and before the development of C. The oldest documentation about pipes that I have seen is in "UNIX Programmer's Manual Third Edition" from February 1973.

Before NPL, the vertical bar had already been used in the Backus-Naur notation introduced in the report about ALGOL 60 as a separator between alternatives in the description of the grammar of the language, so with a meaning somewhat similar to OR.

I think the math usage was first. i.e. absolute value: |x|