There is general wisdom about bash pipelines here that I think most people will miss simply because of the title. Interesting though, my mental model of bash piping was wrong too.
stage1.exe /in:input.dat /out:stage1.dat
stage2.exe /in:stage1.dat /out:stage2.dat
del stage1.dat
stage3.exe /in:stage2.dat /out:result.dat
del stage2.datThis is one of the reasons why, for all its faults, shell just isn't going anywhere any time soon.
You can also (assuming your language supports it), execute gzip, and assuming your language gives you some writable-handle to the pipe, then write data into it. So, you get the concurrency "for free", but you don't have to go all the way to "do all of it in process".
I've also done the "trick" of executing [bash, -c, <stuff>] in a higher language, too. I'd personally rather see the work better suited for the high language done in the higher language, but if shell is easier, then as such it is.
It's sort of like unsafe blocks: minimize the shell to a reasonable portion, clearly define the inputs/outputs, and make sure you're not vulnerable to shell-isms, as best as you can, at the boundary.
But I still think I see the reverse far more often. Like, `steam` is … all the time, apparently … exec'ing a shell to then exec … xdg-user-dir? (And the error seems to indicate that that's it…) Which seems more like the sort of "you could just exec this yourself?". (But Steam is also mostly a web-app of sorts, so, for all I know there's JS under there, and I think node is one of those "makes exec(2) hard/impossible" langs.)
Did I do that right?
or was it
`tar cvzf t.tar.gz *`
import subprocess
subprocess.run(['tar', 'cvzf', 't.tar.gz', *list_of_files])
import os, subprocess
subprocess.run(['tar', 'cvzf', 't.tar.gz', *(f.path for f in os.scandir('.'))])
Guys just write small programs and chain them. The wisdom of the ancients is continuously lost.
I commonly write little python scripts to filter logs, which I have read from stdin. That means I can filter a log to stdout:
cat logfile.log | python parse_logs.py
tail -f logfile.log | python parse_logs.py
cat logfile.log | python parse_logs.py > filtered.log
tail -f logfile.log | python parse_logs.py | tee filtered.log
But who on Earth has the time for that?
Tee can be useful for that. Maybe pv (pipe viewer) too. I have not tried it yet.
Of course, there many reasons you wouldn’t want this—processes can take time to start up, for example—but it’s not an unreasonable mental model.
(Since, as GP said, not an infinite buffer.)
Though now I will break your mind as my mind was broken not a long time ago. Powershell, which is often said to be a better shell, works like that. It doesn't run things in parallel. I think the same is to be said about Windows cmd/batch, but don't cite me on that. That one thing makes Powershell insufficient to ever be a full replacement of a proper shell.
cmd.exe uses standard OS pipes and behaves the same as UNIX shells, same as Powershell invoking native binaries.
A Pipeline is PowerShell is definitely streaming unless you accidentally forces the output into a list/array at some point, e.g. try this for yourself (somewhere you can interrupt the script obviously as it's going to run forever)
class InfiniteEnumerator : System.Collections.IEnumerator
{
hidden [ulong]$countMod2e64 = 0
[object] get_Current()
{
return $this.countMod2e64
}
[bool] MoveNext() {
$this.countMod2e64 += 1
return $true
}
Reset() {
$this.countMod2e64 = 0
}
}
class InfiniteEnumerable : System.Collections.IEnumerable {
InfiniteEnumerable() {}
[System.Collections.IEnumerator] GetEnumerator() {
return [InfiniteEnumerator]::new()
}
}
[InfiniteEnumerable]::new() | ForEach-Object { Write-Host "Element number mod 2^64: $_" }
0..1000000 | where {$_ % 10 -eq 0} | foreach {"Got Value: $_"} > 0..1000000000 | % { $_ }
# Starts printing out numbers immediately
> 0..1000000000
# Hangs longer than I had patience to wait for
> $x=0..100
> $x.GetType()
# IsPublic IsSerial Name BaseType
# -------- -------- ---- --------
# True True Object[] System.Array
Although 0x7D is overly specific, since if a sibling comment is correct (I have no reason to think otherwise), | for bitwise OR originates in PL/1, where it would have been encoded in EBCDIC, which codes it as 0x4F.
I'm not really disagreeing with you, the |abs| notation is quite a bit older than computers, just musing on what should count as the first use of "|". I'm inclined to say that it should go to the first use of an encoding of "|", not to the similarly-appearing pen and paper notation, and definitely not the first use of ASCII "|" aka 0x7D in a programming language. But I don't think there's a right answer here, it's a matter of taste.
Because one could argue back to the Roman numeral I, if one were determined to do so: when written sans serif, it's just a vertical line, after all. Somehow, abs notation and "first use of an encoded vertical bar" both seem reasonable, while the Roman numeral and specifically-ASCII don't, but I doubt I can unpack that intuition in any detail.
The next year the experimental NPL (New Programming Language) has been rebranded as PL/I and it has become a commercial product of IBM.
Following PL/I, other programming languages have begun to use "&" and "|" for AND and OR, including the B language, the predecessor of C.
The pipe and its notation have been introduced in the Third Edition of UNIX (based on a proposal made by M. D. McIlroy), in 1972, so after the language B had been used for a few years and before the development of C. The oldest documentation about pipes that I have seen is in "UNIX Programmer's Manual Third Edition" from February 1973.
Before NPL, the vertical bar had already been used in the Backus-Naur notation introduced in the report about ALGOL 60 as a separator between alternatives in the description of the grammar of the language, so with a meaning somewhat similar to OR.
Untrue: ".OR." in FORTRAN meant ordinary OR, not bitwise OR. I don't remember ever seeing bitwise OR or AND or XOR in FORTRAN IV.
FORTRAN IV did not have bit strings, it had only Boolean values ("LOGICAL").
Therefore all the logical operators could be applied only to Boolean operands, giving a Boolean result.
The same was true for all earlier high-level programming languages.
The language NPL, renamed PL/I in 1965, has been the first high-level programming language that has introduced bit string values, so the AND, OR and NOT operators could operate on bit strings, not only on single Boolean values.
If PL/I would have remained restricted to the smaller character set accepted by FORTRAN IV in source texts, they would have retained the FORTRAN IV operators ".NOT.", ".AND.", ".OR.", extending their meaning as bit string operators.
However IBM has decided to extend the character set, which has allowed the use of dedicated symbols for the logical operators and also for other operators that previously had to use keywords, like the relational operators, and also for new operators introduced by PL/I, like the concatenation operator.
SPONGE(1) moreutils SPONGE(1)
NAME
sponge - soak up standard input and write to a file
SYNOPSIS
sed '...' file | grep '...' | sponge [-a] file
DESCRIPTION
sponge reads standard input and writes it out to the specified file.
Unlike a shell redirect, sponge soaks up all its input before writing
the output file. This allows constructing pipelines that read from and
write to the same file.