Show HN: Ultimate Plumber – a tool for writing Linux pipes with live preview
github.com
github.com
r
rm
rm -
rm -rf
rm -rf ~
rm -rf ~/
rm -rf ~/tmp
# where did my files go?That said, I'm starting to think — should I include a note about the potential dangers in the readme?... they seemed glaringly obvious to me (at least in case of rm-like commands), but can there be readers/users, to whom they'd not be obvious?...
edit: Ok, I tried to add a warning in the Readme, that's something I can do easily, and if it helps someone, worth it...
Even rm has a -f flag to override some safety measures, though the defaults aren't very safe.
(Though it would complicate installation which is effectively non-existent since there is a single binary; and tightly couple the application with the environment. It's a trade-off anyway.)
(Edit: Somebody mentioned the user "nobody" which seems to be a better alternative.)
In any case I think that the whitelist approach would work better as it would be nice if the tool knew in advance which commands work properly and which ones don't. That way it could inform its users about the allowed & unallowed commands.
Commands with side effects (or "impure" in the FP terminology) don't make much sense anyway to be used within this. The main value is fast iteration, in order to verify the expected output of the pipeline. For me it doesn't make much sense to use it with commands whose main purpose is to modify the filesystem instead of generating / transforming data and writing it to stdout.
So the criteria for the commands that would make it into the whitelist may be "not having side effects", "writing to stdout", and optionally "reading from stdin".
Fastly iterating side effects is unsafe as well, as has already been pointed out.
This can be a fine data analysis tool actually, unfortunately only for command line geeks. The experience is actually not so far from analyzing data with VIM interactively by piping it to an external UNIX command and getting the result dataset back into the editor. VIM hackers will know :)
In a world in which shell commands respected the UNIX philosophy, "find" wouldn't have a silly option like "exec", and other commands wouldn't mix read / write / pure data transform operations in a single command.
But it is what it is. So yeah, protection probably needs to be implemented in the user level, for maximum safety.
Maybe an alternative and/or complementary solution would be to profile each inputted command to detect if they are attempting write operations (maybe with "strace" or something like that), and cancel the evaluation of the command in the next iterations and/or show a warning.
A single-purpose user with its own group would have this problem only to a lesser degree (you'd be able to mess with other users' "up" invocations, but not any system processes).
Do you mean something like this?
:r! lshw | grep network -A2 | grep : | cut -d: -f2- | paste - -
I'm not versed well enough in vim scripting but I suppose there's a way to loop on that on each <CR> or even keypress (like fzf/ctrlp). :r! lshw | grep network -A2 | grep : | cut -d: -f2- | paste - -
Not exactly. More like:- Open vim with the output of `lshw` as content:
lshw | vim -
- Examine the raw data- Send the whole content of the buffer as standard input to the given command and rewrite the buffer with the data read from the standard output of it:
:%! grep network -A2
- Examine the returned dataset. Iterate: :%! grep :
- Examine & iterate: :%! cut -d: -f2-
- Examine & iterate: :%! paste - -
This way, you can examine the output of each step of the pipeline individually, so you can construct your command incrementally. And it is up to you to decide at which point a command will be run instead of it being run automatically following every keypress.Since the dataset returned by the last command will be visible in the current buffer, you will be able to examine & play with it full screen. You will be able to clean or transform some parts manually (this is frequently needed in data science).
You can always return to the previous/next dataset by pressing u/Ctrl+R (for undo and redo), and examine your command history by pressing ":" and then Ctrl+P/Ctrl+N. (Or you can open the command history quickfix menu by pressing "q:" to view/copy the last several commands at once.)
And since you are in a full blown text editor, you can take advantage of other helpful features such as folding, saving the current dataset to a temporary file, etc.
If you are comfortable with more than a few UNIX filters, VIM can be a very convenient and fun tool to play with data.
Could you have this command run as user “nobody” by default, and have a flag to run as a real user? Or as others have mentioned, use capabilities on the spawned processes to prevent anything other than standard input/output?
But if there is a way for you to drop the capability to modify the filesystem before running the first pipeline, you should definitely do it by default (and provide a switch to override it if somebody knows what they are doing.)
It's always been surprising to me that their isn't a built in "undo" to `rm`
e.g. why doesn't `rm` just move files to /tmp?
And do what when tmp gets full and services & applications start crashing?
When someone implements that undo, someone's going to need to write a tool that really does remove files, really. Will the next guy wonder why really-rm doesn't have undo built in?
Same question as parent: "And do what when tmp gets full and services & applications start crashing?"
It's not rocket surgery.
We could all just work with WORM drives instead.
The default rm doesn't do this because shell users are supposed to be able to figure out how not to delete things they want to keep.
Kind of like not pointing a gun at people who you don't want to kill.
The undo for rm is regular backups.
On my system I have snapshots of my home directory created every 15 minutes. Not quite the same thing, but it also helps me when I have incorrectly modified a file.
I don't do this because my idiocy must be punished disproportionately. Also, you end up automatically pressing y enter.
Aliases can be way more dangerous; think about what you're doing. This is what backups are for.
Have a look at either the `trash-cli` package by Andrea Francia.
Meant to type:
rm -rf node_modules && npm install
Question: what's immediately next to the & key on a keyboard?Answer: *
Actually typed
rm -rf node_modules ** npm install
Oops.Theory is: rm is dangerous and should not be used day to day.
Strukt doesn't do 'rm' yet (and nobody has ever requested it), but it does have other operations that do writeback. Once you add one, automatic updates are disabled, and you have to explicitly run the pipeline each time you want it run.
We really need a new cross-platform low-level framework for lego-like operations, like shell commands but updated for the 21st century.
WhatIf instructs the command not to make any changes, but merely to log what it would do. You can "Remove-Item C:\Windows -Force -WhatIf" and it will output "What if: Performing the operation "Remove Directory" on target "C:\windows"" but not delete anything.
(implementing support for "whatif" is optional, so you can't casually use it on any given cmdlet without first checking that cmdlet supports it and makes sensible use of the parameter instead of ignoring it, but the idea is there).
(That's the common Unix shell name for it, though it's not entirely consistent -- which is another problem with using Unix built-ins!)
One thing I've noticed is that it's really frustrating (and not at all helpful) to require perfect validity from a shell-like tool. There's just too many times where you want to edit something into something else, without needing to find a perfect path of valid and meaningful syntax at every intermediate step. So if you have a flag on an operation that doesn't understand that flag, the flag will have a (?) badge to indicate that it won't be understood or used, but the pipeline will still run.
I don't think I'd ever want to use this tool. Every keystroke could be an arbitrary command, and I wonder about the performance of commands with a lot of I/O or processing.
Conceptually it's really neat and I get it, but practically I find things like parameter completion in Fish much more useful.
``Remember, a temp file is just a pipe with an attitude and a strong will to live'' - Randal Schwartz [comp.unix.questions]
I've never seen `paste - -` before, but that is amazing too! I can't count how many times I've wanted something like that. Surely that will help my own shell pipelines in the future.
Also I've never seen `foo |& bar` before. What a neat idea! The surprisingly-valid creativity of it reminds me of Duff's Device. Has anyone else ever found uses for that idiom?
After so many years working in the shell, it's such a rare and delightful treat to learn new tricks. Thank you for sharing this with us!
function cache {
cmd="$*"
name=$(echo "$cmd" | base64)
if [[ -f ./cache/$name.exit ]] && [[ $(cat ./cache/$name.exit) -eq 0 ]]; then
echo "cached"
else
mkdir -p ./cache
eval "$cmd" > ./cache/$name.out
echo $? > ./cache/$name.exit
fi
cat ./cache/$name.out
}
cache ls /It hooks specific functions in Bash and keys off of their arguments.
What I do is typing the command in one acme window with the working directory, and middle mouse swipe it. Acme shows the result of that command in a +Errors window with the same working directory. I change the name of this new popped up window (to something like `+Errors-cmd`, type a command that operates on the file name `/mnt/acme/$winid/body`, and middle mouse swipe it again. Acme shows the result of the new command under `+Errors` again. Rinse and repeat until I'm satisfied.
I guess this way it gives a convenient temporary place for all the intermediate texts. I can of course edit them in place if I want.
You know it probably, but there is also a mouse chord for piping selected text into a command. (2-1)
2-1 is just a concatenation of two strings and run the whole string. I wouldn't call it pipe.
As is stated in the README, if you write 'rm' or anything like it... oopsie oops.
...? :D
Of course, much more thought is needed to try something like this. Somebody could as well use awk with its system command to do whatever..
This would save you, for example, if you had a custom command called "gr" which was short for "get rid of current directory" (obviously chosen as a pathological example since it's a prefix of grep). As you type the word "grep", auto-running is paused because "g", "gr", and "gre" are not whitelisted, and then once "grep" is fully typed, it recognizes that "grep" is on the whitelist and resumes auto-running. And it never ran the dangerous "gr" command.
Firejail could do exactly the same thing, but without requiring the user running it download an entire second operating system, or requiring them to have root. Also, the sandboxing mechanisms that docker uses are just generally available and aren't hard to use, so if they went that way they may as well just use the actual syscalls that do what they want instead of importing and entire other operating system to run your commands.
This is where my rant about docker, and the habits it encourages, would go. If I could figure out a way to phrase it politely.
The only solution that I would find acceptable would be not to execute incomplete commands as they're typed.
Giving a user an opportunity to correct mistakes before they kill themselves is a major feature of Unix. Sure when you’ve decided you know what you’re talking about but don’t it will quite obediently shoot your face off. We don’t want to make that bit easier though.
Edit: please don’t take this as derision but more factual. It certainly looks cool and opens the idea for further discussion.
Inversion of the output (viewing the top, instead of the bottom, without scrolling on longer outputs) is also of note
Imagine if this were simply the shell, with all its normal editing facilities, but laid out in this fashion. It seems obvious to me theres a usefulness anytime you’re trying to create a longer pipeline.. and doing the same thing the shell normally, but leaving a mess behind
Your list of issues are all resolvable (and probably trivially so), and thus not really relevant to the question of whether this thing is worth having in the first place (which requires more fundamental questioning). IE the issue with rm executing when you’re typing in ‘rmore’ is absolutely fatal in the current design and makes it unusable. That can’t be fixed without a bunch of hacks, or losing one of its main features (swapping to enter-to-run is more than fine imo). With such a fatal flaw, it could be fair to call it stupid and useless.
But for lacking readline support on first public listing..?
That's the complete opposite philosphy of Unix. That's, in fact, MIT/LISP philosophy.
Unix' Motto is DIE fast.
> But you'd be careful writing "rm" anywhere in Linux anyway, no? Also, why would you want to pipe something into "rm"?
But the thing is that you don't need to intend to pipe to "rm". Maybe you were typing something else, like having a command "rmore" or something. This danger is also not strictly limited to `rm` and `dd`, and you have to be careful that no substrings from the start of the command you intend to write cannot be interpreted as something dangerous.
greetings, eMBee.
EDIT: As an example, I may exploratively be playing with find options to match different sets of files in a directory hierarchy. Once I see I've matched the files I want, I may want to add a `-delete` to the end of the find conditions I specified to delete them all. That seems like a useful use of up, but making the filesystem immutable would disallow it.
edit: I totally get you as far as KISS goes; that's why I built and shared the tool as is in the first place ;)
Yeah, I still think it's not a good approach. rcthompson gave an example with network access, but that doesn't mean that bad things can only happen with filesystems or network access. You can also kill processes, shutdown the computer, and many more things. I don't think you can guess what mistakes the user might do, or which were really mistakes or were intentional.
Also, if you can avoid special permissions or capabilities for your core functionality, then that's better. People should be conservative in giving special permissions to programs, and I can't think it makes sense to give "up" the ability to make a filesystem immutable even if it's in an isolated namespace.
so at the very end you could apply a special action that now runs the resulting command with safety disabled.
or toggle between safe and unsafe mode at will, with safe mode being default.
it's not necessary to have the whole up utility run with filesystem disabled. but it could apply the restriction only to the pipeline its executing. that would allow it to selectively apply the restriction as requested by the user.
greetings, eMBee.
And rather a blacklist, since a blacklist would be much smaller than a potentially an infinite amount of programs to manage for a whitelist.
Btw, if you want to see all the commands you've ever piped data into, sorted by frequency, you can use this command (which I put together using `up`!)
history | grep -o '| \w\w*' | sort | uniq -c | sort -n | nlI think requiring a keypress by default to run any commands and maybe even throwing up a warning for a list of known dangerous ones (perhaps user configurable) would work well. A really fancy solution would be to do this stuff via some sort of linting.
An example of such a foobar with those exact options would be `inotifywait`. :)
I'd have replied to this on lobste.rs but don't have an account. I'm not a heavy poster but I'd love an invite if anyone has one going :)
1 - https://www.kernel.org/doc/Documentation/filesystems/overlay...
In case you aren't aware, your email address isn't visible in your profile either. I've emailed the GMail address that I found on your website :)
For example, one way to address the rm problem would be to give it a "safe mode" that drops user permissions and runs as nobody. You might also create a blacklist of common, potentially-dangerous unix commands that you don't execute. Both of those are examples of how to address the need without resorting to Linux-only tricks.
In practice it’s only as good as your users ability to avoid the temptation of ‘chmod 777’ (for example) but it seems a good place to start.
As an aside, I had the same idea to do a tool like this as well. Except mine would have been built into the $SHELL I’m writing (as that has a heavy focus on being more IDE-like than traditional shells). I scraped the plan for pipe previews precisely because of the dangers we’re discussing. However your tool makes a lot more sense because at least that is manually invoked whereas my original plan was to have that feature automatic and baked into the shell - which is an order of magnitude more dangerous. So I’m glad someone else has ran with this idea
I've seen somebody mention the idea of applying this to a shell, but only after pressing some special key (e.g. Ctrl-Enter). Would probably make it un-dangerous enough? This seems to be what people want of up too, anyway.
As to the "nobody", from what I'm reading, it seems you'd first have to be root, to be able to switch to "nobody"... so this doesn't really seem to be useful to me in this case... :/
Perhaps you could simply put those chown/chmod commands in the docs:
sudo chown nobody:nogroup path_to_up
sudo chmod ug+s path_to_up
I've tested it, and it seems to prevent deleting files with rm. What doesn't work, however, is that it also prevents writing the results to up1.sh. Perhaps if writing to the file fails (or you detect the process is running as nobody), you could send the finished pipe sequence to stdout instead of a shell script. Then, people could run it like: cmd | up > up1.shI appreciate this is a personal project and sometimes there is nothing more annoying than having feature requests; but if you did ever decide to add a flag for choosing alternative shells then drop me a message ( raise it as an issue on github.com/lmorg/murex ) and I’ll add ‘up’ as an optional 3rd party plug in.
greetings, eMBee
I ultimately just didn't find I used it much. If i am iterating, org-mode with shell sources fits a bit more natural for me.
Edit: and to be clear, I meant specifically to have the output below the command with the command still in edit mode. Was not trying to say it is the same as up.
I'm quite careful writing command lines because the whole experience of the shell is that of working with sharp knives: you get a lot of power but if you screw up, you'll feel the pain. The point of this tool is to take away a lot of the pain that teaches people caution. Its whole theory is "just try a lot stuff as you explore what you want".
In their shoes I'd look at using some of the container/security magic as a way of nerfing commands. If the on-a-keypress runs work in a way where they can't make changes to a filesystem, that seems way better to me. Even better if the tool then reports, "Would have deleted 532 files in a warning color at the top of the output."
A safety measure I picked up from a sysadmin while watching over their shoulder: start writing nifty command lines by prefixing them with # first, to prevent havoc when fat fingering the [enter].
For those who don't know... "The fc utility lists or edits and reexecutes, commands previously entered to an interactive sh."
The reason is that you'll see expanded variables and wildcards before commiting to them.
CREATE TABLE mytable_backup AS SELECT * FROM mytable;
SELECT * FROM mytable WHERE condition;
DELETE FROM mytable WHERE condition;Not exactly a pipe into `rm`, but `foo | xargs rm` is a common enough pattern.
https://www.kernel.org/doc/Documentation/prctl/seccomp_filte...
Once the command looks promising on the test container, run the command again in a fresh container to confirm, and maybe see a list of affected files. Finally run on the main filesystem outside of any test container.
Edit: similar ideas with family snapshots already suggested elsewhere in the thread
Perhaps one could run them as a fresh user with few permissions, except it could still write to files if they are writable by “other”
there should not be to many commands that need configuring.
greetings, eMBee.
I wonder if there’s a way to do this without requiring the creation of a new system user. Some way to revoke all write access for the current process.
https://www.freebsd.org/cgi/man.cgi?capsicum(4)
https://wiki.freebsd.org/Capsicum
https://www.cl.cam.ac.uk/research/security/capsicum/freebsd....
Capsicum is convoluted though.
OpenBSD has pledge and unveil, which from what I have seen are very elegant.
To become something more than a hack you have a few choices:
- a blacklist - will never be enough, still arguments may be problematic
- a whitelist - will always be not enough, arguments may be problematic, but a bit less than with a blacklist
- limiting permissions somehow - tricky
The last and best option IMHO would be to wrap it in a sandbox, where all filesystem access is behind an overlay (i.e. mount namespaces and overlayfs). This way if you are satisfied you can apply changes (if there is anything to change). The overlay would be removed and created on each keypress. It may be also possible to wrap process access, so one could safely play with kill, but I'm not sure. Even network settings to some extent. But there is nothing you can do with curl -X POST or something similar.
Then I was like, I bet quite a few ancient wizards out there have some pretty amazing techniques they have picked up over the year. And that it would be great to be a "fly on the wall" watching some of these wizards do simple tasks. And that a YouTube channel watching them do simple tasks could be quite revealing and therefore very educational.
Paul Irish did something like this a few years ago and I still use a bunch of the tools he introduced into my purview. Really, really improved productivity!
Does such a channel exist? All I've ever found has been tutorials. I want to watch unix veterans use the tools. I've found that to be the best way to learn. Hell, that's how I learned vim.
https://www.youtube.com/watch?v=qsWY-8n9igM
If anyone has taken his courses, I'm sure they could give better feedback on those. Either way, watch the video.
I do a pretty similar version of this with command-| in textmate, but I've wished for/thought about building a command like this to speed up the process!
The textmate approach does have the advantage (or perhaps disadvantage, for large enough inputs!) of emitting immutable "views" of the data at each step, rather than (presumably?) re-evaluating the pipeline each time, which is nice if the steps of the pipeline are the slow part, or to go back 2 steps and start a new pipeline. Maybe a potential future flag? :-)
Automatically emitting an `up<N>.sh` script is super clever too!
consider the command
A | B | C | D
when you process it in up, split the command at the | and process each one separately. that is first run
A;
save the output into a buffer (BA)
then pipe BA into B, save that into BB, and so on, until you have 4 buffers containing the outputs of A; A|B; A|B|C; and A|B|C|D;
now when the user edits, you can see which section is changed. say the user changes the command C into C1, so the pipeline becomes: A|B|C1|D
at this point you can pipe buffer BB (which has the output from A|B) into C1; write that into a new buffer BC1; and pipe that into D;
the old buffers C and D can be removed at this point, or you can keep them around in case the user undoes their change and goes back to an old version.
greetings, eMBee.
Bash lets you provide file arguments from pipes via the <() syntax (it's turned into a fd reference via proc behind the scenes). And you can also wire up additional pipes with fd renumbering, though this gets ugly in bash, some other shells are more flexible there, e.g dgsh.
A tool to iteratively build a graph of commands and pipes would be the next step.
https://youtu.be/CZHEZHK4jRc?t=1273
When I see stuff like this, I check the date to see if it happened before or after "Inventing on Principle" was presented in 2012. (It's always after)
I use it all the time to select a command from my shell history. It is a very convenient addition to zsh-autosuggestions (one of those things that you use hundreds times per day without even thinking about it).
E. G. I type a command in the shell, end it with pipe, and press enter, it doesn't delete the command but runs a preview of the output top 10 lines? This way I can continue to the edit the command without losing the context, the pipe, or I can decide to redirect it to a file!
For something a lot dumber: here's a tiny thing to print out pipe contents to allow for cancelation before continuing the pipe.
https://gist.github.com/jasisk/be34ae93b74a3d3e8f0a4daac1237...
I will definitely try this out too though when I make a pipeline with sed, cut, etc.
There is already some "pre-release" discussion going on over at lobsters btw.
https://lobste.rs/s/acpz00/up_tool_for_writing_linux_pipes_w...
Sure, the shell has some powerful tools like jq that editors usually lack but otherwise several levels combo piping through cut, wc, awk, etc are quite rare and are usually only done in crazy shell scripts that actually should be rewritten in more appropriate language.
And the dangers of automatically running any commands shouldn't be downplayed. If you only whitelist a few programs this would be much safer, most users would only use this for grep. Or run the whole thing through a docker container.
What would be really cool is if this tool ran in a pretend mode and showed you what the results would be without actually mutating the system. Then, once the user is happy with the results, the command can be executed for real.
This whole concept reminds me of helm for emacs :-). Thanks for sharing!
lshw | grep network -A2 | grep : | cut -d: -f2-
A better idea would be to work on tools that have structured output, so people can select the keys they want and don't have to scrape. $ Get-NetAdapter | where status -eq 'up' | select InterfaceDescription
InterfaceDescription
--------------------
Qualcomm Atheros QCA61x4A Wireless Network AdapterVery similar result and using rm is not that dangerous as you can finish writing your code and it gets executed as soon as you save the file (if you don't like the watch delay you can use one of the many inotify based watch tools or set -n 0.1 or something that suits your needs).
Thank you
``Remember, a temp file is just a pipe with an attitude and a strong will to live'' - Randal Schwartz [comp.unix.questions]
Please at least respect the SHELL environment variable. (-:
If I type "grep", I don't want it to try to execute "g", "gr", and "gre" as I type it -- not if those commands don't exist, and especially not if they do.
Just don't execute a command unless the user requests it (say, by typing Enter). Problem solved.
The obvious thing to do is only run what's been typed so far on a trigger key like tab or something.