Adventures in /usr/bin and the likes | Hacker News Reader

bin and the likes

ablagoev.github.io

268 pointsmubaris78 comments

Worth it just for learning that there's a "replace" command, for multi-file search-and-replace without having to remember sed syntax.

Can't believe I've been running Linux for 16 years and I didn't know about that one.

On my system (Ubuntu 16.04) replace is not installed by default.

It appears to be provided by the following packages:

    mysql-server-5.7
    mariadb-server-10.0
    percona-server-server-5.6
    percona-xtradb-cluster-server-5.6

I downloaded mysql to check, and it appears that the replace provided is as described in the article; I assume the others are as well.

I'm not sure how I feel about installing something as big as mysql-server just to get one utility.

Unfortunately it seems to pull in some MySQL/MariaDB specific code - https://github.com/MariaDB/server/blob/bb2c1a52c61706dde8c52...

It's also worth noting that in MySQL, the replace command is deprecated in 5.7.18 and removed in MySQL 8. [0]

I don't know its status in MariaDB.

[0] https://dev.mysql.com/doc/refman/5.7/en/replace-utility.html

I was tempted to poke a little fun at you for having trouble with sed -- among my peers, 's/foo/bar/g' used pretty much daily, even just in informal emails/conversations.

But I'm glad I looked at the list first, because I found this wonder: `lstopo --of txt`. I'm going to use this in class next week, I can't believe I've never seen that before.

My biggest hardship with sed is having to figure out how to escape the operators within regular expressions on the command line. I think it's pretty reasonable to spend 15+ minutes on a complicated replacement regex.

Edit: It looks like I agree with you; `replace` only solves trivial problems that can easily be done with `sed`.

You can use single quotes to escape everything except single quotes from the shell. Then use backslashes to escape single quotes where needed.

If you haven't taken the time, I'd say it's well worth learning exactly how the shell escapes work. It's surprisingly simple and natural once you get used to it.

Thanks for the advice; I should block some time to fully grok shell escapes.

Even with single quotes, sed still requires escaping parentheses, + operators, and other operators that would otherwise be interpreted literally (but not all operators should be escaped). In the languages that I learned regexes this wasn't required.

There's a command line switch -E for sed that gives you what we think of as "normal" regex escaping.

You have saved me many hours of future command line hacking. Thanks!

Also sed can use other characters instead of / for separating fields, like using _ instead for / heavy stuff:

  sed -e 's_https://www\.example\.com/_/_g'

I think it's nicer than:

  sed -e 's/https:\/\/www\.example\.com\//\//g'

Oh, the basic syntax is fine. I'm a Vim user ;) And even before that, s///g was also part of my slang.

My issue is remembering how to invoke the command - whether it edits in place by default, what the order of arguments is, etc. To be fair, that's not necessarily that much easier with a dedicated replace command either.

Not covered in the article, but `rename` is a multi-file renamer. I wrote a script which used an `echo $f | sed | mv` loop before I learned about this.

Be careful there's at least two implementations of rename in different Linux distros. I've run into one made in perl that uses regex in Debian (e.g. `rename 's/foo/bar/' * `). And another one that uses simple strings (`rename foo bar * `) in CentOS.

Indeed. The Perl `rename` is based on the module [File::Rename](http://search.cpan.org/~rmbarker/File-Rename-0.20/).

This name clash makes that the perl module cannot be easily used on CentOS, since the CentOS utility is required for building RPM packages.

You'd have to edit Makefile.PL of the Perl module before installation, to avoid that clash. (I'd prefer to keep the name as "rename.pl", in this case; though file-rename, as on Windows, would work too.)

I use moreutils' vidir for renaming, I don't know rename though.

rpl is pretty nice as well.

I was super confused at your comment. Search on Arch Linux packages, it looks like the replace command is shipped with mysql/mariadb.

Reminds me of my sysadmin days when I was trying to learn shell scripting and would lookup man pages of random binaries in /usr/bin. Before we could bring down a box for nightly maintenance/backup, we would send a shutdown broadcast notice like:

banner "Shutdown"|wall; banner "in 5 mins"|wall

I found out "write" command and setup a hacky little one-to-one chat script - it was terrible and geeky but I really liked it.

I'm probably forgetting some more obscure commands that I enjoyed :)

PS: forgot to mention that when I told my boss about the "write" based scripts, he gloated that he had setup a quiz program which supported multiple participants using named pipes. Awesome guy :)

banner is awesome - any way to make it output slowly instead of always at once?

while IFS= read -r line; do echo "$line"; sleep 0.1 ; done < <(banner Hello)

ISTR it was Æleen Frisch (author of Essential System Administration) who recommended that all nix admins take one day a year to read through the entire manual sections (1) and (8), which I implemented thus:

$ cd /usr/bin ; man

(and repeat for other bindirs)

You will be amazed at what you've forgotten, and what your system can do.

  $ cd /usr/bin ; man
  What manual page do you want?

Is there a command to get all the man pages? I tried `man *` and the internal help, but no luck.

Edit: q then Return did it. D'oh. The obvious follow-up question is how do I go back to the _previous_ man page?

$ man man ;)

Actually, I could not find anything relevant by searching it for "file" and "next". I guess it's purely the responsibility of the pager, which presumably is not part of man itself.

Here ya go; all section 1 (user commands) man pages:

man $(man -s 1 -k . | cut -d' ' -f 1 | sort | tr '\n' ' ')

Ah sorry that got escaped, was supposed to be

     cd /usr/bin ; man *

I was thinking about something along this: get a random number (say dd'ing from /dev/urandom count=1 bs=2 or using shuf), then get the modulo M with number of entries in /usr/bin, then find the Mth entry of the ls, and print the man page. A file containing titles of already printed man pages can be used to exclude doubles.

After reading this I'm reminded that if an IDE just supported replacing highlighted text after it was piped through a command it would have a lot going for it. A lot of these are features some that IDEs have been lacking for years yet they've been hiding away in /bin.

FWIW, you can do that in Vim. Select text then:

  :'<,'>!command

And `%!command` to replace the whole file. That's how editing hexadecimal in vim works. `%!xxd` to turn it, then `%!xxd -r` to turn it back.

Another useful one is `%!jq .` to pretty print a json file.

I use this often. I usually enter `:%!xxd -g1` to show each byte value separately instead of showing two bytes together.

    $ printf "\x00\x01\x02\x03" | xxd
    0000000: 0001 0203
    $ printf "\x00\x01\x02\x03" | xxd -g1
    0000000: 00 01 02 03

While editing binary files in Vim, one needs to be careful about setting the binary option before editing a binary file or opening the binary file with `-b` option.

For example,

    printf "\x00\x01\x02\x03" > a.bin
    vim a.bin
    :%!xxd -g1

would display

    0000000: 00 01 02 03 0a                                   .....

(The `printf` and `vim` commands are entered in the shell. The `:%!xxd` command is entered in Vim.)

In the above example, a newline has been added by Vim where none existed in the binary file. In fact, if you then run `%!xxd -r` followed by `:w`, this additional newline would be written back to the file. Here is the right way to preserve the binary data in a binary file.

    printf "\x00\x01\x02\x03" > a.bin
    vim -b a.bin
    :%!xxd -g1

would display

    0000000: 00 01 02 03                                      ....

Only the four bytes in the file appear now. Alternatively, one may enter `:set bin` in Vim before editing a binary file.

You can also use a range e.g. `5,10!column -t` would align columns for lines 5 to 10.

MPW - macintosh programmers workshop circa 1988 (when I first used it) had the notion that selections within windows could be treated as just another file. It had a lot of posix-y commands but lacked multitasking and all pipes were done via temp files, alas.

In emacs, `C-u M-| cmd <enter>` does this.

Acme, the text editor from Plan 9 has this. Piping, inputing and outputing from commands are some of the most basic operations there.

Check out:

- http://acme.cat-v.org/

- Russ Cox' tour: https://www.youtube.com/watch?v=dP1xVpMPn8M

Is there an updated version with less complicated mouse interfaces?

This is what BBEdit filters do:

http://bbeditextras.org/wiki/index.php?title=Text_Filters

In my previous job i wrote an extension for Visual Studio to do exactly that: call a program with the selection, pass the input to its stdin and replace the selection with the output.

Then i wrote a bunch of Python scripts to do things like recalculating command id's for wxWidgets, generating the registration, header and body parts from event definitions (which i cut/pasted later, but it was much faster than doing that by hand), aligning some declarations to fit with the code style, etc.

The extension would also scan a folder inside My Documents and register any executable, python script, bash script and tcl script to a menu in the IDE. Many people in the studio knew Python and some new Bash (i think i was the only one who used Tcl though, i mainly used it to quickly search the filenames for the project and open a file in VS or Vim. I made it to search as i type and since we had a more or less 1-to-1 mapping between objects and files, i could search for object definitions faster than using VS's search functionality - especially when VS was busy building or doing stuff).

The KDE text editor, Kate, has this. It is absolutely wonderful, and Kate is full of such gems.

Kakoune makes this a central concept. Just hit pipe ('|') in normal mode. It even works for multiple selections!

What a list of gems!

Also, I noticed you can rewrite this function:

    # Copy an ip address from a domain
    # Usage cip domain.com
    function cip() {
        ip=$(resolveip -s $1)
        echo $ip | xsel -b
        echo $ip
    }

as a oneliner without needing process substitution or echo:

    function cip() {
        resolveip -s $1 | tee /dev/fd/2 | xsel -b
    }

I'm never 100% sure I understand FD redirection, but I think you can shorten that even further to:

    function cip() {
        resolveip -s $1 1>&2 | xsel -b
    }

Since this'll copy stdout to stderr, and then stdout will get eaten by xsel

It won't copy stdout to stderr; it'll move stdout to stderr. Stdout will be empty after the redirect, so no, thos won't work :)

Have you tried it out? It worked on my system.

The parent is correct, the given command redirects stdout to stderr, it's not copied (so stdout will be empty).

Again, have you tried it?

    # echo "foo" 1>&2 | sed 's/foo/bar/'
    foo
    bar

Edit: I've done some more testing, and discovered that the above works on zsh, but not in bash

2nd Edit: Ahha! http://www.cs.elte.hu/zsh-manual/zsh_7.html . So this is because zsh w/ the stock config (MULTIOS option enabled) will open as many outputs as you give it. So it can both copy FD 1's contents to FD 2 and to the pipe'd command.

A little spelunking may help:

    $ lsof  -d 0,1,2 -a -p $$

That will show you what files correspond to what file descriptors for your current shell process. You can then experiment with redirects. To see how a pipe works we can do something similar:

    $ sleep 1 | lsof -d 0,1,2 -a -p $(sh -c 'echo $PPID')

Anyway, thinking of `>` and `<` as just FD setters made things a lot clearer for me personally. They just set the file descriptor on the left to file on the right and `&n` can be used to "dereference" a FD to its corresponding file. The only real difference between `>` and `<` is that the former opens the file with write permission while the latter opens it with read. Well, and the default FDs are different.

Incidentally, this makes it clear why the following doesn't work as expected:

    $ curl 'https://kernel.org/' >/dev/null 2>&1 | tee curl.errs

Since FDs get set in the order they are specified, 1 gets set to /dev/null and then 2 gets set to whatever 1 is, i.e /dev/null. So it's clear that we probably intended the following:

    $ curl 'https://kernel.org/' 2>&1 >/dev/null | tee curl.errs

Which lets us ignore stdin and maybe pipe stdout to some other command.

Now I hope Linus doesn't hate on me for abusing kernel.org :P

I would favor "dig en.wikipedia.org +short". I think it is installed by default (ubuntu, redhat). It does not require string parsing.

This article is very interesting, but I prefer to use sed or readlink /proc/<pid>/cwd instead of learning additional commands like rename and pwdx.

Also:

    resolveip -s $1 | tee >(xsel -b)

(which I prefer)

Apparently "resolveip" is part of... MySQL.

You may want to use "getent ahosts" instead:

  $ getent ahosts en.wikipedia.org
  91.198.174.192  STREAM en.wikipedia.org
  91.198.174.192  DGRAM  
  91.198.174.192  RAW

or simply "host" !

That's an interesting command! (note: on FreeBSD, just "hosts", there's no "ahosts")

I usually use drill(1) (from unbound, which is part of base on FreeBSD), or dig(1) on Linux boxes that don't have unbound.

Also turns out that that replace command is also part of MySQL. LOL WTF MYSQL?!

A command I often use for a nice overview of running stuff:

  $ netstat --numeric-ports -elp46 | sort

    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
    tcp        0      0 0.0.0.0:22223           0.0.0.0:*               LISTEN      root       12159       3370/sshd       
    tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      root       781140      4940/nginx -g daemo
    tcp        0      0 0.0.0.0:4000            0.0.0.0:*               LISTEN      postgres   780857      4916/postgrest  
    tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      root       13126       3537/master     
    tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      postgres   12526       3406/postgres   
    tcp6       0      0 ::1:5432                :::*                    LISTEN      postgres   12525       3406/postgres   
    tcp6       0      0 :::22223                :::*                    LISTEN      root       12168       3370/sshd       
    udp        0      0 0.0.0.0:123             0.0.0.0:*                           root       12479       3400/ntpd       
    udp        0      0 127.0.0.1:123           0.0.0.0:*                           root       12485       3400/ntpd       
    udp6       0      0 ::1:123                 :::*                                root       12487       3400/ntpd       
    udp6       0      0 :::123                  :::*                                root       12480       3400/ntpd

Apparently netstat is one of the commands being depreciated on Linux, to be replaced by ss.

Wow this list is tremendous! It's fun to still discover new commands after all these years.

I actually used `taskset -c` just recently, when I was running lots of sidekiq processes and wanted to make sure they all used different cores. It helped me get full utilization out of the box, whereas before I would frequently see some sidekiqs competing for a core while other cores sat idle.

Be careful with taskset and chrt, assigning realtime priority to processes and pinning on them on CPUs might lead to un-expected behavior, priority inversion or your system locking down (say if you pgrep some kernel tasks/threads in there as well by accident, I've done before).

Thank you for the warning! I can see how that could happen with chrt. I can't work out how it could happen with just taskset. Are you saying to be careful when combining them? That would make sense to me.

Also I can't figure out what pgrap is. Do you mean pgrep? I think you must mean something else, because I don't see what pgrep has to do with priority inversion. But it shares a manpage with pkill. . . .

I don't do real-time work, but I've always been curious to know more about it. And priority inversion seems like an interesting and challenging problem.

> Also I can't figure out what pgrap is. Do you mean pgrep?

Sorry meant pgrep.

> I don't see what pgrep has to do with priority inversion.

Grepping processes doesn't have anything to do with priority inversions and locking your system. But grepping processes then setting them to realtime priority sched_fifo 99 or could wreck havok

> I don't do real-time work, but I've always been curious to know more about it. And priority inversion seems like an interesting and challenging problem.

It can be fun. It is kind of a separate world on its own. Interestingly most of the popular OS-es by default are tuned for throughput not low latency. So configuring and getting everything just right is an interesting challenge (sometimes involving applying kernel patches, although lately that might not be necessary as some have been mainlined).

A lot of really cool stuff here! shameless plug I have a small side-project site that goes through a lot of common activities on the command line https://cmdchallenge.com. Some great ideas that I think I will incorporate.

Going off a comment on the article, using mainly awk to find your commands, I expanded it a little bit, as most commands I use take a first argument that changes how it works. (Aliasing this command might not be a bad idea.)

    history | awk '{print $1 "__-__" $2}' | sort | uniq -c | sort -rn | head -30 | awk '{gsub("__-__", " ", $2); $2} !max{max=$1;}{r="";i=s=100*$1/max;while(i-->0)r=r"#";printf "%50s %5d %s %s", $2, $1, r, "\n";}'

It produces a histogram of the 30 most frequent used commands.

Unsurprisingly for me, `git commit` was first. Kinda surprisingly, `apt-get` was third.

Caveat: I convert spaces to the symbol sequence: __-__ and back. If your command includes that symbol sequence, it'll break something.

Pipes are not exclusive to *nix based systems. Windows e.g. has had them for quite some time, dating back to at least DOS 7[0] from Win95.

[0] - http://www.lagmonster.org/docs/DOS7/pipes.html

No reference, but I'm pretty sure I remember it from DOS 5.

In early versions, DOS pipes weren't real pipes. It would run the first command and save the output to a temporary file; then once that was finished, run the second command with the file as input. I'm not sure when it gained real pipes.

Around windows nt/95 when multi-tasking was added would be my bet.

Windows pipes appear to still be second-class citizens. Look at this mess: http://stackoverflow.com/questions/34504970/non-blocking-rea...

The eventual solution using SetNamedPipeHandleState isn't really supported: https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... "Note that nonblocking mode is supported for compatibility with Microsoft LAN Manager version 2.0 and should not be used to achieve asynchronous input and output (I/O) with named pipes."

But pipes were original and first to *nix based systems, first appearing in Version 3 of UNIX:

https://en.wikipedia.org/wiki/Pipeline_(Unix)#History

So? The article mentions pipes as a distinctive feature of *nix systems. Neither it nor I claimed who had it first. The truth is piping from devices, programs and files is pretty much the same in Windows :)

I never said anyone was wrong; only pointing out the origins.

I think one might be able to successfully argue that NIX-like systems and the programs executed on them are distinctively* biased towards pipes.

Regardless, as you said, the point is moot.

These are named pipes, not command line pipes. Windows NT was the first version of DOS that had them and they were copied from UNIX.

In Windows, specifically using |, the pipe is anonymous[0]. Named pipes can be created, but named pipes (fifos) on Windows are substantially different from those on unix. Named pipes are not relevant to the pipes my reference above introduces in Windows 95's DOS system.

[0] - https://en.wikipedia.org/wiki/Anonymous_pipe

[1] - https://en.wikipedia.org/wiki/Named_pipe

Windows NT was not a version of MS-DOS at all. This is why it had true pipes, of course. Their semantics largely came from OS/2, not from Unix.

fwiw wc -l in macOS Sierra 10.12.3 reports 1059 entries. There are quite a few programs that are Apple specific (macbinary, appletviewer). But many of the ones talked about in the article are here too.

> date > /tmp/test_at.log will be executed at 9.26 PM on the same day

well that's not guaranteed on all systems :D On FreeBSD:

> at is implemented through the cron(8) daemon by calling atrun(8) every five minutes

Sigh. /tmp is not your personal playground. Forget that it exists, unless you're a security expert.

Novice question. Why the temp folder is problematic? Some cursory google search shows up this http://security.stackexchange.com/questions/11606/what-are-t...

But generally what kind of attacks /tmp folder enables ?

Consider the headlined article, for starters. A simple

    ln -s /etc/passwd /tmp/test_at.log

which any unprivileged user can quietly do, before running its examples for at, will ruin the system administrator's day just after 21:26.

This is the basis for many of the problems with files in /tmp : predictable filenames written-to without care by privileged processes. It is a widespread disease that is the reason that the systemd people arrange to run many programs with PrivateTmp=true .

In this particular case, root's own home directory would have made a much better private playground for the test_at.log file.

Reply on news.ycombinator.com