Cool, but obscure Unix tools | Hacker News Reader

Cool, but obscure Unix tools

kkovacs.eu

534 pointslarelli121 comments

  abcde            - CD to mp3 ripper
  apg              - random password generator
  base64           - better than uuencode
  boxes            - draw any kind of boxes around your text
  bsdiff           - binary differ
  bspatch          - binary patcher
  bvi              - binary vi (yet another hex editor)
  ccx2             - console xmms2 client
  clive            - flash video downloader
  dvipdfmx         - dvi to pdf converter
  enfuse           - poor man's HDR
  get_flash_videos - yet another flash video downloader
  glark            - advanced grep
  indent           - code beautifier
  lshw             - list hardware configuration
  mcurl            - multiple part downloader using curl
  mktemp           - safely create temporary files and directories
  msort            - sort records in complex ways
  netbrake         - bandwidth limiter
  od               - octal dump
  par              - paragraph reformatter
  par2             - archive verification and repair tool
  ped              - sed done right with perl
  pinfo            - color info reader
  pipe.vim         - make vim part of a unix pipe and allow it to
                     edit the pipe contents
  pv               - Pipe Viewer: a tool for monitoring
                     the progress of data through a pipe
  pydf             - pretty df (disk space viewer)
  qmv              - use your favorite editor to rename files
                     (part of renameutils)
  qodem            - modem program that can do serial, telnet, ssh,
                     zmodem, kermit, etc
  rdiff-backup     - like rsync, but can do incremental backups
  recode           - like dos2unix and unix2dos, but with many more encodings
  recordmydesktop  - make screencast videos
  remark           - great logfile colorizer (part of regex-markup)
  rkhunter         - find rootkit infections
  rlwrap           - add readline editing support to any command
  safecopy         - data recovery tool (better than dd)
  sponge           - soak up stdin and write to a file
                     (for things like pipeline editing)
  sux              - su while transferring X credentials
  unbuffer         - force flushing of stdout
  upx              - executable compressor
  utimer           - countdown timer and stopwatch
  vared            - edit shell variables (part of zsh)
  watch            - run a command multiple times and display the output
                     (with differences highlighted)
  xdotool          - simulate keyboard and mouse activity
  xxd              - hex dump
  zargs            - a version of xargs that makes the find command redundant
                     (part of zsh)
  zed              - very small and fast vi-like editor (part of zsh)
  zrun             - automatically uncompress arguments to command

This sounds kind of silly to say, but when I spent 5 minutes getting abcde set up (to rip, encode, and tag into Ogg Vorbis, MP3, and FLAC at the same time), and then had it finish encoding everything just 30 seconds or so after it finished ripping, I felt like I was in the future.

I mean, I remember even 5 years ago ripping a CD took actual effort and a long time to get all the encoding done. So yay for abcde.

This is a very nice list. I didn't know most of these tools before, and they look very useful; I'm going to try and learn some of them. I found the link in the topic a bit disappointing. It mostly contained tools for system diagnostics rather than work flow optimisation.

Great post, I didn't know about a lot of these - I wrote my own version of qmv, now I don't have to use it :)

recode is seriously useful if you need to take the output of a PowerShell script and process it on your Unix box.

trafshow - ncurses based utility showing detailed network traffic

How is vim an obscure tool? I'm pretty sure quite a few people use vim daily. After looking over this list, I suspect it's also true for many listed programs.

I knew most of them but not all. However as he targets OSX users, I suspect many of the programs are unknown to them.

Ah - thanks clarifying that he targets OSX people. I didn't think that dstat, xargs, curl, vim, screen, rsync, or ack were obscure; but it makes sense if it the audience is OSX users.

I don't think OSX is specifically that important - I've met plenty of "new" linux users who have no idea htese tools exist either - linux has a gui too, you know.

Converseley, there are plenty of long-term unix experts out there who also find that a mac makes a nice workstation that still satisifies their need for unix tools plus other stuff.

Please see my earlier response to mitjak, i think it addresses both of your points.

Why?

Not a dig at OSX users. I have a macbook in addition to a thinkpad running a less common distro in front of me right now. I'm thinking of it in a Bayesian way: if a person is a Linux user or an OSX user, I expect them to know all/most of those tools. However, given that a person (limited to Linux or OSX users) doesn't know about those tools, I expect them to be an OSX user.

I don't think he's targeting OS X. He just has an OS X skin on his UI. Some of these tools absolutely will not run on OS X.

There really is no "target" audience of this post. :) It just started out as a list of "recommended" tools, then people started suggesting stuff -- some of which I knew, some of which I don't, some of which run on every unix, some only on Linux.

And it was easiest to take the screenshots from my OS X laptop. For some tools, like powertop, I had SSH-d into various Linux or FreeBSD boxes.

Some of these are uncommon enough that they are not ported to every package manager system, too. :(

Out of curiosity, which?

Edit: looking at the site again, at least one of those screenshots is from OS X. htop lists /sbin/launchd and several processes from /Library/Frameworks.

Nice catch, I thought the same as GP because of powertop.

quite a few people uses vim daily?? My life is spent inside vim :) -- ups sorry got the wrong impression.. You should use sarcasm tag. Shame on you!

vim is actually more popular than Emacs (FOR NOW), so I don't really know where he's coming from on this one.

          ====        ________                ___________
      _D _|  |_______/        \__I_I_____===__|_________|
       |(_)---  |   H\________/ |   |        =|___ ___|      _________________ 
       /     |  |   H  |  |     |   |         ||_| |_||     _|                \_____A
      |      |  |   H  |__--------------------| [___] |   =|                        |
      | ________|___H__/__|_____/[][]~\_______|       |   -|                        |
      |/ |   |-----------I_____I [][] []  D   |=======|____|________________________|_
    __/ =| o |=-~~\  /~~\  /~~\  /~~\ ____Y___________|__|__________________________|_
     |/-=|___||    ||    ||    ||    |_____/~\___/          |_D__D__D_|  |_D__D__D_|
      \_/      \__/  \__/  \__/  \__/      \_/               \_/   \_/    \_/   \_/

tip: try different combinations of options (sl -la, sl -DF, ...)

once upon a time I wrote a patch to sl, for s/car/cat/ typo.. you can get it from my site: http://www.yalazi.org/2008/11/08/slssl-ve-scatcar-yamasi/#mo....

If rsync is obscure, then I'm a dumpster wizard. That's a wizard that lives in an overturned dumpster.

I really like pipe viewer (pv). I wish it as more common in base installs of Linux.

http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

Although not as featureful as pv, try hitting CTRL+t under OS X or any of the BSDs during a long running process. It provides status of the current running process without having to prefix the command with pv. I blogged about this a while back: http://jz.posterous.com/bsd-tip-of-the-day-ctrl-t

> nethack & slash'em

> Still the most complex game on the planet.

Dwarf Fortress.

Definitely agreed, but is there anything more complex than DF?

As a sidenote, to anyone who doesn't know the story of how Dwarf Fortress is being made, you should check it out. It's one guy, working full time on it, and managing to support himself and his brother through monthly donations. It's released completely free of charge, so this is just solely from people donating. That alone would be a pretty cool story, but when you look at the immense, mindboggling, ridiculously ambitious scale of the the planned finished game, it's incredible. He's basically trying to build a completely "generic fantasy world simulator", with procedurally generated world, history, etc. The amount of detail he goes into in this is amazing, and he has so much more planned. If you ever need to be inspired that one person can make a living doing what they love, or that you can actually implement features that are ambitious beyond most AAA developers' wildest dreams, look to Toady One.

As a bonus, it's a pretty fun game!

Also, check out Boatmurdered[1], a "Let's Play" of Dwarf Fortress. It is pee-your-pants funny and contains some strong language.

The sequel, Headshoots[2], is also funny but not quite the same calibre as Boatmurdered. FYI, the name of your fortress is randomly generated.

[1] http://lparchive.org/Dwarf-Fortress-Boatmurdered/Introductio...

[2] http://lparchive.org/Dwarf-Fortress-Headshoots/

Boatmurdered is what got me to play DF in the first place.

Same here. :)

DF does have very impressive depth and game mechanics. Unfortunately, its interface is one of the worst of any game I've ever played.

It's a pity that DF isn't open source either, as then its interface problems would have long since been fixed. But, as it is, its lead (and only) developer doesn't seem to care enough to fix it himself.

DF is the deepest game (really, toy, like SimCity; there's only one end state within the game and that's getting wiped out, and most of the rest of the game is very open ended) I have ever played which pairs it with a terrible interface and a learning curve like a brick wall (no joke, I only figured out how to get doctors to take medical jobs promptly a few weeks ago after playing the game for months). The DF interface problems aren't just that it's console oriented; there's no straightforward way absent external tools (which I generally can't have, because I play on a Mac) to get a list of e.g. which dwarves have a certain labor enabled, or which dwarves have skill levels in a certain labor. Also the way the game is played today is very stuff intensive which means you tend to have between a quarter and up to half of your fort on full time crap hauling duties.

I heard that it is actually written using GL to draw the character glyphs. Hilarious.

From data/init/init.txt

> By and large, 2D should be the most reliable, while STANDARD has a good combination of speed and reliability. However, all 2D modes are normally fa r slower than even STANDARD, which may be the slowest OpenGL mode.

It is indeed. However:

> Linux/OS X users may also use PRINT_MODE:TEXT for primitive ncurses output.

I think he's said it's on his to-do list, but he wants to add more 'cool' stuff first. Also you can get tools like dwarf therapist which take some of the pain out of managing the UI.

Just an example of the "amount of detail": the game tracks things like beard length, nausea level, ratio of indoor time to outdoor time, noise level, and it simulates the entire history of the planet before you start.

It's just too bad he can't come up with a UI that would appeal to any but the most dedicated players.

Not just in game, but out of game detail, too.

There's a player trying to get a sample of saguaro wood so they can measure some of its material properties empirically and add those to the game. And that's just one small part of some very large, recent threads researching real-world stuff to improve the game's model.

Someone actually went through about a couple of hundred random types of stone and wood to find out more information on them and input proper data for them, rather than using the defaults for wood and stone.

That thread is somewhere on the Suggestions forum.

Definitely agreed, but is there anything more complex than DF?

Until someone writes out rules for Azad, I think Dwarf Fortress will remain king of the complexity hill.

Has there been any attempts to do that?

Dwarf Fortress was already suggested by somebody, and I very much like it, too (haven't had the time to look deeply enough, though), but it's not a terminal game. Yes it uses characters; but one actually needs OpenGL for the game.

Still, a very nice one, that is!

Not actually true; you can play it in terminal/ncurses mode.

Now checking Dwarf Fortress, but must mention Deadly Rooms of Death.

tsort -- perform topological sort

I bet very few people here are aware of its existence, even it has been part of Unix since Version 7. I recently discovered it and have used it to solve some project Euler problems.

Have a look at `join', it joins lines of two files on a common field. Together with `cut' and `grep' you can use text-files as relational databases.

>Have a look at `join', it joins lines of two files on a common field. Together with `awk' and `sed' you can use text-files as relational databases.

Fixed it for you.

awk and sed are useful. But as a principle one should use the weakest tool that gets the job done.

That principle also applies to grep regular expressions vs Perl regexes. Regular languages are quite weak in the right way, and thus can be recognized in linear time. Perl's regexes on the other hand, are so powerful, they can even tell prime numbers from composite numbers with their back tracking.

Choosing a weaker tool also serves as documention---about which features not to worry about.

(Of course you shouldn't try to use a tool that's weaker than what you need. For example, trying to solve some problem crying for recursion or iteration in a spreadsheet will lead to more harm than good.)

I recently discovered nl and felt sad about the amount of time I had wasted figuring out how to add line numbers to files.

cat -n

but nl is shorter :)

They're both annoying. For use in pipelines that is, as they seem more geared towards typesetting. Which I've honestly never had a use for. Consider this input file:

  $ cat > test.txt
  one
  two
  three

  five

Let's try to number every line:

  $ nl test.txt                                                            
       1       one
       2       two
       3       three
       
       4       five

Ok, the leading spaces are annoying (why not use a single tab?), and are also produced with "cat -n". And why is not numbering blank lines the default?

How can you prefix every line, even non-blank ones, with the line number followed by a single tab? Turns out the answer is:

  $ nl -w1 -n ln -ba -s"`printf '\t'`" test.txt

Horrible defaults. HORRIBLE.

Why does cat need the ability to add line numbers anyway? It seems outside the scope of it's purpose. Next it'll need ability to translate Russian to English.

If I ever needed line numbering, I would output into a file and open it in a far more capable text editor/viewer that was designed for doing that well.

Indeed - `nl` should be preferable to `cat -n` for that reason even if for no others. People have been actually saying this since at least 1983 (http://harmful.cat-v.org/cat-v/).

I'm often surprised that cat is anywhere near as popular as it is. I think I've used it for actual file concatenation maybe two times in the past three years.

> Ok, the leading spaces are annoying (why not use a single tab?)

That's because numbers look better when they are right justified.

The blank line thing is stupid though.

> That's because numbers look better when they are right justified.

They don't look better to a computer though. And that's what I don't get. Why are the defaults geared towards typesetting (human consumption) while cut/sort/join friendly output is so painful to produce?

This is a very common scenario when processing log files: line numbering, then sorting on another field followed by join/grep/head to grab a subset of interest, then recovering the original line numbers for the subset of interest.

Honestly half the time I end up doing this, just because it's the shortest invocation that comes to mind:

  perl -lape '$_="$.\t$_"'

I've never heard of your use case before--that is interesting.

Though I'm not sure I see why the "%5d" method is hard to parse...

  nl /usr/share/dict/words | head -20 | sort -n

seems to work ok. "cut -f 2" and "cut -c 8-" also seem to work well in this case.

Yup, use awk for similar tasks:

  awk '{ print FNR "\t" $0 }' test.txt
  1	one
  2	two
  3	three
  4	
  5	five

Yeah, I usually use awk or perl for this (depends on my mood). Just find it annoying that the unix program whose stated purpose is to number lines is basically dead to me.

cat -n is discussed in Pike and Kernighan's paper on Unix style: http://harmful.cat-v.org/cat-v/

You don’t need stdio.h to use puts.

I was in the process of making something similar before I found what I wanted – rlwrap. It provides readline line editing capabilities to command line applications that don’t support them, such as netcat.

Or, more importantly, sqlplus (for those of us that need to work with Oracle).

For that I use gqlplus.

http://gqlplus.sourceforge.net/

For those who want to learn how to write their own UNIX tools, and specifically, how to write tools that work well with other UNIX tools, such as the shell and friends, this article may help - Developing a Linux command-line utility: http://www.ibm.com/developerworks/linux/library/l-clutil/

I recently discovered 'bc'. It stands for 'basic calculator' or more precisely from the man page - 'arbitrary precision arithmetic language'. It is all but a basic calculator, with better floating point precision capabilities than Java/Python.

see also: expr

Hi guys,

Kristof here, creator of this particular list. First, thanks for the kind words! :)

I'll maybe add more tools when I have some time to make more screenshots. It's surprisingly more time consuming than it seems :)

Also, I'm a bit of two minds with the whole list -- many people think that some of these tools are already not "obscure" enough, while others suggest adding even more trivial ones like ifconfig or grep. I'll have to think about this a bit :)

Once again, thanks for your feedback!

KKovacs

htop, tmux/screen, xargs, vim/emacs, rsync, rtorrent, ack are far common to everyone I know (direct and indirectly) that uses the command line.

Yep, lots of them are known to some subset of people. However, if people find one or two really useful ones that they didn't know about previously, the list has done its job.

For me, the major winner was dstat. Did not know about that tool - I've been struggling with the output of iostat. I'll also be experimenting with tmux as a screen replacement/alternative.

Yeah, most of these weren't anything new, but I hadn't heard about ack. Very useful.

+1 for tmux. It's an amazing improvement to screen.

How so? As far as I can tell, the only benefit it has over screen is faster vertically split windows.

Vertical splitting, server/client architecture (shared process), non-crap codebase if you want to dig into it, BSD licensed instead of GPL. I also have no charset/encoding issues with tmux and it doesn't eat my scrollback.

I'm sure the latter points can be remedied by config file settings for screen, but it's nice to use software that just works once in a while.

Vertical splitting screen has. Software architecture, perceived code quality, and BSD license are not features.

I'm glad that there is competition in this space, but you tmux evangelists need to realize that there is a lot of work that needs to be done before tmux is really "an amazing improvement to screen."

> Vertical splitting screen has.

Is vertical split in mainline screen now, or do you still have to patch it?

I use both tmux and screen but prefer tmux. Here are some (non-ideological) reasons:

http://news.ycombinator.com/item?id=1945637

The version available via git from savannah.gnu.org has the old vertical split patch plus a number of more recent scrolling performance fixes. Thanks for your list of non-ideological reasons, better notifications sound pretty compelling. I've been meaning to hack growl support into the OSX screen for a while now...

It's not in the version of screen that ships with OSX. The macports version of screen supposedly supports vertical splits with the +vertical_split variant, but that didn't work for me last time I tried. And that's how I discovered tmux.

It's not any one thing, but it's a lot of small things. When I last looked a couple of years ago, vertical split wasn't in screen mainline and so it was non-starter for me. Check out the list of tmux commands: it's a very powerful program that's continuously being enhanced.

Dunno if screen has this, but in tmux IIRC, you can have two different tmux servers running in totally different processes, and transfer a window from one server to another. That's pretty cool, imho. The graphical equivalent would be running two VNC servers, and moving an xterm window from one to the other.

Another cool thing: pipe the entire contents of a window in realtime to an arbitrary program.

sudo apt-get install ack

got me:

ack - Kanji code converter

On debian (and derivatives) the package is called "ack-grep"

I always install it by downloading standalone version from http://betterthangrep.com/ There is a command on the homepage you can copy paste to shell and execute.

Ack is simply too often used and I don't want to type ack-grep all the time.

  sudo dpkg-divert  --local --divert /usr/bin/ack --rename --add /usr/bin/ack-grep

That way you still get the benefits of the packaging system and you don't have to alias or create a /usr/local/bin symlink.

"man alias" ?

    <nit>
    $ man alias
    No manual entry for alias
    $ man bash

    ALIASES
           Aliases allow a string to be substituted for a word when it is used as the first word of a simple command.
    </nit>

alias is not a standalone, it's part of the shell (in bash at least).

Sometimes there are aliases for bash built-ins that will take you to the manpage. For me, 'man alias' brings up:

  BASH_BUILTINS(1)

Though, I'm using zsh which may be layering something in there.

Try "help alias"

Alias (though not 'man alias' :) ) could work. But I prefer my way (on server) because I have one thing less to worry about, namely putting 'alias ack=ack-grep' to .bash_aliases. Yeah, it's quick and dirty I guess...

`ncdu` is a godsend. It's like `du` but usable for troubleshooting.

None of the following from this list are obscure: screen, vim, rsync, xargs, curl

I'm tempted to put ack in that list.

Even though these aren't really obscure, but they're worth mentioning as some of the best unix "tools" I've seen:

    Irssi - irc client
    Mutt  - email client

These are mentioned mostly because he included the newsbeuter rss client (which is amazing, I highly recommend it).

Here's a few of my favourites.

agrep - approximate grep based on edit distance).

GNU Global - source tagging system that integrates with the shell (less -tfunc displays the function given in the shell).

xmllint - xml validator, pretty printer and schema validator.

Here's one that I find does not get a lot of mention but is quite useful when you need it.

"tee"

Guake/yakuake - A top-down terminal based on the tilda terminal from the game Quake.

Redshift/f.lux - Redshift adjusts the color temperature of your screen according to your surroundings.

Great post! It introduced me to a few tools I wasn't aware of previously. The main OS on my laptop is Ubuntu so this is very appreciated.

My obscure favorites:

yafc: The best command-line FTP client that nobody's ever heard of. Local caching, tab completion, bookmarking, SFTP, and other generally awesome stuff.

clex: Full-screen file manager for command-line junkies. Configurable directory display, smart name completion, enhances the command line without seeking to replace it.

How does yafc compare to lftp, and how does clex compare to midnight commander?

I switched to yafc from lftp years ago and never looked back, so I can't really compare it to newer versions of lftp. IIRC the biggest wins at the time were context-sensitive tab completion, including support for remote paths, and automatic bookmarking with saved passwords (optional, of course). FTP sucks, yafc makes it tolerable.

clex's approach to file management is by complementing the command line; you can generally use it as a pure CLI and it stays out of your way unless until you need to find or select a path. Every action you perform remains on the command line, clex merely makes it easier to quickly build commands. Compared to clex, Midnight Commander (like Norton Commander before it) feels like an attempt at a graphical file manager for the terminal, where the command line is almost an afterthought.

+1 on comparing yafc with lftp

I've discovered a lot of new apps from the post and comments. Give ncmpcpp, an excellent ncurses mpd client, a try and say goodbye to GUI music players.

http://unkart.ovh.org/ncmpcpp/screenshots.php

Also in line with tmux, checkout teamocil and tmuxinator on github.

Libcaca and its various programs:

http://caca.zoy.org/wiki/libcaca

Also, there is neercs, a terminal multiplexer that uses libcaca: http://caca.zoy.org/wiki/neercs

Most of these aren't obscure, but it was a good read, and I didn't know about slurm.

So over on one of the test machines I apt-get install cowsay ... One of the other devs here is going go get a surprise next time he logs into one of the webservers ... ;)

Its amusing to see nethack and cowsay listed as a full-fledged unix 'tool's.

Yeah, no 'sl' though... :)

______________________________

< cowsay is the epitome of a unix tool. >

------------------------------------------------

      \                _
       \              (_)
        \   ^__^       / \
         \  (OO)\_____/_\ \
            (__)\       ) /
                ||----w ((
                ||     ||>>

mytop - a `top` clone for MySQL

You're probably better off with innotop http://sourceforge.net/projects/innotop/files/

The new home for innotop is http://code.google.com/p/innotop/, and for anyone looking for other amazing FOSS tools for MySQL (especially Percona builds with XtraDB)- http://www.percona.com/docs/wiki/percona-xtrabackup:start http://www.maatkit.org/

You can do some amazing things with MySQL, Postgres, and Memcached with some of the maatkit tools.

Ah yep, thanks for that; just Googled it in a hurry and pasted in the first link that came up. Can't agree enough on the Maatkit tools, and I'm also in the process of moving a production database from MySQL to Percona after hearing such good things about it. Xtrabackup for consistent backups of innodb/xtradb live systems + incremental (delta) backups are also very useful.

Depending on the version of MySQL that you're upgrading from and if you're upgrading in place vs starting fresh on a new system you won't be able to use Xtrabackup to handle the mysql.proc table due to backwards incompatibilities. I had an awful go of this upgrading from MySQL 5.0.51 to Percona 5.1.56 and wound up having to massage the stored procs and functions in one by one. Once we were migrated over though the xtrabackup tool is great for incremental backups.

My operating system doesn't have a package manager ;)

Get a better operating system, or install Cygwin

Apparently he hasn't discovered autojump yet :-D

autojump looks pretty cool. I assume your name and that of the github implementation isn't a coincidence?

It's not :)

lftp is cool, though I wish FTP would die (as telnet sensibly did over a decade ago) so I will no longer need it.

lftp is an old favorite of mine that i've used many a time. Its built-in shell with history, support for sftp as well as various flavours of ftp, bookmarks are features I've made use of.

Is it worth mentioning pushd/popd here?

And all these are replaced by Emacs :p

Actually, speaking seriously (for a change), I would like to see an agrep for Emacs (approximate search).

cowsay is used on craiglist's 404 page.

why nmap is not listed?

Another cool one is xmlstarlet. It's like grep and sed and some on top, but for XML files.

Especially, xmlstarlet sel lets you select whatever data you want from XML files, being a command-line XSL templates generator.

Don't know anything else useful for XML, so it fills a vacuum.

xmllint is another hugely useful xml tool.

Deleted comment

Looks like he's running them on OSX which means they're likely to work on BSDs too.

Reply on news.ycombinator.com