Hacker News Reader

Comment by nextos | Hacker News Reader

GNU Parallel is the first thing I usually install on top of a standard Unix userland. It's almost a superset of xargs, with many interesting features. It depends on perl, though.

htop is also pretty much a great replacement for top. And ripgrep a great replacement for the find | xargs grep pattern.

Aside from that, I'm pretty content with the Unix userland. It's remarkable how well tools have aged, thanks to being composable: doing one thing and communicating via plain text.

I'm less happy with the modern CLI-ncurses userland. Tools like e.g. mutt are little silos and don't compose that well. I've migrated to emacs, where the userland is much more composable.

View on HN

I don't feel the plain text portion has aged well at all in regards to composability. It leads to a lot of headache as the complexity of the task grows because of the in-band signaling and lack of universal format. I think it is high time the standard unix tools were replaced with modern equivalents that had a consistent naming scheme and pipelined typed object data instead of plain text.

I think you should check out Powershell.

This made me chuckle, because it's incredibly accurate. I started with bash and am still naturally more comfortable there for general purpose work. PowerShell is annoyingly verbose sometimes and has its own WTF moments, but there are waaay fewer surprises when working with complex scripts and variables.

It’s not included anywhere, but PowerShell is open source and you can install it on *nix now! Obviously the things that integrate with Windows aren’t there, but the object-oriented pipelining sure is.

I believe the AD and Azure tools work, and the other ones needed for remotely managing Windows Server, although I could be wrong.

Is that a joke? The syntax is worse than PHP and it doesn't run on Linux

It runs on linux:

https://azure.microsoft.com/en-ca/blog/powershell-is-open-so...

https://docs.microsoft.com/en-us/powershell/scripting/setup/...

I'm actually a fan of Powershell, but unix people take it as a personal insult if you try and tell them their 70s-era tooling is inferior in some way to something designed with 30 years of hindsight.

Used PowerShell to code a small script and simply making md5 of a string is enough to make you feel nuts.

The UNIX style.

  echo -n 'string' | md5

Compared to that obvious command, this is utter madness. You can't write a single thing without googling.

  $string = "string"
  $md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
  $utf8 = new-object -TypeName System.Text.UTF8Encoding
  $hash = [System.BitConverter]::ToString($md5.ComputeHash($utf8.GetBytes($string)))
  $hash = $hash.ToLower() -replace '-', ''

No shit the Bash on Windows is a godsend.

Of course, because someone wrote a md5 executable for you to call.

You can do the same on PowerShell and write a pipeline just the same way.

You're right that a shell alone can't calculate md5 but a separate md5 binary does it for you, but the question still stands when the common answer on the internet seems to be to write that cryptic code as that's probably the easiest way provided on PowerShell.

The best alternative I could find is some community maintained PowerShell extension (with just 177 GitHub stars now), which is far better but the lack of interest in making PowerShell act more straightforward is weird.

"string" | Get-Hash -Algorithm MD5

(https://github.com/Pscx/Pscx)

Still feels funny to read the GGP's comment.

> 70s-era tooling is inferior in some way to something designed with 30 years of hindsight

Still, all that stops PS from being a better shell in linuxland is some people writing PS-equivalents of all the small executables that ship with your linux distribution.

Powershell is based on the REPL ideas of Lisp Machines and Smalltalk.

- structured data

- ability to call any public function across dynamic libraries, COM objects and .NET frameworks

- Powershell modules can also be called as plain .NET code

Which UNIX shell provides this?

The easiest way is to launch md5sum from Powershell. It's a shell, at the end of the day...

A while ago I was trying to copy a bunch of files and couldn't believe the powershell abominations I found: https://techblog.dorogin.com/powershell-how-to-recursively-c...

    $exclude = @("main.js")
    $excludeMatch = @("app")
    Get-ChildItem -Path $from -Recurse -Exclude $exclude | 
              where { $excludeMatch -eq $null -or     $_.FullName.Replace($from, "") -notmatch $excludeMatch } |
              Copy-Item -Destination {
                if ($_.PSIsContainer) {
                  Join-Path $to         $_.Parent.FullName.Substring($from.length)
                } else {
                  Join-Path $to     $_.FullName.Substring($from.length)
                }
               } -Force -Exclude $exclude

All to do:

    find sourceFolder \! -path */app/* | xargs cp -t destFolder

Screwed up the formatting a bit in the first example. I'm not positive the second works perfectly but I know which one I'd rather debug.

  rsync -a --exclude=app/ --exclude=main.js sourceFolder/ destFolder/

Thank you, thank you unix tools.

First of all, the two examples you present do not do the same thing. The Powershell version preserves paths and excludes more patterns than your find | xargs example. The right way to do this is to just use robocopy.

This example has targeted a very specific deficiency in the way Copy-Item works. I could similarly point out that the following Powershell command would be much more difficult in standard unix tooling:

  Get-WMIObject -Computer $remote -Class Win32_Product | Where-Object { $_.name -like "7-Zip*" } | select Version

Which gets the version(s) of 7-Zip installed on a remote machine.

If Unix pipes gained support for exchanging some kind of out-of-band signalling messages, then CLI apps could tag their output as being in a particular format, or even the two ends of a pipe could negotiate about what format to use. If sending/receiving out-of-band messages was by some new API, then it could be done in a backwards compatible way. (e.g. if other end starts reading/writing/selecting/polling/etc without trying to send/receive a control message first, then the send/receive control message API returns some error code "other end doesn't support control messages")

I have suggested this before: https://news.ycombinator.com/item?id=14675847

(But I don't really care enough about the idea to try to implement it... it would need kernel changes plus enhancements to the user space tools to use it... but, hypothetically, if PTYs got this support as well as pipes, your CLI tool could mark its output as 'text/html', and then your terminal could embed a web browser right in the middle of your terminal window to display it.)

>your CLI tool could mark its output as 'text/html', and then your terminal could embed a web browser right in the middle of your terminal window to display it.

Ha ha, I had dreamed up something like this in one of my wilder imaginings, a while ago: A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline :) Don't actually know if the idea makes sense or would be useful.

http://www.xml.com/pub/2000/06/07/xmlterm/

> A command-line shell at which you can type pipelines, involving some regular CLI commands, but also GUI commands as components, and when the pipeline is run, those GUIs will pop up in the middle of the pipeline, allow you to interact with them, and then any data output from them will go to the next component in the pipeline

There seems to be some precedent for this sort of thing. For example, DVTM can invoke a text editor as a "filter", where the editor UI is drawn on stderr and result saved to stdout.

> where the editor UI is drawn on stderr and result saved to stdout

I did some experimenting recently and found you can open a new /dev/tty file descriptor and tell curses to use this, the rest of the application can continue reading stdin and writing stdout as normal.

It does. For example, you could pipe an output of diff and a GUI diff viewer starts up and it's a whole lot easier to see or merge the context and even you may be able to launch a 'system default' app for that instead of a predefined app.

And another apparent one is image viewer or editor.

Sure it makes sense. Just put `gvim /dev/stdin` in the pipeline you desire and write to stdout when you're done. You can add to your configuration to remap `ZZ` (which usually saves the file and exits) to `:wq! /dev/stdout` when stdout is not a terminal. I'm going to do that when I get back to my computer.

That sounds like a cool use of the concept. Will try it out. Thanks.

I think you could use rofi for that, just insert it to the pipes and filter accordingly.

Thanks, had not heard of rofi. Just took a look. Seems like a useful tool.

I am not sure how this would work. rsh/ssh may be involved. I wouldn't even know how to express:

ssh carthoris ls /mnt/media/Movies | grep Spider

(this is just an example). Note that in this example, we have two processes running on two different machines. Indeed, the OSs and systems on these machines may be, um... different. Indeed, I routinely include "cloud" machines in pipelines. Indeed, with ssh, the -Y (or -X) option can introduce a GUI to a part of the command.

I have wished that shar was part of SUS. Also, I find that "exodus" is useful (across Linux anyway -- the systems have to be "reasonably" homogenous). https://github.com/intoli/exodus

> I am not sure how this would work. rsh/ssh may be involved.

In order for this to work over SSH, the SSH client and server would need to be enhanced to exchange this data, and also an SSH protocol extension would need to be defined to convey it across the network.

One might define IOCTLs that work on pipes and PTYs to send/receive control messages. So sshd would read control messages from the PTY and pass them over the network, and the SSH client would receive them and then pass them on to its own stdout using the same ICOTLs. (Alternatively, one might expand the existing control message support that recvmsg/sendmsg supply on sockets to work on pipes and ptys as well.)

Any program supporting such an out-of-band signalling mechanism would have to gracefully degrade when it is absent. If your SSH client or server, or some program in your pipeline, or your terminal emulator, etc, doesn't support them, just fall back on the same mechanisms used today to determine output/input formats.

(rsh is such a deprecated protocol, there would be no point in trying to extend it to support something like this.)

This would be excellent. Finally we get to view images on remote side (except by the iTerm hack) and view diff in a local GUI in the middle of a session?

Unix domain sockets. Bidirectional, and you can use datagrams to bypass the effort of framing data yourself.

Plain text was chosen to interface the various programs of Unix OSes because it's the least common denominator all languages share. It also forces all tools to be composable with each other. You can take output text that was obviously not formatted for easy consumption by another program and still use all the information it outputs for input into another program. Programs that were only thought to have users handling its input and output (ncurses apps) can also be forced to be used by programs through things like the expect TCL program or Ruby's expect library.

If programs used typed data, they'd still need the option to output text to present results in a format the user can understand. To do this, a negotiation protocol could be established, like skissane said. This, in my opinion, is BAD, because then there's the possibility or probability that they'll be differences in the information conveyed in the different formats.

I believe that the use of plain text as the universal format of communication between programs is one of the greatest design decisions for Unix and CLI.

How hard is it to come up with a object format (more like a data structure format since I wouldn't want logic/code being passed around) and then come up with a standard text serializer for it? Not that hard, in my opinion.

You'd standardize it once via an RFC and you'd be done with it.

I don't think your problem is as big as you say it is.

The real problem is that this pile of code we already have kind of works and it's already making trillions for its users. Changing the whole ecosystems would cost millions and millions, for only a very long term and unclear benefit.

In other words, worse is better.

It's not about backwards compatibility. It's about the fact that text is what we read as humans and if commands parse the same format there is only one output format to implement.

I rarely want text as output format, I want structured data that I can explore in a structured fashion.

As oblio said, you can come up with a standard conversion of structured data to text. The other way round you need write a parser for every textual output format, and typically people come up with fragile ad-hoc parsers that don't deal with edge cases properly.

I don't believe that. Text for human consumption, in a well designed UI (and I mean even a CLI one!), should be different from text for machine consumption. Human consumption generally optimizes for characteristics almost diametrically opposed from machine consumption.

Of course, who am I kidding, in real life we have some sort of crappy text interface which is half-baked both for humans and for machines. But we've been using it for almost half a century and it's too widespread to redo, so there we are, plowing through it daily.

Let's imagine the OS thought so, too, and had programs require implementation of both UIs, one for machines which is hard to look through by humans, and one for humans which is automatically presented in GUI form, meaning its hard to control and it's hard to parse the information it's presenting in a bitmap window (IOW an unautomatable interface). Now, I see 2 reasons to prefer the scenario we have now with unix and text based communication:

1) We don't need to depend on each individual program's programmer to present every control and information consistently between the 2 interfaces.

2) Automation matches normal, manual use. Just put what you normally do on the command line in a file and you're done. There's no need to look through documentation on how to do what you so frequently do, only in a manner that you rarely do.

You might look into PowerShell

I think we have enough formats that fit the requirement of serializing data structures, no need for a new one (xkcd ref goes here). You still need to be able to tell the other tool what to do with that data though. In essence instead of a series of greps and seds and awk you need a bunch of options for the next tool in the chain to tell it how to treat your serialized object. That's merely shifting the complexity around.

Also there is no need really to change anything (as in, breaking existing scripts). Selecting a different output format can simply be a command line option. Many tools already offer Json, XML or CSV output. But since development of those tools is so decentralized you'd be hard pressed getting them all to agree on one. But theoretically you can pick any tool you want right now, add --json support and submit a patch.

> You'd standardize it once via an RFC and you'd be done with it.

If you think something like this is easy much less "you standardize it once and you're done", then you are only cheating yourself out of an essential life lesson.

You've misunderstood me. It's not that it's hard; it's that, however nicely you do it, the result sucks.

Are TUIs like htop, tmux, vim, emacs, less, etc. going to be impossible now, or will you do the negotiation protocol? Both options suck.

When programs have both normal output and errors intermixed are the objects going to be intermixed in the output that's presented to the user? For example, if you do a `find /etc/pacman.d`, instead of:

    /etc/pacman.d
    /etc/pacman.d/mirrorlist.pacnew
    /etc/pacman.d/gnupg
    /etc/pacman.d/gnupg/trustdb.gpg
    /etc/pacman.d/gnupg/crls.d
    find: ‘/etc/pacman.d/gnupg/crls.d’: Permission denied
    /etc/pacman.d/gnupg/.gpg-v21-migrated
    /etc/pacman.d/gnupg/private-keys-v1.d
    find: ‘/etc/pacman.d/gnupg/private-keys-v1.d’: Permission denied
    /etc/pacman.d/gnupg/tofu.db
    /etc/pacman.d/gnupg/openpgp-revocs.d
    find: ‘/etc/pacman.d/gnupg/openpgp-revocs.d’: Permission denied
    /etc/pacman.d/gnupg/gpg.conf
    /etc/pacman.d/gnupg/secring.gpg
    /etc/pacman.d/gnupg/pubring.gpg~
    /etc/pacman.d/gnupg/pubring.gpg
    /etc/pacman.d/mirrorlist

will you have:

    [
      "/etc/pacman.d",
      "/etc/pacman.d/mirrorlist.pacnew",
      "/etc/pacman.d/gnupg",
      "/etc/pacman.d/gnupg/trustdb.gpg",
      "/etc/pacman.d/gnupg/crls.d",
    [
      {
        type: "Permission denied",
        path: "/etc/pacman.d/gnupg/crls.d"
      },
      "/etc/pacman.d/gnupg/.gpg-v21-migrated",
      "/etc/pacman.d/gnupg/private-keys-v1.d",
      {
        type: "Permission denied",
        path: "/etc/pacman.d/gnupg/private-keys-v1.d"
      },
      "/etc/pacman.d/gnupg/tofu.db",
      "/etc/pacman.d/gnupg/openpgp-revocs.d",
      {
        type: "Permission denied",
        path: "/etc/pacman.d/gnupg/openpgp-revocs.d"
      },
      "/etc/pacman.d/gnupg/gpg.conf",
      "/etc/pacman.d/gnupg/secring.gpg",
      "/etc/pacman.d/gnupg/pubring.gpg~",
      "/etc/pacman.d/gnupg/pubring.gpg",
      "/etc/pacman.d/mirrorlist",
    ]
    ]

That's sometimes going to lead to a syntax error.

You could have every function incorporate their errors into their normal output, but that means giving up a standardized way of working with errors and warnings. I don't know if you know this, but when you do substitution or piping, by default, only stdout is used. That means that we you do piping, the programs in the pipelines normally do not see the errors in their inputs, and the errors of the multiple concurrently running programs are shown to you intermixed while the pipeline is working. That's a friggin' incredible effect that came from simple design, but when each one would output objects, you'll could get syntax errors or a completely different object like what happened in the above example. You could say, "well, only make stdout an object and let stderr be text," but the fact that they're both the same type means that you can work with the error or only with your errors in pipelines and other shell constructions. For example, `find /etc 2>&1 >/dev/null` will output the directories in /etc you can't read for whatever reason. You might want to pipe that to `xargs chmod` (for whatever reason) after preparing the output to only include the paths.

Right now, programs can strike a good balance in presenting its output in a format that is both readable to humans and other programs. By forcing their output to be structured as objects, and not giving them the option of presenting 2 formats (because we don't want that either), you're removing their ability to present the output in a manner that is readable to humans.

Take for example, rspec's output (a unit testing framework):

    $ rspec spec/calculator_spec.rb
    F

    Failures:

      1) Calculator#add returns the sum of its arguments
         Failure/Error: expect(Calculator.new.add(1, 2)).to eq(3)

           expected: 3
                got: nil

           (compared using ==)
         # ./spec/calcalator_spec.rb:6:in `block (3 levels) in <top (required)>'

    Finished in 0.00131 seconds (files took 0.10968 seconds to load)
    1 example, 1 failure

    Failed examples:

    rspec ./spec/calcalator_spec.rb:5 # Calculator#add returns the sum of its arguments

Mind you, that's full of colors in the terminal. It's output that easy to read with the eye and parse with a bit of awk. Can you imagine that being output as a JSON with a generic pretty printer? How will it compare when reading with the eye?

The main thing is, though, that, in the question of what the universal format of communication between programs written in different languages should be, text is the simpler, more natural choice over objects. Take note, I don't mean easier. The fact that it's easier is merely coincidence. Simplicity leads to good design because it means less arbitrary choices to make. Less controversial choices to make. Choosing objects leads to more questions: What should the primary types be? Should arrays/lists allow multiple types of elements? Floating types or decimals? Precision restriction on the decimals? Should integers and numbers that allow fractional parts be the same type or different? Should we have a null type? Should we have a date primary type? What about a time primary type? What about a datetime primary type? Whatever answers you give, there will always be groups of people that will dislike them. When you chose text, the only question is really, what encoding? Utf-8. done. Natural, simple design is what we want to be the foundation that myriads of programs and languages can base themselves on and depend on.

There's only one way I'd agree with you that structured output would be nice, and that's with mono-language OSes, like a lisp OS or some other OS where all code is in the same language, and there would be no concept of programs or shared / dynamically loaded libraries or such. In an OS like that, every function is a program, and your shell is the language's REPL. This is bliss when the OS is done in your favorite language. The problem with these kinds of OSes is that we don't all like the same languages and so it'd lead to ridiculous situations where we'd translate a new language into the high level language of the OS. That's what we do in the OS known as the web browser and why we're coming up with WebAssembly.

In conclusion, multi-language OSes like those that are Unix based are awesome, and text as the basis of communication in multi-language OSes is awesome. Therefore, text as the basis of communication in Unix is awesome. :)

...As long as you don’t have spaces in file names.

I’d like to see something like cvs used more, where it can handle the edge cases without breaking, but still doesn’t need a translation step

is there any filesystem that does not enable spaces in filenames? I find the idea ridiculous. Would you design a programming language that allowed spaces in its variable names? Because that is the same level of atrocity.

Some languages allow spaces in variable names by using quotation. To some extent Ruby, Kotlin, and SQL come to mind.

Yes, but filenames are not variable names, they're values, and they often break plain text piping

This is not the unix philosophy. Filenames are indeed variable names. The values are the contents of the file.

> Tools like e.g. mutt are little silos and don't compose that well. I've migrated to emacs, where the userland is much more composable.

What do you use for reading email in Emacs?

Notmuch, which is IMHO superb and totally underrated.

I like it a lot due to its clever architecture. It never ever touches your email. It operates on a separate tag database.

Then, it's the task of a backend to translate tag changes into maildir actions before and after syncing email. Keeping tag to actions decoupled from the GUI is extremely clever, because it allows implementing basically any email workflow you can imagine.

For simple workflows, calling a one liner notmuch command is sufficient. You don't really need to implement anything.

Mu4e is an alternative client to Notmuch. Quite similar to Mutt. Gnus is the other big alternative. It's quite old, and complex to configure. Besides, the codebase is overcomplicated as it tries to do email in a news-like fashion. Still, it has lots of great ideas on how to deal with email from many sources. E.g. using predictive scoring.

Thanks, I'm in the process of migrating from Thunderbird to mutt/neomutt and am wondering if emacs might have advantages. I already use Emacs for Org Mode.

I'm also looking for a scriptable, text-based RSS reader: https://community.codeselfstudy.com/t/text-based-rss-atom-fe...

> am wondering if emacs might have advantages

Fresh mu/mu4e user here. The advantages over mutt/neomutt are that your e-mail now becomes the part of the same consistent UX of Emacs, and then every improvement to any of your workflows in Emacs is automatically inherited by your e-mail workflow. This depends on how much you like customizing Emacs and/or writing Emacs Lisp to solve your problems. Some practical examples include:

- org-mu4e & org-notmuch will let you link directly to your e-mail messages from your org-mode files; potentially useful if you're using org-capture to quickly add TODOs and notes.

- it's trivial to add any kind of template responses, template subresponses, etc.

- you can make Emacs automatically do things in response to particular e-mails, or you can compose/send e-mails directly from any elisp code

Emacs is a fully programmable environment with lots of existing software packages and the best interoperability story I've ever seen.

As you're already using Emacs for org-mode, you could take a look at org-feed [1] for text-based RSS. It places all RSS feeds into a normal org file with headings per source and article. As it's Emacs + org, it's naturally scriptable.

[1] https://orgmode.org/worg/org-contrib/org-feed.html

+1. I spent a while trying to persuade Apple Mail to let me receive notifications only for threads I was 'watching'. I never did come up with a sane answer, and finally decided "well, I use Emacs for almost everything else. Might as well see how email pans out..."

With notmuch and mbsync I have the best email setup I ever have. I wish I'd done it years ago.

Can you share your setup?

Sure. It's all available in my dotfiles and .emacs.d repos on GitHub. It's not elegant or pretty, but it has worked well for me so far.

My crontab (https://github.com/NateEag/dotfiles/blob/master/lib/.crontab) runs a check-mail script (https://github.com/NateEag/dotfiles/blob/master/bin/check-ma...) every five minutes.

The check-mail script is just a wrapper around mbsync (http://isync.sourceforge.net) that invokes 'notmuch new' after running, so notmuch can index and process new messages.

My notmuch post-new hook does a bunch of tagging for me, so I have to actually look at as little email as feasible, but the main thing it sounds like you're interested in is the notification setup:

https://github.com/NateEag/dotfiles/blob/858cabd9436d377f8fc...

With that, I can batch-process email a few times a day, while staying responsive to any discussions I actually want to be interrupted for.

The missing piece is reasonable logic for knowing when I should be notified of new threads. I'm currently in a job where people don't expect insta-responses to email, thank God, but I've been in ones where they do and I'd have to think about how to handle that more.

For reading and writing email, I just use notmuch's Emacs modes. You can see most of my interesting config here: https://github.com/NateEag/.emacs.d/blob/75d117befeaaab37cc7...

My one annoyance when reading email is that large inline images aren't auto-resized to fit. They should be, but the Emacs build I use doesn't have ImageMagick support compiled in.

My custom.el probably has some notmuch settings too.

That's the highlights. Hope it was helpful.

Would love to learn more about that, do you have a blog post or any pointers please?

I'm not sure if HN sends response notifications or if people depend on them, but just in case, I answered the sibling question here:

https://news.ycombinator.com/item?id=17888628

I had issues using gnu parallel in the past because of --will-cite. I ended up not using it.

There is a Rust clone[0], that works seamlessly for my -- arguably simple -- use cases.

[0] https://github.com/mmstick/parallel

What problems arose?

I usually just run --citation once on a new system, and never see the notice again, and I have never had the notice cause any problems as it only shows if the output is to the screen.

Is the Perl dependency actually relevant? I've never seen a Linux distro that doesn't install Perl 5 by default, and it's also installed by default on macOS and OpenBSD. The other BSDs all have Perl in their ports, and you almost always end up installing it anyway due to the sheer amount of stuff that depends on it.

Why not grep -r or rgrep?

Reply on news.ycombinator.com