What modern utilities should be a standard part of a modern unixy distro? | Hacker News Reader

What modern utilities should be a standard part of a modern unixy distro?

exple.tive.org

50 pointszdw56 comments

I may be a little cynical but when I read the words "modern utilities" my first associations are: too much output coloring (typically implemented by relying on several libraries), no proper documentation, output formats that are not machine-friendly, overuse of TUI interfaces where simple writing to stdout would integrate better with the rest of the world, complicated configuration files (typically requiring even more libraries), etc.

Honestly - the only thing i really miss among the default utilities is the hypothetical command "sortby" (and maybe it's cousin "uniqby") which could be used, for example, like this:

    ... | sortby cut -d: -f2 | ...

    ... | sortby tr [:upper:] [:lower:] | ...

I know there are ways to implement practically any type of sorting by using flags to sort and a little bit of {pre,post}processing but having a "sortby" would be so much more elegant. You would have the full power of awk/sed/whatever integrated seamlessly with "sort".

>> output formats that are not machine-friendly

As a long time unix sysadmin, this is the thing that always rears its ugly head.

> output formats that are not machine-friendly

Not a new problem. It was always like that.

Not sure if you know about the `jc` tool. It converts the plain text output of a subset of UNIX commands to JSON. It's helpful, I've used it.

> complicated configuration files (typically requiring even more libraries)

Not sure I can stand behind the "more libraries" argument, some formats like TOML are stable and predictable and having one more system-wide library is worth it to have them accessible.

On config files: if the tool allows you to specify 100% of what you need on the CLI then I usually make an alias and I'm done.

I too dislike tools that mandate a config file though. I view the tool authors as not very sympathetic to their users in this case.

Yeah I shouldn't have included that bit about libraries - if there is a config file, it doesn't matter how you parse it. My preferred ways of configuring in order of preference would be: having a config.h (for a few, highly customized programs, like my terminal, browser, etc.), environment variables (has the advantage of applying to subprocesses, which can be useful for some types of programs) and flags. Aliases with flags can be lost in a lot of contexts, but you can always place wrapper scripts in PATH which just exec the real command.

As a side note - the wrapper script approach is really powerful for fixing programs that you don't have the time to patch. In particular, if a TUI program doesn't clear the terminal after exiting, you can put it between "tput smcup" and "tput rmcup".

> I may be a little cynical but when I read the words "modern utilities" my first associations are: (...)

I think this comment is too short-sighted and naive. In the end at best it's an apples-to-oranges comparison between what you personally like and are used to with a cherry-picked set of tools and what you personally don't like and have an axe to grind with a cherry-picked set of strawmen.

There are plenty of decades-old tools that provide colorful TUI interfaces that don't integrate with standard tools and make them hard to reuse with UNIX tooling. There are plenty of excellent non-standard tools that are quite recent and raise the bar in UX. This is not an old-vs-new comparison. This is reads like a rant directed at strawmen.

Let's face the facts: JSON is excellent as a data interchange format, and YAML is a far improvement in expressiveness over INI-type formats. Colors are important to make things human-readable, and TUI is a UX godsend in applications that involve view-heavy data presentation and ensure complex commands can be a key press away.

> sortby

Something like this? (But of course made more robust. I'm not really into shell scripting.)

    t=`mktemp sortby.XXXX`
    tee $t | "$@" | awk '{ key = $0; getline <"'$t'"; print key "#" $0; }' |
        sort -t'#' -k1 |
        cut -d'#' -f2-
    rm $t

As few things as possible. People can install software themselves.

IMHO, linux distributions are an odd thing in the software world. It's basically people taking it upon themselves to take other software and repackage it for their distribution. That made some sense 30 years ago when you had to install linux from a CD (or floppies) and people necessarily had to package things up to fit on those CDs. But these days, that's much less of a thing.

The recent trends of distributions supporting things like docker, flatpak, snap, etc. enables software producers to package things up once and have them work on many distributions. So, there's less need to rely on package managers and their maintainers to figure out how to package up other people's software. Of course some of these things aren't perfect. But there's a large ecosystem of software that you can get this way that no longer requires packaging up. And even things like docker you can actually get via snap.

So, the fewer packages in a linux distribution, the better. Less that can go wrong, less space used on disk, less work to build and package things up, less hassle to update things, etc. All good things. Add the rest yourself as needed.

> and repackage it for their distribution

There's no "re" in there. They decide how to package it for their distribution. The idea that you can take something non-trivial and just install it without issues across multiple systems is flawed. There's a lot of work involved in getting things to work together correctly. (If you don't think that's true, see how many of the current 4k broken packages can you fix in nixos https://hydra.nixos.org/jobset/nixpkgs/trunk )

Upstream software often has assumptions about how things work in your system. It's often not correct.

> But there's a large ecosystem of software that you can get this way that no longer requires packaging up.

Making non-trivial things work correctly in docker, snap, or flatpak is packaging. You effectively still target a specific distribution with a specific setup. It takes work, it takes maintenance, and it likely will degrade over time as your host changes as well.

> The idea that you can take something non-trivial and just install it without issues across multiple systems is flawed

Why? If anything, the fact that you can't is the real flaw

Some examples I fixed just this week:

- feature name changed, feature autodetection fails, package doesn't build anymore

- new version of the package requires a new Apple SDK which is not used by default in the system yet

- SDL gets updated which bumps the minimal required Apple SDK version. The package depending on SDL declares lower minimal SDK which fails at linking

- package requires a specific flag when building with latest GCC, but that flag doesn't exist for clang, so it fails the build

- new C++ standard adds std::bind which breaks unscoped calls to bind. Upstream didn't force an older standard, so build breaks on clang update.

- Makefile makes assumptions about HOME being writable during a build.

- build system assumes where the completion files will go

And those are still the trivial problems. Some of those are fixable upstream, some will be "don't care" for the author or the package is already abandoned. Some exist because the author doesn't have particular hardware to test on. It's on the distributions to make it work for you regardless.

Then the much better cause to fight for is to make this packaging and installing not resemble splattering a bucket of paint on the wall (installing stuff in /usr, /bin, /etc and so forth).

The Linux distros did a good job packaging stuff but in the end it's all a very wasted effort that should have gone to unifying the way stuff is installed so we could all be more productive after.

I can dream, I know. Some people will likely pull out a gun if you say that /usr/bin should exist only as a collection of symlinks.

I'm not arguing that packaging shouldn't be done but that it should be done, and is in fact commonly done by upstream developers. What things like docker, snap and flatpak allow is packaging things up in a distribution independent way. Which means it can be done by the developers of the software directly. Of course the landscape is a bit fragmented and you could also see this as yet another package variety to support and there is the usual bit of tribalism around which to use and not to use. But overall, it means a lot of software is now available that you would in the past have had to install via distribution specific packages.

> It's basically people taking it upon themselves to take other software and repackage it for their distribution. That made some sense 30 years ago when you had to install linux from a CD (or floppies) and people necessarily had to package things up to fit on those CDs.

That might have been the original reason, but another has emerged in the meantime: distro maintainers are simultaneously unattached enough to the software to make decisions to make for the users instead of the authors and technical enough to actually implement those decisions[1]. OK, no, not really, they also very much implement their own vision of what users should want, but there are (still) enough distros that a user can choose one that fits them once and not be forced to pick or abandon individual programs. In that respect even Flatpak et al make things worse—unsurprisingly, given that Flatpak (AFAIK) is GNOME people saying they should make distro-style decisions, and nobody else should be able to. (Of course, in other respects—such as being able to install arbitrary things from the ’net—Flatpak et al do make things better.)

https://drewdevault.com/2021/09/27/Let-distros-do-their-job....

Right. Build the environment from scratch and I bet you will immediately find newfound adoration for the work distro makers do.

That kind of utterly ignorance driven complaints are nothing new

While I like the idea of flatpak and docker for isolation the ability to include libraries make developers lazy. Now libraries with vulnerabilities are up to the developer to update because you have several versions on the same system and they're no longer forced to make their package work with the latest patched version.

Great in theory, not so much in practice.

You have completely misunderstood the purpose of distributions. The packaging of software for easy installation is certainly a benefit, but by FAR the primary purpose of (enterprise) distributions is to ensure all those packages are tested to work with each other to provide a usable and stable platform to build on. You can’t just have packages changing to random versions whenever a developer feels like it, as it would break other things using the same package. Easy examples would be conflicts in Python 2 vs 3 scripts, different versions of glibc, etc.

Containers are just an admission that developers are so resistant to working with others that now we have to essentially install a full custom Linux distro for every app, and at that point you’re relying on that same developer to also provide updates to not only their own software, but all dependencies included in the binary blob.

As a long time user of utils-linux, I can say this: purge!

We need a big, big, huge purge of this ancient decrepit nonsense. We don't need blkid, lsblk, fdisk, wipefs, parted, df and a bunch of other lopsided semi-functional horrors to ask questions about storage. Just one program that works.

We need integration of MD, DM, LVM and possibly stuff like Btrfs volumes. ZFS was close to it. And commands associated with these utilities need to make sense in the way they are named and associated with each other. The way it needs to work is something like:

    $ storage volume-manager physical-volumes list

(This doesn't mean you have to type it out in full, useful abbreviations would still apply, but there must be a way to spell it out so that non-initiated may understand the purpose and the reason of the "spell")

Instead of guessing whether you should run pvdisplay or btrfs subvolume list

Networking seemed like it went in this direction sometime ago with iproute2, but it's still not enough. Stuff like nmcli or all kinds of nslookup, ss and so on need to be brought under one roof, generic enough to allow extensions and implementations by different concrete programs.

Today, just in order to find the configuration of one's network interfaces one must scout different places on their filesystem and try to figure out what utility is reading those and whether it's able to understand the config, and whether there's any other config in some other place that overrides the one they are looking at.

The more time goes on the more I start thinking a clean wipe of the entire thing might be in order. Linux is great and all, but it's so full of old crap.

It's getting better, but all the old crap still just sticks around fragmenting everything, and there's still no container system that can match APKs, because there's no API level to build to, every system is different, and everyone tries to support 50 different configurations.

Nmcli would be part of the OS itself if this were android or windows, and there wouldn't be ConMan and others unless you fork the whole OS or use some plugin API.

But developing the new stuff doesn’t make the old stuff go away; there’s a huge userbase with trillions of lines of scripts and stuff to support.

It’s thinking like this that led to the fragmentation in the first place.

Right. A new project can:

- get changes into the kernel, and all newly installed systems after some point maybe three years from now will have them, probably

- get changes into the core requirements of a distro, and get newly installed and upgraded systems to have the feature in a couple of years, but only that distro. If the changes are too obnoxious, some organizations won't upgrade and some will leave for another distro

- make an implementation so attractive that lots of distros will include it either by default or as an option.

Nothing will get the project traction on systems that are not upgraded, regardless of the reason being can't or won't. And as long as people have a high expectation of working on those systems, the replacement will be another thing people have to remember.

Windows has lots of old systems, but they just continue to run the old systems for a few years, and then 20 more years after support ends.

It's a lot easier to maintain old versions when there are fewer of them and they are all corporate backed. One team can keep a version going. But with Linux there are thousands of versions, arguably millions because it encourages customization even beyond just choosing a distro.

With Windows, Android, and just about any other commercial style thing, there are versions that come out once per year or so, with either no forks, or forks that promptly get ignored and forgotten, or very minor customizations that don't really add or remove anything beyond the wallpaper and launcher.

And they generally preserve backwards compatibility better than Linux, because the stack can just be application+bundled libraries -> OS stuff fully controlled by OS vendor.

There's no systems made of small pieces meant to be interchanged by the installer, any one of which could stop working on a new system, all by different teams, you know your GUI toolkit works on the new OS version because it ships with the new OS version.

There are no other distros, because even when they are possible, like with AOSP, there's no culture of tinkering, people just take what they get, which both prevents fragmentation, and on the other side, gives the developers a reason not to make migration too hard, because they are trying to make one distro for all, not just a distro for their 10% of users.

Ubuntu does a good job of trying to be "The standard Linux", but things still change and break between versions far more than on Android.

Not necessarily, you can support the old stuff but require the new way going forward. Even better, you can script the migration of a lot of the old stuff to the new one.

It's pretty difficult to do but at one point we should get to it.

Android essentially does exactly that already with API level targets and it's great. They even have security features where they say "From now on, if you target this API level you can't do this without going through this API, and we won't allow new things in the play store below X target".

Well, we have the administrative side of it down more or less. We have LTS releases, deprecation policies, even sometimes rudimentary automation to port old configuration and tools to match the new tooling. This part doesn't scare me.

The part that does scare me is that we'd need something like a task force that will have enough authority to compel all involved parties to follow their design as well as to have a good design in the first place.

Standardization is a powerful tool that enables significant advances in the industry that any individual player cannot accomplish on their own, and would be very beneficial to the most widely used operating system on the planet. But making useful standards is hard. Even harder in the presence of players who might want to achieve exclusivity for their product by obstructing standardization process. The later would've been my greatest fear.

The problem is that it would need to be near perfect or else you're ending up with another kludged-in mess like networkmanager. I'll take my one-tool-does-one-thing-well over that anytime

The problem is that currently Unix (well, Linux) doesn't have tools that do one thing right. They are too small and too overwhelmed by the problems they need to solve.

And this gets worse the more generic a tool you want to make, because, inevitably, you will have to account for the lower-level tools that your tool has to use. Let's, again, take storage as an example.

So, you want to write a program that shows users what block devices are currently available to them. Seems conceptually simple and potentially very useful. And, right off the bet you hit complexity: do loop devices count? Do separate partitions count, and if so, what to do about special partitions that keep data necessary for hardware RAIDs to function, that, in general, shouldn't be exposed to users)? How do you go about RAID devices, do you acknowledge that RAIDs exist and display them as separate entities, or do you pretend they don't and only display their constituents? How about physical and logical volumes? Do you think that you need to support ZFS volumes? What if some FUSE filesystem decides to expose some information as a device file, does that count as a block device? What about iSCSI (potentially with multipathing)? Encrypted devices?

And there are more... there's archaic NBD alongside with the blazing hot NVMe over IP and so on. And you don't really have any unifying interfaces. You can look into /sys/block, but depending on how you answered the questions above, you'll either find too little or too much info there. And there's not a single tool that can really, in all circumstances, give you a correct answer to the question "what block devices are currently available".

Unix started with a bunch of simplified assumptions and sloppy (if any) planning. It's OK for computers with predictable and not very diverse hardware configuration. But, because Linux is free (as in beer) enterprise sector decided to capitalize on this freedom. They no longer needed to develop the platform to run their stuff on. They took a tool that was supposed to enable individual users and small businesses to take ownership of their hardware and turned it into a money printing machine. And they keep using it w/o major investments as long as they can make profits.

It means that it sucks to be a system / infra guy (not financially, but in terms of what you have to work with), because your stuff is broken and will never get better, since the business decided that they'd rather run their stuff on a toy but free (as in beer) system than invest into developing a more grown-up one.

> So, you want to write a program that shows users what block devices are currently available to them. Seems conceptually simple and potentially very useful. And, right off the bet you hit complexity: do loop devices count? Do separate partitions count, and if so, what to do about special partitions that keep data necessary for hardware RAIDs to function, that, in general, shouldn't be exposed to users)? How do you go about RAID devices, do you acknowledge that RAIDs exist and display them as separate entities, or do you pretend they don't and only display their constituents? How about physical and logical volumes? Do you think that you need to support ZFS volumes? What if some FUSE filesystem decides to expose some information as a device file, does that count as a block device? What about iSCSI (potentially with multipathing)? Encrypted devices?

I think PowerShell had right idea here, even if it maybe went too hard into "object oriented".

If tools had option to pass structured representation between eachother the above problem could be solved by tool that returns everything + set of generic tools to filter information, like say

    lsblk | f(type=disk,bus=nvme,name != nvme0n1) # all physical nvme devices except nvme0n1
    lsblk | f(type=lvm_lv,vg=data,mountpoint =~ {^/data} ) # any LVM LV that is in data VG and mounted under /data

I wonder if it would be possible to retrofit into existing design. Like, have additional filedescriptors and some way of signalling so shell can connect "structured out" with "structured in" of the next app in the pipe.

> And there are more... there's archaic NBD alongside with the blazing hot NVMe over IP and so on. And you don't really have any unifying interfaces. You can look into /sys/block, but depending on how you answered the questions above, you'll either find too little or too much info there. And there's not a single tool that can really, in all circumstances, give you a correct answer to the question "what block devices are currently available".

I'd be interested in which case lsblk is not showing block devices, so far it has been pretty bulletproof for me, even in extremely weird setups

Microsoft isn't at the place yet where they may possibly have this problem. They aren't making an operating system that's extendable by others, that needs to incorporate interests of others. Where anyone may come and take any part of the system and repurpose it to their need. They have the luxury of not having to coordinate with others. This takes away from the usefulness of their product (eg. I see no value for myself in using such an operating system), but it makes it easier in engineering terms to make one.

So, PowerShell doesn't solve the problem. It didn't have one from the start. But, it also has nothing to offer to the Linux world as a consequence. The problem we have in the Linux world is more of a people problem before it's an engineering problem. We need an authoritative body that can mediate the interests of many parties using the system, incorporate their wishes and workflows, and then design a generic and extensible way of using the system. Right now it's chaos and anarchy which hinder or prevent the creation of useful programs. PowerShell would only increase the entropy of the system.

> I'd be interested in which case lsblk is not showing block devices

What about device files in a FUSE system randomly mounted somewhere? Also, I don't know of any way of showing loop devices with lsblk. Also, lsblk doesn't show RAIDs as individual devices, it shows them piece-wise. Also, I'm not sure about how it integrates (if at all) with ZFS block devices (I think it does on FreeBSD though). From what I understand, lsblk will only show a (large) subset of what may be found in sysfs. If it's not there, it's not a block device from its standpoint.

Ahem. AIX: mk*, ch*, rm*, cp*, ln* almost anything.

The BSDs: ifconfig anything.

Any Linux? NIH everthing!

I think having a JSON manipulator like jq be standard would be good, since it would encourage other tools to include JSON as an output format. I think it would reduce complexity as a whole, since tools would no longer need a lot of switches to configure their output, without making the user resort to sed scripts.

Ripgrep and fd-find are tools written in Rust replacing grep and find. They are in my experience orders of magnitude faster, with arguments I can actually remember.

The website does not open at this moment in time: https://downforeveryoneorjustme.com/exple.tive.org?proto=htt...

About the topic: I always install pv, jq, dstat, perf, mc, rg, and clickhouse.

I always right away install htop, iotop and iftop.

If you like htop you'd appreciate btop as well

I'd mention glances here as well - in particular because it can show resource usage on a per container basis if you run docker which is something I haven't seen in the others.

If you like bat, you may find these useful: https://github.com/eth-p/bat-extras#batman

Forcing me to install things manually is what appeals to me. In other words, a system that has 5 tools that do the same thing slightly differently is against my philodophy of minimalism. What could be cool maybe is an install time configuration wizard that allows me to select the right tool from a list of variants: maybe have them controlled by the env (like $PAGER, $EDITOR, ...).

Only works if you have local admin privileges. If you're on a locked corporate station, it sucks. Good luck opening a ticket to corporate IT every time you need some utility of package. That makes me get the chills every time Python developers talk about prunning the standard library.

Are there even any technical reason why mainstream package managers don't have an easy way to install packages per-user without requiring root access? I'm not a fan of Nix/Guix but IIRC this is something they got right from the start.

The point is that users should not be allowed to install apps that are not allowed by IT/security. So many “legit” apps these days are essentially spyware on their own. We’re not even talking about things that have malicious motives.

Unfortunately the majority can't be trusted to make safe decisions. Look at how even the biggest companies get compromised by simple phishing attacks.

"Hi, I'm your boss can you give me your private ssh key I need to update something"

"Sure, here you go!"

When a company gets big enough the probability of having at least one of those people ends up being 100%.

"Ouuh I can have emojis? sudo npm i -g". Sure it's a headache but when things go wrong it's ITs fault, so they need to be very careful and your niche tool that might objectively be better is another tool to audit in their eyes.

Working link while the page is down: https://web.archive.org/web/20230218100221/https://exple.tiv...

Tangentially related but it's kinda curious that there aren't any popular opinionated power-user oriented distros around. There is Arch for power-users but its opposite of opinionated, while mainstream distros (Ubuntu etc) chase the user-friendly segment. What I feel missing is essentially preconfigured sway desktop with all the goodies that are out there; the out of the box experience of e.g. arch+sway is very barebones; just because I prefer tiling wm and cli workflows doesn't mean I want to configure everything from scratch. Back in the days crunchbang was kinda in this direction, but unfortunately it did die away..

Manjaro-Sway is really close to what you're looking for.

Probably because us powerusers can't agree on the tools we use.

If your stereotypical regular Linux user is already what it is, imagine an opinionated power-user, which I would jokingly classify as the epitome of insufferable.

Every Linux power-user is opinionated - very opinionated, so every single one of them should make their own distro. Which is actually a good thing ( Freedom, remember? ), Everyone should make whatever they want, everyone else is free to criticize or ignore.

On the practical side of things, we want someone else to make the things we want, how we want. That's what employees are for and "the market" disagrees on the viability of that.

nushell looks great, like powershell without weird syntax and ui. Normal programming syntax and great visualization of output!

Always wondered why shell languages have to be so ugly.

I would add micro as default editor. It uses sane keybindings and ui, but isn't quite as brain-dead as nano.

Helix is maybe more in line with the rest of the suggested items

It should - discover other computers

- be able to exchange files with them.

- set the right permissions for sharing folders

- show a descriptive and actionable error message if something goes wrong.

That is all I ask for, the rest I can install it myself.

I install some of these myself, but I would never suggest making them standard. Standard to me is the bare minimum to make a system usable.

I've never used OhMyZsh even if zsh is my shell. Alacritty on a server? Not even cURL is installed by default on any distro I use so why should HTTPie be (even if it is a very cool tool)? Or a csv visualizer?

If the title was "Cool modern unixy tools" I'd have no complaints, but it does set a high bar when claiming they should be standard.

Personally I'd like fish, git, jq, a neutral editor that isn't too polarizing, and a good package manager.

Yes but the problem is that fish is not POSIX and so can't be the standard.

Agreed that allowing non POSIX might improve an OS.

Midnight Commander (mc), it's a bit like XTree Gold on DOS.

mtr

tcptraceroute

bat, ripgrep, fzf, tmux, fd, git

Reply on news.ycombinator.com