Hacker News Reader

Comment by Snarwin | Hacker News Reader

> There's no native Unix equivalent to frequencies, this sort | uniq -c is the closest we can get. Not only is it less performant (it has to collect the full input into memory before continuing), but it ties aggregation to ordering.

One of the core features of the Unix command-line is that it is user-extensible. If there's no "native" command equivalent to frequencies, you can write your own, and it will be given the same first-class treatment as any other binary in your PATH. This is entirely in keeping with the Unix philosophy of simple implementations.

View on HN

Yeah, that point was weak. The argument went from "Unix pipelines are not simple" to "unix based operating systems don't have an equivalent to Clojure's frequency function by default so it's worse"

OK but how would it look like if you had such a program? Shells are not known for having the extensive set of functions that real programming languages have.

I think the point was more that it's hard to compose shell utilities to make something new that's not supported out of the box. Instead you do have to write an entirely new utility and use that.

That said having key/value semantics and not just stream of bytes would make the shell much more versatile, at the cost of making it bigger.

That's not what the linked article is saying. The point made in the article is that the 'uniq' utility couples together two unrelated concepts--sorting and frequency counting--and this coupling forces you to write more complex code, compared to the version where frequency counting is decoupled from sorting.

All of this is completely correct, but it has nothing to do with bash vs. Clojure. The same decoupling can easily be achieved in either language.

I think the article is a bit weak when it's making this point, because it's not about Unix pipelines. It's about POSIX shell utilities being too bare-bones.

But Unix pipelines are not simple too. They have a couple of nitty-gritty details that often come out and bite you.

1. They can only stream raw bytes, so all the programs that deal with lists like sort and uniq have to separate items using a delimiter (usually newline). If you want to process data with that delimiter in it, you're in for a ride. And if you want to write a custom tool, you have to do all the splitting yourselves (luckily it's so common most programming language will provide a ready-made facility for you to do that). This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".

2. Error are hidden by default in shell. Nowadays you can explicitly change this behavior, but you have to remember to do `set -o pipefail` and I don't think it was always there.

3. There is no data typing at all. Everything is binary or text. Nowadays a lot of programs just output JSON, and the users (if they even stay inside the shell) almost always reach for jq to parse it. But jq is not a Unix philosophy program: it's an entire streaming functional programing that can do quite a lot. But even jq gets hairy when you have to do a bigger query or transformation. In that case users often reach out to Python or another language and just move the business logic there.

I think this fits well with what the article is trying to say: Unix pipes are pretty easy (small) to implement on your own (compare that to something like Nushell's pipes). But the moment you to do something that's a little different than the happy path it was built for, you need to go for another tool (jq) that has its own built-in pipe and small programming language, because Unix pipes won't cut it. And for more complex (hehe) things, you'll have to reach for a larger (and simpler) tool: a full-fledged programming language.

> This is New Jersey approach again: "I'll make my code (the OS, the shell) easier to write, and in return make life harder for my users (the tool writers)".

I'm not sure if you're actually trying to argue for a specific position here, but you highlight several negative results of one "philosophy" of development, with the implication that the alternative wouldn't have those negatives.

This is a tricky point to refute because you're right, these are flaws and there could be a system that doesn't have them.

So why do these flawed systems exist and proliferate?

Because the real life result isn't actually a choice between "sloppy but quick to develop" and "elegant well engineered but slow".

The choice is actually between "sloppy but exists" and, well, nothing, because the other version never actually materializes.

(And of course, I feel compelled to point out that bash is a user interface not a programming language. Any attempt to replace it or improve it without focusing on that main point is doomed to failure, which is why you see so many people who apparently think that what bash really needs is strict type checking or something and end up creating a completely awful user experience)

I imagine just ask would be enough for most usecases:

awk '{ freq[$0] += 1 } END { for (n in freq ){ print freq[n] " " n } }' < file.txt

That is nothing special about UNIX, every other OS with command line experience, or REPL windows, has similar capabilities.

This was my thought, too. Either you can extend your toolkit—make a utility like dedup (https://codeberg.org/napcakes/dedup) and put in on your path (yay! so easy)—or you can't and thus must define the boundary of the allowed toolkit. In this case, it seems overly disingenuous to not allow awk, which will do the thing for you just fine, no Clojure needed at all. awk is hashmaps galore.

Honestly, so much of our conceptions of "what's wrong" is more to do with lack of familiarity with history than any actually unsolved problem.

Reply on news.ycombinator.com