Hacker News Reader

Comment by ChicagoBoy11 | Hacker News Reader

I've found this seriously cool!

While not my primary role at my job, I often find myself dealing with lots of disparate data sets, usually needing to do some sort of manipulation, cleaning, searching, etc. Every now and then encounter something like this and it seems to me that there are potentially a nice set of command line tools/utilities that I should be adding to my belt. Anywhere I should particularly start taking a look? Like, if my goal is to because much better at wrangling CSV/text-delimited files, searching across folders of docs for numbers, etc., where is my first entry point into trying to become much more proficient at it?

View on HN

Here's a recent gateway post: https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-l...

There are even more. :)

I do a lot of that sort of thing, and my go-to tools are grep (with full regular expressions), sort, uniq, head, tail, sed, and, the 200 pound gorilla (not as heavy as it could be) awk. Then if all else fails, Python

I highly recommend you learn the basics of AWK, which is a little language that lets you match on and run commands on every line of input (so AWK "scripts" are usually one-liners). The very basics can be learn in 5-10 minutes, the whole language in a couple of hours (read the first three chapters of "The AWK Programming Language" by A, W, K.)

For searching recursively through folders, I recommend ripgrep (rg), though it won't do number conversions and the like.

Self-promotion (but I think it's useful here) is my GoAWK project, which is a POSIX-compatible version of AWK, with the recent addition of proper CSV support: https://benhoyt.com/writings/goawk-csv/

Why would you do a posix compat awk? I loved awk, but it needs to be updated adn refined into a better language instead of just recreating the old one.

Compatibility: so it can run all the existing AWK scripts without modification, and people can refer to existing AWK documentation when writing scripts, and they'll just work under GoAWK.

For a 45-year old language, it's actually surprisingly good. There are a few annoyances and clumsy things (getting substrings is a function call without nice str[start:end] syntax, regex matches are harder than they need to be, string concatenation syntax is weird, arrays aren't nestable (except in Gawk), and so on).

I've actually thought it'd be great to "fix" these things and make a nicer AWK language, but the problem is then no one would use it. :-)

fd, bat, fzf are good starting points

also mdfind if you're on macos

I'm hesitant to go fully into "new" stuff so I can maintain skills on random *nix boxes I need to ssh into.

Try jq for parsing/selecting JSON data.

Reply on news.ycombinator.com