Hacker News Reader

Comment by benhoyt | Hacker News Reader

I was privileged to be one of the technical reviewers for this book. There's a fair bit of the original content (which is still great), but Kernighan's done a great job with some good restructuring and some significant updates, too. The early chapters are very hands-on, with something of a focus on "exploratory data processing", particularly with CSV files. Big data with AWK, you could say.

Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode (parsing files with quoted and multiline fields per the CSV RFC). I'm really glad Arnold Robbins added a robust "--csv" implementation to Gawk, too, because that's really the most-heavily used version of AWK nowadays. I've already got CSV support in my own GoAWK implementation, and I'll be adding "--csv" to make it compatible.

I'm really glad this new updated version is coming out!

View on HN

Its a crying shame we never settled on a control character separated text format. There's a ascii control characters for record and field (unit) separators. A bit of user space support for that would have been great.

It is a shame. I have been using tab-separated sheets recently as it allows me to simply not care about almost any possible character in my strings...apart from tabs of course. But those are far less common than commas, and putting strings in quotes 100% of the time looks messy to me.

Way less common would be using ascii 30 and ascii 31. ascii 29 and you can cram multiple datasets into one file

As I recall, you can tell Awk to use the control characters as record and field separators. Not helpful if you're getting your data from others, but if you're working by yourself, you have the option. I've come to use control characters as a default because it makes life so much easier.

What do you recommend for viewing and editing such files?

Visidata works with arbitrary separators. I just tried with a CSV separated with ␟ (ASCII unit separator) and it worked just fine.

Excel too?

lolive, VisiData has some Excel support. However, don't expect VisiData to be a full blown editor for Excel files. It can provide a view of the data in an Excel spreadsheet.

I think GP means to ask if Excel can read files with ASCII 29,30,31 separators.

If you have a python installation available, openpyxl[1] is great both for converting to .csv and for packaging .csv outputs as .xlsx (which is really zipped .xml, anyway).

[1] https://openpyxl.readthedocs.io/en/stable/index.html

Some discussion of that here: https://news.ycombinator.com/item?id=31220841

To be really useful as a format it would just need for text editors to: -display something distinct for the field separator (some editors do this) -treat the record separator character like a carriage return (not aware of any editors that do this)

Right. The issue is the user space support at the end of the day.

> To be really useful as a format it would just need for text editors to

This made me think of WordPerfect's "reveal codes" functionality. :)

(Word's "Reveal Formatting" is supposedly similar.)

>To be really useful as a format it would just need for text editors to: -display something distinct for the field separator

Which would be trivial too.

The programming might be straightforward. Trying to persuade the product owners to do it is a different matter.

Pull request?

After you. ;0)

Yeah, that would work /s

Miller handles it.

* https://miller.readthedocs.io/en/6.8.0/file-formats/#csvtsva...

I have programs that handle it.

* https://jdebp.uk/Softwares/nosh/guide/commands/console-flat-...

Tab-delimited "csv" formats are quite common (e.g. the CONLL format family for many natural language processing tasks) and also supported by common tools such as MS Excel for decades already.

Most important comment I have ever read on HN ever !

> Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode

Awesome!!!! Super excited to see this!

Awk is really great, for those knowing nvm [1], I used awk to make `nvm ls-remote` run more than 10 times faster [2] by replacing the related shell script with around 60 lines of awk script [3], and I was quite happy with the improvement.

It's not really a one-liner, neither something big, but one can take that as an example regarding that awk is really not just for one-liners.

Meanwhile having `--csv` support is really nice. I'd also like to see things like a builtin `length` function to be standard.

[1]: https://github.com/nvm-sh/nvm/ [2]: https://github.com/nvm-sh/nvm/pull/2827/ [3]: https://github.com/nvm-sh/nvm/blob/9a769630d7/nvm.sh#L1703-L...

But length() is standard POSIX, no? Even length(array) has been approved by POSIX [1] but not yet included in the spec (they're very slow to update the spec for some reason). Both forms have been supported in onetrueawk, Gawk, mawk, and Busybox awk for a long time.

[1] https://www.austingroupbugs.net/view.php?id=1566

Ah good to know, thank you! Meanwhile I found it, it was the awk from Debian 10 that lacked `length(array)`, see the related nvm pr [1].

[1]: https://github.com/nvm-sh/nvm/pull/2917/

Our data product is delivered in CSV format. Even though I create user documentation mainly using csvkit, grep and sed, I would love to convert all those solutions to AWK. Sometimes AWK is more readable than sed and csvkit requires installation.

It will be nice to have a awk cookbook for CSV. In terms of CSV maniupulation and querying there is only a limited number of operations and I think there is potential to standardize those operation using AWK.

Ben is not just any old technical reviewer. He wrote a version of AWK in go and has done a ton of other work in the AWK ecosystem.

Thanks! ICYMI: GoAWK [1] - A POSIX-compliant AWK interpreter written in Go, with CSV support.

[1]: https://github.com/benhoyt/goawk

And it can be embedded in your Go programs as well.

It's nice that everyone is supporting this, I've written a portable awk module that takes control of the parsing and it is SLOW (and a little buggy). I'm a little bummed that nobody will use it but this is truly a step in the right direction.

I guess for the people that are still using nawk, you can set up an AWK envvar so you can { awk -f $AWKU/ucsv.awk -f <(echo '{print NR, $1}') }

https://github.com/Nomarian/Awk-Batteries/blob/master/Units/...

Would you say the first few chapters are enough to get the 75-80% usefulness for mere mortals like me who will never try to master the full language? Or is the material fairly sprinkled throughout the whole tome?

Yes, definitely. The first three chapters would be more than enough for that: 1) An Awk Tutorial, 2) Awk in Action, and 3) Exploratory Data Analysis. For most people who just want to use AWK for one-liners on the command line, you can stop there. The rest of the chapters are about writing larger (still small! but not one-liner) programs in AWK to create reports, little languages, and experiment with algorithms.

Thanks!

Fantastic news. I’ve tried lots of new CLI tools but they always seem to fall between too little functionality (eg. xsv) and too much (VisiData). AWK is just right.

This is amazing, I may never use pandas again

Reply on news.ycombinator.com