GoAWK: an AWK interpreter written in Go
benhoyt.com
benhoyt.com
It's kinda nuts how far the unixy idea of "just streams of text" has gotten us.
As much as I agree that the tooling is, object-based tools would be much nicer.
for i in `ls`; do echo $i.size; done; ls -la | awk '{print $5}'Omg
But yeah, the solution he proposed was to start combining different information namespaces (such as the "metadata" namespace that you are alluding to) into the filesystem namespace, as well as allowing for the filesystem to be pluggable so people could write databases that could be searched with the filesystem.
So your example would look like
for i in "$(ls)"; do echo $i/..../size; done
echo 1000 | tee */..../uid
for i in *; do du -bs "$i"; done > for i in *; do du -bs "$i"; done
du: illegal option -- b
usage: du [-H | -L | -P] [-a | -s | -d depth] [-c] [-h | -k | -m | -g] [-x] [-I mask] [file …]
I'd like to ensure my tools are all speaking a format which is too simple to ever fail.
stat -c "%s %n" *http://nongnu.org/txr/txr-manpage.html#N-000264BC
* Implements almost all salient POSIX features and some Gawk extensions.
* awk expression can be used anywhere an expression is allowed, including nested within another awk invocation. Awk variables are lexically scoped locals: each invocation has its nf, fs, rec and others.
* awk expression returns a useful value: the value of the last form in the last :end clause.
* can scan sources other than files, such as in-memory string streams, strings and lists of strings.
* supports regex-delimited record mode, and can optionally keep the record separator as part of the record (via "krs" Boolean variable).
* unlike Awk, range expressions freely combine with other expressions including other range expression.
* ranges are extended with semantic 8 variations, for succinctly expressing range-based situations that would require one or more state flags and convoluted logic in Awk: http://nongnu.org/txr/txr-manpage.html#N-000264BC
* strongly typed: no duck-typed nonsense of "1.23" being a number or string depending on how you use it. Only nil is false.
Recently accepted Unix Stackexchange answer featuring awk macro: https://unix.stackexchange.com/questions/316664/change-speci...
I'm all for scratching an itch, but why rewrite all these well established tools?
While you're right this is reinventing the wheel it can make sense to reimplement old tools to improve safety, security, and to allow them to be embedded in new environments.
Have you ever run a fuzz-tester against (GNU) awk? I have. Even now you can segfault awk with bogus programs, for example:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816277
No doubt this new implementation won't be perfect, but segfaults should be ruled out via rust/go/ML implementations..
On the other hand, I think what you're saying about expanding a well known tool's range of use in new environments makes sense.
If you are interested in more depth, either Crafting Interpreters[1] (mentioned in the article) or Writing an Interpreter in Go[2] looks promising. I've read more of Crafting Interpreters and really enjoy it, though it isn't yet finished. One of the aspects I really enjoy is that the language is implemented and re-implemented in different languages to gradually introduce lower level concepts.
Finally, this one may be a little more "out there" than what you are looking for, but if you are interested in designing a language more than the plumbing of an interpreter Beautiful Racket[3] is really good.
caveat: not an expert
[0]: https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...
[1]: http://www.craftinginterpreters.com/
Recently there is an official Github repository with commits from Brian kernighan.
There was a similar article about writing the LuaVM in Go, to package it in bigger Go applications. I've done lots of C based systems and bolted Lua on, so the Go version makes sense.
But is imbedding Awk into a program that gets done on a regular basis?
Also, pjw, the W in AWK is on the Go team at Google and Kernighan, the K, wrote the canonical Go book.
An AWK interpreter written in Go is unlikely to be an improvement, except, well here is another blog post you might be interested in that has a similar sense of adventurous tinkering (it's about improving on grep): https://ridiculousfish.com/blog/posts/old-age-and-treachery....
That's from 2006 and the tl;dr was graybeards did things a certain way for a reason. And yet nowadays with have things like rg (and ag and a bunch of others).
Finally, I think it is good to see C software being rewritten in more robust languages like Go and Rust, if people are inclined to do so. I don't think rewriting all the C software should be our top priority, especially high-quality programs like AWK, but it's a good overall direction to go when people are interested.
For example I have wished for a long time that Gawk could parse gzipped files so that filenames could be used directly. I could take a run at implementing that in this interpreter where the C version would be more difficult personally.
C++ projects used to add ++ as suffix, for example Rogue Wave Tools.h++ and Motif++ libraries back in the glory days of commercial C++ compilers.
So this fashion is already quite old.
Programming languages are software products as well, and their customers want to feel they have made the right choices sticking with their options.
This kind of people tends to do badly in generalist coding interviews that rely on their fundamentals.
The language tells you loads! This is a stupid complaint.
Imagine if it was PerlAWK, or PHPAWK.
In general, I think that it is acceptable to include the language name in the project if you are trying to to convey that the project is an implementation; of another project in that language and is not really meant to differ significantly from the original. Such projects are typically meant as an experiment, a library implementation for the target language etc. Anything else should be named differently — especially if it is going to differ significantly from the original.
Another thing to keep in mind is that by naming the project GoAWK, the author is as good as claiming that this is the canonical implementation of AWK in Go— something they may not have intended.
If I'm rewriting a tool like Awk in go, then coming up with new name is hard and naming it as goawk, means it is a reimplemention of Awk in go.
If one names it as xyz, then it would probably gain less attraction, and in comments people would complain that this is a reimplemention of Awk in go.
Because that was a great motivation for them being written in the first place. "Let me write X in Y, to have Y easily used from Y or as a learning exercise or because Y is fun or bring benefits (e.g. Rust and safety, Go and easily parallelizable/static build).
>Does “Go” or “JS” or “Swift” or “Node” make these project more attractive somehow to an end-user (even if the end-user is a programmer)?
Yes, very much. I prefer Go and Rust utilities now whenever I can find them over an equally good alternative.
(Plus, you seem to have forgotten that "if the end-user is a programmer" they might be interested to play with the source, and there the language is very important).
To me it signals lack of creativity. The problem of these projects seems to me that they have no reason to exist (beyond playground for the developer) other than "a tool written in that language". For example, here, awk already exists. Why would someone use GoAWK other than to indulge their love for Go? These are the cases that bother me the most; pushing a language or a tech to give your project validation because it has little as a standalone.
Well, if you do see it, it hurts you how?
>To me it signals lack of creativity. The problem of these projects seems to me that they have no reason to exist (beyond playground for the developer) other than "a tool written in that language"
Well, to me "a tool written in that language" is quite important. As a developer I don't just care for the tool's functionality, but also its hackability, portability and ecosystem.
That's irrational. What difference does it make what language the utility is written in, if it works as expected?
The language influences whether I can easily hack the utility or not.
The language influences how easy the utility is to port.
The language influences how easy the utility is to use as a lib, integrate with some other project.
The language influences how many memory bugs the utility might have (e.g. C vs Rust).
The language influences how fast a utility is to build (e.g. Go build times).
The language influences how easy a utility is to deploy, or to have many different versions of (e.g. static builds vs a mess of classpaths and virtualenvs and the like).
The language influences how easy is to install (e.g. go get/install, or cargo equivalent vs messing with languages with no package managers like C, where new or obscure projects are almost never in the official distro repos).
The language influences how easy is to build (e.g. go build vs the C/C++ autoconf/automake clusterfuck and dependencies libraries hunt).
The language also influences how performant a utility will be (e.g. csv query/processing tools written in Python vs xsv -- or the canonical example, Electron monstrocities vs native tools).
Not all of these traits are guaranteed given a language (to preempt the first knee-jerk objection), e.g. a Python tool could be faster than a Go tool if written well enough.
But historically and statistically speaking, and in a regression to the mean sort of way for each language platform, I've found those things to work better in some language X over another Y.
Hence, for me, the language matters.
(I dislike the term “hack” as it denotes to me irresponsible way to throw code, rather than properly design features and implement them in a considerable manner.)
I agree with most of the things you say above, they are very important. I just don't think it should be part of the name and identity of a utility. A tool should not have to scream to the world "I am a fast tool because I was written in C++ and not JS"—that should be a given for every utility.
No, but not all the utilities I use are open source either. When they are, and for some domains (e.g cli text manipulation tools and development utilities) I like to be able to hack them, even if I don't, so for that (and for the other reasons) the language they are written in is important to me.
I never said it's the sole criterion.
>I dislike the term “hack” as it denotes to me irresponsible way to throw code, rather than properly design features and implement them in a considerable manner
For deployment code, yes. For utilities, usually that's exactly what I want though: to irresponsibly throw code for my own purposes, when I find it convenient to fix an annoyance or bug in utility I use or add some small feature I want to it.
It's advertising or propaganda for the language.
Whenever a language is "hot" and going through a full scale PR campaign, you see something like "[fill in the blank] written in Rust/Go/Ruby/etc" over and over again.
No different than all the current "facebook is evil" spam or the "bitcoin is great" spam of a few years ago.
What advantage do you see in command line tool running in its own container? Given how important pipes are, that's going to be a lot of overhead punting data between containers.
Having to manually track memory liveness C adds a large amount of complexity to tools like awk, sed, grep (which are already complex beasts themselves).
Sure, those commands could just allocate and never free memory (a-la early C compilers, or the D compiler), but now any use-case that involves a large amount of data will leak noticeably. Not going to fly if you need these commands to be durable and efficient. And unix commands need to be both.
A GC gets you the freedom to operate on large streams for free, without having to worry about memory management (modulo optimization, but that happens later anyways, regardless of GC presence or not).
* Some deal with only one pixel at a time, or sometimes two. No dynamic allocation is needed.
* Some deal with one scanline at a time, or sometimes more than one (but a fixed number) at a time. The same buffer can be used for each scanline.
* Some deal with the entire picture (such as those that distort the picture).
But one possibility can be that a program might load multiple pictures and each picture needing the entire picture at once, but does not use them simultaneously, in which case it is sense to free each picture after it is used.
(Or maybe I somehow misunderstood your message or something else.)
Almost all unix command line utilities fall into this category. Having to worry about pairing your `free`s with your `malloc`s is a strict increase in cognitive overhead, which should have been spent on verifying the program's semantics are correct.
Messing up low-level memory operations, when you just want to worry about semantic correctness, potentially leads to bugs like RCEs, or dosing somebody with too much radiation.
There is at least some data that GC does have an impact on command line tools like this: https://boyter.org/posts/sloc-cloc-code/ --- More experiments like that would be great to crystallize the exact trade offs here.
Something I ran in to the other day when I was trying to produce a reverse sorted list of strings:
sort.Sort(sort.Reverse(sort.StringSlice(dir_contents)))