I think that if you have a good mental model of how awk works, then you know times that it’s the right tool for the job. And on those occasions you’ve got plenty of time to go search for some syntax, make a coffee, maybe even go for a run all with change to spare compared to using pretty much anything else.
I will admit, that if I can do it with a few greps, seds and cuts then I tend to just use those first, because there no thinking involved for me.
I'm going to let you in on a little secret: many of us who have been doing development or systems administration in some form another for multiple decades are the same way. I have a terrible memory, but it turns out that does not have to be a big hindrance for tech work.
When I'm working (or tinkering at home), I keep a tab to my personal wiki open at all times. I will not usually bother to write something down the first time I do it. But I have to google it twice, I throw it into my wiki. This does two things: 1) It makes me (slightly) more likely to actually remember it for next time, and 2) The next time I need the information, I know where I can find it without having to wade through SEO-encumbered blog spam or outdated StackOverbutt answers.
A database of personal notes is a powerful multiplier for technical ability and productivity. To the point that whenever I interview a candidate for a job, one of the things I ask is what system they keep their notes in. Their system doesn't matter at all to me but if the answer is none, that's a definite strike against.
>I really don’t think non-interactive CLIs are a good way to do complicated tasks.
Many of us thrive on the Unix command line because of the raw power you can wield with it. "Why waste time trying to learn arcane shell commands when you can just write a script in Python?" is a common refrain I hear from developers with only a few years under their belt. My response to that is, "Why would I waste time writing a Python script when I can do it in one line of shell?"
Imagine having this attitude towards tools in general, you'd throw out most of everything. Anything that's powerful has a learning curve
The first time you look something up, it might not be obvious if you'll ever use that info again, so why waste the time and clog up your notes with a bunch of noise? But if you've had to look something up twice, odds are good you'll have to look it up again someday.
And like you said, writing a note in a way that's clear and understandable helps you to recall the thing a little better next time
Any suggestion for a remotely accessible personal wiki? (I'm assuming that access is restricted for potential "personal" stuffs).
It may not be as powerful but it's just one thing to learn. Also it understands regexs like '\d', etc which are kinda universal but for some reason don't work with many of these utils.
I use sed for a lot of simple stuff, but I use Perl all the time for more advanced stuff. Perl is great for grabbing complex chunks out of a line and doing something with them that requires a bit of state. For example, writing a quick and dirty logfile parser to figure out how many requests and how many bytes were used by each IP over a time period (the time period limiting usually being a prior grep before being passed to Perl). I've probably taken 60 seconds to write out something to parse exactly what I want from a log hundreds of times.
I think the problem is that they fall into an uncanny valley between "simple tool" and "verbose programming language"; they are really terse languages wearing a simple tool's sheepskin.
And when one is using awk/sed that script is often invoked from a shell, bash for instance. And since the awk/sed can't hold its own as a general purpose programming language, now I am writing bash code, a personal sin.
Awk/sed made sense for the time and place they are from, but things have changed.
Don't do that. Do not assume something is of no use because you don't know how to use it. Double that recommendation for tools that have widespread use. There is a reason they're used everywhere.
In the case of awk/sed/perl: Nothing beats them for composition in one-liner text processing pipelines.
Don't assume I don't know how to use it. :)
One line verses ten, I'll take the more readable thing over the APL.
Can't find links now, but fairly certain were stories in past about how switching to AWK from other tools increased performance
Okay, acknowledged. Python and Ruby are superior in every regard. Would you mind letting us chat about sed and awk now for a minute?
"Please don't sneer, including at the rest of the community."
Also.. I use Awk every day, for all kinds of things. I forget the order of parameters in split, sub etc, which takes a few seconds to look up with man awk. I got into sed a while ago but never use it. Awk, together with sort and uniq sometimes, is all I need.
And it is interactive. You have an input. Print that out and look what needs to be done. Write the first transforming step in Awk. See what that did. Add the next step. See what that did. Extremely interactive. After a few minutes I have a one-liner (often 2 or 3 wrapped lines) that does exactly what I want, and easy to adapt to other similar things. I have very large bash history enabled so they stay Ctrl-R-accessible.
(I might be wrong about "everything" but that's what bomb-tossing is all about)
Great way to have confidence in a port and provide for a fallback path. Eventually the new code will take over the old code and the old code will wither and die.
And if you do use them frequently enough, you might not even need to refresh/relearn anything.
To me it's bash, sed and awk for the basic text-processing things, and then perl for anything that requires more programming. These are very similar syntaxes (and ways of thinking I'd say), so they fit together great - and also are close enough to php and js that I do at work to make the transition painless. But if I had to write it in python or ruby or God forbid C++ - which all I used a lot, but long time ago - I'm sure it would be a big struggle for me now.
I sometimes forget the exact syntax for pipes and file open/close, but it's not hard to find in 40 pages.
AWK is in busybox, and a lot of other places that Python simply cannot go.
The control structure syntax is also the same a C/JavaScript/PHP/C++ etc., so it certainly does not hurt to know it.
https://archive.org/download/pdfy-MgN0H1joIoDVoIC7/The_AWK_P...
I think by {} you mean an argument to the find command.
I love awk and whenever I __do__ find a task where awk helps me, it's really an amazing experience and it is always a boon, never a burden.
But I really suck at remembering the syntax as I only hit tasks that awk solves well rarely. What I do remember is situations where awk definitely can likely help. Same with sed, tr, etc.
Keep in mind, every time we automate something or introduce a new tool/process, it's a cost benefit analysis; when I have dozens of gigs of logs to parse to figure out which hosts out of hundreds are having issues, it's a no-brainer for me; the 30 minutes to revisit a few awk and bash tricks is a far better investment than trying to grep/scroll my way through literally millions of lines of logs/code.
When I first learned awk I definitely went through a 'hammer' phase (i.e., when you have a hammer, everything looks like a nail); once phase passed, I find that I'm much more disciplined on when I break out awk or similar tools.
Ultimately it's about identifying when you have the "right" tool for a specific job; I might need to invest time to revisit some syntax, but if the end result is that my 30 minutes saves hours, then it's a good 30 minutes. I'm at the stage where the basic text munging with awk is a no-brainer (or at least I remember the mistakes I made previously and how I fixed them); this was achievable only after a few projects where I did have to spend a bit more time wrangling awk than I preferred, but it saved me a lot of time in the future. Just knowing what I __can__ do with a given tool helps me make educated decisions on where I should invest my time on a given issue.
What about setting up the project for the application, setting up instrumentation and testing, debugging, etc.?
Even supposing that all that extra work makes sense for your unique string manipulation needs, what about the next time something like this comes up but is out of scope for your first tool? Do you end up with lots of little applications, with lots of little features and arguments? When you come across that similar-but-not-quite need down the road, do you have to scour your library of full blown apps to find something that fits? Not to mention maintaining them against future system updates...
OR - you could just force yourself to get over the hump, learn the conceptual basics at play and then when you come across a novel situation down the line you can just rattle off a tailored one-liner and have your answer in a few moments...
What we really need is something that can "compile" down to sed for deployment in that kind of environment.
Then that compiler can contain all the weird stuff you need to remember.
curl cheat.sh/sed
Will show you several examples.
Also, LDP: Advanced Bash-Scripting Guide: https://tldp.org/guides.html
I basically use it for simple text substitutions; in that case it is the same as what I use for vim editing.