It is a reason to “just use Python” for scripting, or Java or C for that matter.
There are some things shell does really well.
It was actually just a simple act of crawling over source files! Amazon was looking to remove their customer support phone numbers from all web pages (to save money, they wanted people to stop calling them; back in the early 2010s)
I share the wonder that it took several days (perhaps that is the anecdote taking a life of its own, perhaps it was actually several hours. Or maybe it was was several days interspersed with on-call support, meetings, and other work running in parallel).
Though.. the file parsing libraries back in the early 2010s in Java are somewhat basic, probably using Java 5. Which does make me wonder, how long would that exercise take in Java 8 or later compared to Java 5?
Here are some facts for you to ponder:
1. Our Contact Reduction team really did have exactly this problem in 2003. This isn't a made-up example.
2. Someone on our team produced the list within an hour, and the list supported more than just the 2 formats above.
3. About 25% to 35% of all software development engineer candidates, independent of experience level, cannot solve this problem, even given the entire interview hour and lots of hints.
[0] https://sites.google.com/site/steveyegge2/five-essential-pho...
There is a very simple way to write a breadth-first webcrawler that works in stages and saves the frontier in an ordinary text file, this kind of webcrawler can be implemented in almost any programming language, particularly shell. That is, you write a script that loops over the urls in the frontier, fetches them, then writes new urls it find in the next frontier, you can use sort, uniq and such to manage the frontier.
That simple breadth first webcrawler has the big advantage that it does not get stuck in web traps (say a calendar that has a button to go the next year which could easily scan up to the year 9595, if man is still alive). Or rather, after a certain number of passes (maybe N=10-15] all the real URLs have been crawled and the remaining URLs are web traps.
In a language like Java it is tempting to write a much more complex crawl manager that has the potential to crawl faster with threads but takes more care to not crash target sites, get caught in web traps, etc. The first web crawler I wrote (1999) had a highly complex crawl control system, future web crawlers I wrote got simpler and simpler because the simple breadth first crawler works very well for small projects.
http://www.leancrew.com/all-this/2011/12/more-shell-less-egg...
https://buttondown.email/hillelwayne/archive/donald-knuth-was-framed/
While often framed as "a contest" the two approaches are really solving different enough problems as being apples-oranges. So, the only meaningful judgement of a contest is "who better achieved their narrow goals?"
I've found in discussions with people that they seem unable to separate their judgement of "a winner" from their own personal weight on the value of the two narrow goals ("detailed articulation mixing code & text" vs. "just getting shit done").
There is probably some broader lesson about people in that (and indeed probably a specific term in the science of psychology) - those who really like apples or really like oranges being inclined to force a non-contest into one of their favored "Eat oranges!" rhetorical moves.
EDIT: and it's actually even more subtle - the very same person can side with either depending on broader context or framing. There are those who would praise detailed documentation over write-only hacks requiring detailed memory of tool innards in one situation yet decry it as a waste of time in another. (All part of why survey results are not to be accepted naively.)
When I write bash scripts which will go to git and probably stay there for a while, I use long form of the switches, like --long-option-name instead of -lon. They make the script more readable when after a year someone wants to modify it.
It allows things like history | fzf to work very well.