Hacker News Reader

> Tell me one thing about that script that's easier in bash than in Python

It's free of Python dependency hell: easier on the downstream consumers.

Python would be a nice language if it were more like Awk: a single executable with a few satellite files and no goddamned "ecosystem".

https://imgs.xkcd.com/comics/python_environment.png

In all other regards, of course proper data processing in one language is vastly superior to some awkward string chopping through external processes.

> You should invest more time using languages with docs you can actually read, and with syntax you can actually remember. You might like it.

Didn't grandparent say: "I've written more Python than I have Bash (by a substantial amount)".

People who aren't stupid and know other languages still whip out little shell scripts; we should try to understand why that is.

The current year is 2019. It's not as if every modern Unix based operating system didn't already include or even require at least one goddamned Python ecosystem.

Nobody's running such an old version of Ultrix on their MicroVAX with a big yellow thick wire Ethernet cable and vampire tap and static /etc/hosts file derived from HOSTS.TXT downloaded weekly from SRI-NIC.ARPA without checking certificates for man-in-the-middle attacks and running unencrypted rlogin and telnet demons instead of ssh any more, that they can't easily and securely update Python and install modules.

Also: You think the Python "ecosystem" is goddamned? Remember that bash by its very nature depends on a terribly goddamned inconsistent "ecosystem" consisting of all the essential utilities it requires to do the most elementary of tasks, including awk, sed, dc, cut, grep, yes, and many others, whose locations and apis and features and parameters and behavior and performance vary widely from platform to platform. Bash is much more tightly coupled with the operating system distribution that Python.

While most of those essential features (like cough cough string manipulation and arithmetic rolls eyes, among others) are built into Python or available as standard plug-in modules (like os, system, threading, multiprocessing, pexpect, mmap, readline, Beautiful Soup, SQL Alchemy, json, yaml, etc) that are identical across all platforms.

And most of those Python modules are vastly more efficient than their bash counterparts, because they don't require forking separate processes instead of making local procedure calls, communicating over 1-dimensional pipes instead of shared memory, serializing and deserializing text parameters and data instead of passing pointers to live data structures.

What good is a scripting language these days that doesn't have built-in json and yaml parsers (not to mention a nice way to parse html and talk to databases)? I see so many half-assed bash scripts desperately trying to parse field separated values and process output with regular expressions, grep, cut, awk and sed, by forking off hundreds of ridiculously inefficient processes.

Yes I know there are xpath/jsonpath utilities you can execute from bash scripts that look up elements and keys in XML documents and json dictionaries -- that's my whole point, that there NEEDS to be something like that! God forbid you need to process 150 million records.

And as to what the grandparent poster actually said: I explained in the cousin posting below that I quoted and addressed what he ORIGINALLY said, and then he NINJA-EDITED it after I replied, to substantially change the meaning and remove several ridiculous, indefensible statements.

https://news.ycombinator.com/item?id=19898052

Unless he's just gaslighting and actually MEANT for you and others to mistakenly believe that I was putting words in his mouth (in which case we should expect his radio silence to continue), I'm still waiting for him to acknowledge that fact, and finally reply to the simple straightforward question that he kept avoiding:

Name one thing about that script that's easier to do in bash than in Python.

But I'm not holding my breath. Do you care to chime in with an answer instead?

>People who aren't stupid and know other languages still whip out little shell scripts; we should try to understand why that is.

Well the grandparent's excuse (which he deleted) for whipping out little shell scripts was that he "only really looked at the docs a couple of times". I don't think that's a legitimate excuse, myself, and apparently neither did he, so he ninja-edited it out, instead of admitting it wasn't a good defense of bash to brag about only reading the manual twice.

I'm not really arguing for bash or awk as a surrogate for Python. Just wanted to remark on some of these comments. (Also, I'm not the person you replied to, FWIW.)

> The current year is 2019. It's not as if every modern Unix based operating system didn't already include or even require at least one goddamned Python ecosystem.

If you're just talking about Linux, this is kind of true (Fedora's dnf and yum before it are built on Python). But AFAIK Debian/Ubuntu apt-get is written in C/C++ and does not require Python.

Non-Linux unixes like, say, FreeBSD, do not have Python in the base distribution (nor bash). It's available in ports, which might be what you're getting at.

That said, my experience has been that non-Linux "posix" Python is a second class citizen to Python on Linux. Just for example, signal/thread semantics happen to align in a certain nonspecified-by-POSIX way on Linux and differently on FreeBSD. IIRC, FreeBSD delivers nonspecific process signals (e.g., external INT or TERM) to the most recent thread, whereas Linux delivers signals to the oldest thread. Or vice versa, but Python has its own weird signal semantics it tries to impose on top of this that works better on Linux's model than FreeBSD.

As a second example, Python fork+exec invokes close() on every possible integer between N and the file descriptor limit. This is typically tolerable for naive python programs on Linux because the default fd limit is usually 1024. FreeBSD scales the default fd limit with memory, often resulting in a much larger limit (e.g., ulimit -n = 939258 on this 32GB system). For years we have tried to get Python to adopt the closefrom(2) API, which accomplishes the same goal much more cheaply. But because most Python developers are on Linux, they simply don't feel the pain of a subprocess.Popen() taking noticeable fractions of a second.

> Also: You think the Python "ecosystem" is goddamned? Remember that bash by its very nature depends on a terribly goddamned inconsistent "ecosystem" consisting of all the essential utilities it requires to do the most elementary of tasks, including awk, sed, dc, cut, grep, yes, and many others

No disagreement there.

> What good is a scripting language these days that doesn't have built-in json and yaml parsers (not to mention a nice way to parse html and talk to databases)?

jq is a wonderful tool for parsing and manipulating JSON in sh, though it has its limitations. I certainly wouldn't attempt to parse HTML in (ba)sh. I don't encounter YAML very often.

> Name one thing about that script that's easier to do in bash than in Python.

"ls -l"? There are lots of quick and sloppy path manipulation things that are easier to bang out in shell than in Python -- that's why most people use (ba)sh et al as interactive shells, instead of python.

Anyway, Python is powerful, and quite possibly more consistent platform to platform than bash. No disagreement there. I'd also argue it's a much better tool for writing programs that handle adversarial input than bash is. The ways Python differs between platforms are mostly areas bash doesn't attempt to cover at all. And the datastructures available in Python are much more powerful than in bash.

> For years we have tried to get Python to adopt the closefrom(2) API, which accomplishes the same goal much more cheaply.

Has anyone tried coding it and creating a locally applied patch?

> You're the one who offered up that code as an example in the first place, to prove your point about bash being really great for people who don't want to bother reading documentation.

I posted a link to it next to the sentence that talked about the types of tools I use it for, such as things on my dev box to help me in my day to day.

> Have you ever written any 10k+ line bash scripts? No, because it doesn't scale (and machine generated configure scripts don't count)

I've written a bunch of 1k+ line scripts. They aren't a problem to maintain and a number of them are being actively used and maintained by other people. These are for real production systems too. Some are single files, others are broken up into multiple files.

I haven't personally written a 10k+ line script (the need for that has never happened), but I've seen a number of multiple thousand line scripts out in the wild that are very well maintained and are part of mission critical systems like setting up a PKI realm for a bunch of servers. That PKI script was something like 4,800 total lines of code but it being 10k+ wouldn't have made a difference for the worse.

Shellcheck helps a lot with writing pretty well formed shell scripts. It's a really good linter.

I just wanted to be clear that I'm not just making this stuff up, and you really did write what I quoted (claiming to have written 10k+ lines of shell scripts, that you only really looked at the docs a couple of times, and that bash syntax is humanly impossible to remember), but you ninja-edited it out later, after I quoted what you originally wrote:

>I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash.

Only really looking at the docs a couple times isn't something to brag about, and doing that doesn't make bash good.

Nor is it good for languages to be designed so you don't have to read the manual (or else you get COBOL or HyperTalk, and then you still have to read the manual).

Nor is it good for languages to have syntax things that are humanly impossible to remember.

All those claims you deleted actually support my point about bash being terribly inferior to Python (whose syntax is extremely simple and easy to learn and remember and most importantly read, and it even has built-in interactive "help" documentation), which is why I quoted what you originally said in the first place.

Again, what is it about that script you offered as an example that is easier, more elegant, or more maintainable and understandable to do in bash than in Python?

If you don't have an answer to that, can you show me any other bash code that was better off written in bash than in Python? And explain how it's better, please.

Thanks, Don, for continuing to stand for truth in a energetic and entertaining manner.