Hacker News Reader

Comment by SOLAR_FIELDS | Hacker News Reader

For basic file system operations it’s a necessity. Write 10-20 line python script or type a one line grep/awk/sed piipeline to spit out the contents of some files?

View on HN

It's a 10-20 line Python script if you write it without any support code. If you're writing any amount of Python code that does shell things, you should either write or get an existing library.

Also, it really isn't 10-20 lines to spit out the contents of some files in Python:

    >>> filelist = ["tmp.pl", "tmp.go", "Tmp.hs"]
    >>> for f in filelist:
    ...     print(open(f).read())

It's trivial to code golf that down to 1 line, but if we're writing in Python at any sort of scale, this is more realistic. Easy things are still generally easy. They just aren't optimized down to shell level in terms of keystroke count. They're often more optimized than bash in terms of conceptual complexity, though.

For instance, if one of my files has a space in it, shell becomes a conceptual minefield, requiring you to understand the half-a-dozen ways it might process strings and how that interacts with filenames and arguments passed to commands, whereas the Python just keeps doing what you probably meant, because it knows what's a string and what's something else. Shell makes a lot of sacrifices for that concision, and there's a lot of "Well, this will probably be OK on anything I run it on...". I never put spaces in my file name, but use periods or underscores instead, precisely so I don't screw myself over with shell. I shouldn't have to do that. Personally I think the crossover point is in the high single digit number of bash lines. The only advantage bash still holds over Python at that point is large pipelines, and 80% of those can be replaced by Python functions. For the remainder, use a pipelining library.

    for i in ‘ls /tmp/foo’; do echo $i | grep bar | cut -d’-‘ -f3,4; done

Is just so much easier to remember than Python’s OS library, right? I can intuitively string that together but can’t write python without looking up the docs and reading a paragraph about idiosyncrasies.

I said Python wasn't going to be more concise. I said it has a variety of other useful properties. For instance, at least as I write this, you have "cut -d’-‘"; I assume you meant apostrophes (something getting too polite in a c&p, I assume) but the apostrophes are unnecessary, but you're so used to the compromises in shell I talked about you probably put those in there automatically. I'm not criticizing that; it's a reasonable accommodation to shell's quirks. But it is a quirk.

Your code also breaks if the filenames have spaces:

    $ ls -1                                                                      
    bar-def-gh i-jkl-mno-p                                                                         
    bar-def-ghi-jkl-mno-p                                                                          
    $ for i in `ls`; do echo $i | grep bar | cut -d- -f3,4; done
    gh
    ghi-jkl

The natural Python code you'd write to replace it will do what you expect and print the file with a space in it in the way you'd expect. You can fix this in bash via "for i in [asterisk]" (or, in this case, "for i in [asterisk]bar[asterisk]"), but, well, remember when I said bash requires you to understand the half-a-dozen ways strings and filenames interact...? In Python, there's just the one way that strings work.

(HN is forcing me to say [asterisk]. AFAIK there's still no escaping for them.)

You lucked out a bit here too; I don't think I can get echo to do anything very interesting based on file names. Had you interpolated that somewhere more interesting I'd give you some security issues too, if anybody not trusted can create files, which, again, the naive Python code would be much more robust to, because in Python, you have to explicitly ask to bring a string up to an execution context or a function parameter context, unlike shell's complicated rules for interleaving them. You pay a steep price for the convenience.

Of course, this is a matter of scale. I use that sort of pipeline interactively every day. I just start shuddering when it goes in a file for a script, and shudder more if it goes into source control.

It's hilarious in these discussions that every time someone comes to show how they can do the equivalent code in much less lines of bash there are always always always bugs of these sorts.

And then the parent even specifically pointed out that files with spaces would be a minefield.

Also complaining about looking up docs is just a matter of where you have your knowledge. I would have the opposite problem to understand the "idiosyncrasies" of what the -d flag into cut does whereas I know the python standard library by heart.

You’re taking HN too seriously, it’s not a formal argument for nitpicking - it’s an opinion.

Sorry I wrote that in a taxi from my phone, not a serious usecase but that’s beside the point I was trying to make- I personally have a hard time using python over bash. Thanks for pointing out the security aspect, I don’t typically think about that.

I figured; you clearly knew enough bash to know what the real backticks are. Those... frontticks?... aren't even on most keyboards.

On the plus side, when I fixed the typographical issues, it did work. I'm pretty sure I could do this more-or-less first-time right in Python, too, but it clearly took me longer to get there.

You seem to be imagining some sort of situation where bash is used to deal with input from customers (random users). You do not use bash for that. Filenames with spaces (lol) do not happen unless you do it yourself. Bash is useful for productivity when getting stuff done. For the work I do (where code never comes close to a customer and one-off tasks are common), if a candidate were to write a python script for the example line of bash I would be strongly biased towards rejecting them.

My real point about security is not that it is always in play per se, but that the naive Python will tend to do the right thing and the naive bash will not, and that's a non-trivial difference between the two. It goes beyond security. Generally as soon as you step out of bash into perl/python/ruby/whatever, you get an immediate boost to correctness of all kinds, unless you go out of your way to be sloppy. Yes, even Perl is an instant correctness boost.

As for file names not having spaces in them, after I said I personally don't use spaces in my file names,

    find -name '* *' | wc -l

on my home directory here yielded 12985. Uncompressed archives I'm not the source of. Dot directions I did not creat. Some tmp files. Probably the majority of my media files, which are the bulk of that number, named in all sorts of different ways that only share in common that I didn't pick them. An Anki directory. Yeah. They happen.

(Also, because I know someone's gonna try to go for the zinger here, I am not being hypocritical. I do not have an objection to one-off scripting jobs, especially ones like this that are read-only so there is no permanent consequence for error.)

> Filenames with spaces (lol) do not happen unless you do it yourself.

This is a cop out. File names can totally contain spaces, and the difference between the Bash and Python solution is that one will continue to work when a file with spaces inevitably shows up and one will fail, possibly silently.

What do you do that causes filenames to appear with spaces? And why do you not fix the cause instead of forcing yourself to deal with the possibility of a filename with a space in every possible situation?

This attitude is where it brings unexpected bugs in the long run. Don't expect but just be sure.

This is going to come down to what you're most familiar with. I personally can never remember what the arguments to `cut` are due to rarely using it, while I'd be able to write the equivalent python from memory.

Os.listdir isn't much harder to remember, and of course os.walk is actually probably easier for recursive filesystem actions.

One nice thing about using a pipeline is that it will be multi-threaded by default. That is each process will have it's own thread(s) and distribute any work across multiple cores.

Also with something like Gnu Parallel you can easily scale across multiple machines. Maybe there are more efficient approaches but a pipeline and gnu parallel is pretty easy to scale up without thinking much about it.

That's just one personal anecdote though. I can do the Python example from memory but not the bash example.

There's a great talk by bwk where he complains about exactly this kind of behavior...before admitting that anything useful in a language will be abused in order to do silly things.

https://www.youtube.com/watch?v=Sg4U4r_AgJU

I'll admit, I'm no fan of Python, but this has little to do with the language itself. I found Bash and Ruby first, so Python just seems like too much work to get right.

Requisite xkcd: https://xkcd.com/1987/

Reply on news.ycombinator.com