Hacker News Reader

Comment by yuriko | Hacker News Reader

If you bother to write a python script to parse the integers, why not use python to solve the whole problem?

This is one of the many reasons I think PowerShell did UNIX philosophy better: you don't need to parse text because the pipelines pass around typed objects. You can kinda almost get the same behavior from some UNIX commands by first having them dump everything into JSON and then having the other end parse the JSON for you, but you're still relying on a lot of text parsing. Personally I think it is high time the UNIX world put together a new toolset.

Indeed, in PowerShell you could do:

    1..500 | ?{ !(Test-Path ('{0:0000}_A.csv' -f $_)) }

Explanation:

1..500 generates a sequence of numbers 1 through 500

| pipes the numbers

?{ … } is a filter that is evaluated for each item (number)

! negates the following expression

Test-Path tests that a file exists

-f formats the string left of -f with the parameters (zero-based to the right of -f

'{0:0000}_A.csv' is a pattern which formats parameter 0 as 4 digits, zero-padded.

EDIT: Explanation

Great example. PowerShell is definitely underrated. I don’t care to use it as an interactive shell, and that likely turns many off to it, but now that it’s cross platform I think it’s massively underrated for these sorts of tasks.

I don't think that example really shows the benefits of the "Powershell way". There's hardly any need for objects and what-not when solving an easy problem like this; strings will do just fine. With Bash, a close equivalent would be:

  for f in {0001..0500}_A.csv; do test -e $f || echo $f; done

Take a look at osh (Object SHell): https://github.com/geophile/osh

It is a Python implementation of this idea: OS objects like files and processes are represented in Python. You construct pipelines as in UNIX, but passing Python objects instead of strings. E.g. to find the pids of /bin/bash commands:

    osh ps ^ select 'p: p.commandline.startswith("/bin/bash")' ^ f 'p: p.pid' $

- osh: Runs the tool, interpreting the rest of the line as an osh command.

- ^: Piping syntax.

- ps: Generate a stream of process objects, (the currently running processes).

- select: Select those processes, p, whose commandline starts with /bin/bash.

- f: Apply the function to the input stream and write function output to the output stream, (so, basically "map"). The function computes the pid of an input process.

- $ Print each item received in the input stream.

Osh also does database access (query results -> python tuples) and remote access. E.g., to get a process listing of (pid, commandline) on every node in a cluster:

    osh @clustername [ ps ^ f 'p: (p.pid, p.commandline)' ] $

That said, that might not be the best example, because that same first command with Unix utilities is just:

    pgrep -f /bin/bash

OK, how about this:

    osh timer 1 ^ \
        f 't: (t, processes())' ^ \
        expand 1 ^ 
        f 't, p: (p.pid, int(t), p.commandline)' ^ 
        sql "insert into process values(%s, %s, %s)"

- timer 1: Generate a timestamp every second.

- f 't: (t, processes())': Take the timestamp as input and generate a timestamp and sequence of Process objects.

- expand 1: Expand the sequence of Processes, so that there is one per output tuple, e.g. (123, (p1, p2)) -> (123, p1), (123, p2).

- f 't, p: (p.pid, int(t), p.commandline)': Take the timestamp and Process as input, and generate (pid, timestamp as int, command line) as output.

- sql "insert into process values(%s, %s, %s)": Take the triples from the previous command and dump them into a table in a database.

Why replace your hammers, screwdrivers, and chisels just because someone invented a 3D printer? They have tradeoffs. Powershell has some good ideas, and benefits from having been invented altogether, rather than evolving over four decades. But in practice it's not as efficient for doing simple things. It's oriented towards much more complex data structures, which is great... but there's no need to throw out your simpler tools just because you think they look ugly.

For sure, if you are used to bash, use it, because you will ve productive.

But dont say powershell is not practical for simple things. Its very practical for simple and complex things. And I am productive with it.

They're full of footguns, esoteric behavior, and have arcane names. They're actually pretty awful tools now that it isn't the 70s anymore.

Yet when I visit the unix sysadmins' office I see people chaining commands to administer hundreds of boxes. On the windows side I rarely see powershell prompts. Powershell looks so much better in theory. However it's just an okayish scripting language with a good REPL. Unix tools are a far better daily driver.

That's probably because for daily tasks we have much better tooling in Windows already that doesn't require us to use the command line and interactive construct it. I can easily administer the configurations of thousands of computers through AD, for instance, and while I could use PowerShell to do so, using ADUC is just easier most of the time.

If you do a lot of work with Exchange though, you'll probably end up using PowerShell much more, because the web UI for it is not so great.

No matter what you think of the specific implementation, a lot of PowerShell's ideas are good ideas. Unfortunately UNIX culture is such that they'll probably never implement any of them.

> That's probably because for daily tasks we have much better tooling in Windows

ssh, docker, ansible, kubernetes, grafana, prometheus, etc... All coming from Linux/unix. This statement is clueless. Most of the cloud is not running microsoft, and for a good reason.

To automate, we have python, which has a much better syntax. It's pointless to use powershell.

And it takes a microsoft head, without knowledge of programming language's history to say that powershell's ideas actually come from powershell. Method chaining/fluent interface with a pipe instead of a dot does not look that new.

Also, some attempts have been to implement posh clones on unix. Being redundant with either perl/python or bash/zsh, none succeeded.

PowerShell sort of cheats, which enables the nice object pipelines; all cmdlets are .net modules that are run within the same runtime. That makes PowerShell much closer to "normal" programming languages with repls than traditional shells. That is also why PowerShell model is not directly a good fit to the UNIX world.

I would like to see more work done in the realm of object shells (and have some ideas myself), especially around designs that meld in more the UNIXy way of having independent communicating processes. But it is a difficult problem domain, and many approaches would involve rewriting lot of the base system we take for granted that is just huge amount of work.

PowerShell had the benefit of having stuff like WMI, COM, the whole .NET, and of course all the resources, funding and marketing from MS. Even then it has seemed to have been an uphill struggle, despite there being far more a need for PS in the Windows world.

Fair point! I'd argue there's a difference in time spent in writing a python script to solve it all, and just parsing the ints as I did. Python was my first thought for how to parse the ints.

Great article. I didn't know about comm actually, so that was new for me!

With regard to the 'ints' it helps to not think about them as ints but rather just some text that follows the pattern ^[0][1-9] or some equivalent to that. "Starts with any number of zeros or no zeros followed by any number of numbers between 1 to 9 or none at all."

So long as you know the repeatable pattern you can always use sed to just replace the part of the pattern you want gone with nothing which effectively deletes it from the output. sed is like a Swiss army knife in that regard because you can do nice simple deletions like that and even iterate on them if you need or you can do quite complicated capture groups if you need to as well. Sed can get you unbelievably far in terms of shaping text in a stream.

I have a few tricks I've learned with various tools that I thought were worth writing down. Hopefully you can find some more useful stuff.

https://github.com/nicostouch/grep-sed-awk-magic/blob/master...

Your regex got munged, but shouldn't it be:

    ^[0]*[0-9]+$

to avoid matching empty lines?

Oh I see. Yes the output got a bit messed up.

  ^[0]*[1-9]*

You could use + if you want. I only do it if not doing will produce incorrect output for my given input, which happens occasionally but is rare.

Bear in mind in this case there is also _A.csv or _B.csv you need to account for as well which your version doesn't. Mine will still pick it up because I'm not specifying the end, as I'm assuming I've done the necessary preprocessing steps to get good data so it will produce the expected output.

I'm not usually fussed on being that strict when I write regex as I tend to do various preprocessing steps like grep -v ^$ to filter out any blank lines if I don't want them in the stream etc.

Ya, I dropped in an unwanted $.

You can use numfmt to parse the number:

    $ seq -w 0001 0005|numfmt 
    1
    2
    3
    4
    5

Or just use plain sed:

    $ seq -w 0001 0005|sed 's/^0*//'
    1
    2
    3
    4
    5

yeah that was super weird. Why write an article about the merits of shell tools if you put some python in the mix...

Python basically acts like a subshell with its own language... I don't see why anyone would think unix shell scripts are really that different from Python scripts, especially if you're not doing the subprocess control things that command shells are optimal for. Invoking python to do something doesn't seem any more awkward to me than invoking sed, awk, etc..

Python has no bindings to the underlying OS (unless you import os modules, and even then it has its own way of working), and does not interact directly with environment variables either, and as such can not be seriously considered like a "shell" tool at the same level as the other ones. It just was never designed to act in this way. And in the embedded world you don't want to install a python interpreter if you can do without.

If you're just sorting numbers, you don't need the OS or environment variables.

Reply on news.ycombinator.com