Comment by PeterWhittaker | Hacker News Reader

Comment by PeterWhittaker | Hacker News Reader

Parent Full thread

I started my career with a X term hosted on a SunOS pizza box, have written thousands of scripts, more than a few in the KLoC count.

Several years ago, I had to write a complex unattended robust data transfer system in PowerShell. (I know, I know, without more info these “requirements” beg many questions, but they are all out of scope for this reply.)

I enjoyed the experience so much I switched all my shells, on MacOS (my DD) and on Linux (my most common work environments) to PWSH.

What I liked most about it was the power of passing objects in pipelines, and being able to extract/manipulate some of the properties of an object in the first filter and still have access to others, along with those of objects created by that first filter, in filters later in the pipeline.

Immensely powerful.

The consistency of commands, of error handling, and of object properties was also very nice.

Eventually, as the nature of work changed I switched all shells back to bash, as the older muscle memory asserted itself. PWSH as a shell made sense when I was working and thinking so much in that space, but when I left it, it was more effortful to think in PWSH than to resume bash.

There are times I miss it. There is nothing else in the shell space that comes close, or at least not close enough to justify the effort of switching.

a few years ago I had to write a data integrity check for tens-of-thousands of files. Just crawl the dir and compute filename, size, checksum, maybe date, can't remember. I thought "ooh neat, I bet I can use some powershell trick to compute that tuple for each file", and no. Immediately ran into some obscure limitation with their "pipe" - I can't remember the details, it was like impossible to create many-things from a single-thing - and thought "how disappointing. the one time I give it an honest shake to do something entirely conceivable by the creator and it falls flat on its face.".

Nothing in your description sounds difficult to do in powershell. You can certainly output "many-things" from a part of the pipeline that takes "single-things" as input. Crawling files is a single command, then you can do whatever you want with each one in the next part of the pipeline - "map" from file info object to something else (e.g. custom object with filename, size, checksum, etc props) 1-1, multiplex each file into N output objects, buffer file inputs until some heuristic is met then emit outputs, etc.

A lot of years ago, circa Exchange 2010, there was a limitation on pipelines-within-pipelines which had to be worked around. Files already include name, size, dates, so select all of those and add a calculated property of the file hash using Get-FileHash (MD5 and SHA options):

    gci | select *, @{L='Hash'; E={ ($_ | Get-FileHash).Hash }}


    Get-ChildItem | 
      Select-Object -Property FullName, CreationTime, Length, 
                              @{Label='Hash'; Expression={ ($_ | get-filehash).Hash}}

but even then you could do what you want with a traditional loop and no pipeline:

    $results = foreach ($file in get-childitem) {

        # a hashtable of things you want to be
        # in each object ('row') of the output:
        $data = @{
            Name   = $file.Name
            SizeGB = $file.Length / 1GB
            Hash   = ($file | Get-FileHash).Hash
        }

        [pscustomobject]$data
    }

Thanks for taking my belly aching memory and being constructive. I just vividly remembered the sour grapes taste but I guess it was just due to my incompetence. I think what I didn't know was how to do the nested sub command for Hash. Cheers

Do you remember what the common commands for pipelining you used were?

I feel like my PowerShell struggles are really about not knowing 2-5 core, general-purpose pipelining commands well enough to use in any situation.

Not OP but the "-Object" commands are pretty fundamental to creating useful pipelines:

Select-Object - Pick out specific fields from an object, create calculated fields etc.

Where-Object - Drop non-matching objects from the rest of the pipeline.

Group-Object - Cluster objects into groups based on a shared property value.

Sort-Object - Order an array of objects based on a property value.

Get-Content - Read from a file.

ConvertFrom-(CSV/JSON) - Parse a json/csv formatted string into a powershell object.

ConvertTo-(CSV/JSON) - Serialise a powershell object into a csv/json string.

(Parallel)ForEach-Object - Loop over each item in the pipeline, performing one or more actions on it. Usually occurs at the end of the pipeline, or when you need to call an executable that can't handle pipeline input.

One thing I struggled with in the beginning, was not knowing what properties an object might have. You can pipe any object into `Get-Member` and it'll list its available properties and methods.

Many of the "-Object" commands support the use of script blocks if you need to carry out more complex filtering/projection.

Thanks for the detailed write-up! I'll give them a try today.

    |? PropertyName -gt 5            # where-object filter numbers
    |? PropertyName -match 'text'    # where-object filter text
    |? {$_.thing ...}                # where-object filter script

    |% PropertyName    # same as select-object -expandproperty propertyname
    |% { $_.thing }    # foreach loop
 
    |sort
    |sort prop1, prop2
    |sort {$_.Name.Substring(0,5) -as [int]}   # sort calculated thing

    |fl *        # format-list , all properties
    |ft -auto    # format-table, autosize table column widths

    |sv q        # set-variable -name q, same as $q = <...>
    |tee -var q  # same but show the results as well as storing in q

Helpful! Thanks!

Others have provided better answers than I could have, as I don’t remember clearly enough.

What I do remember is that until I understood what I was trying to do and why (mostly error handling around edge cases, all specific to the app), I couldn’t really grok the various object management calls, but that once I’d done a few rounds of PoCing and RTFMing, everything fell into place.

Get-ChildItem, Select-Object, Where-Object, Sort-Object or anything that is operating on a collection feel very natural. Often times I'm piping Get-ChildItem into sort to find the most recently created file in a directory.

Reply on news.ycombinator.com