Hacker News Reader

Comment by mauvehaus | Hacker News Reader

That's nice. Can you pipe binary data without powershell corrupting it, or is that now a feature?

https://brianreiter.org/2010/01/29/powershells-object-pipeli...

It's amazing, isn't it. They went to the effort of building a typed shell but didn't realise there are multiple different types of string.

I bet it still truncates output of the ">" operator to the console width, too.

    'troll' * 1000 > out.txt

Open in NotePad++, no weird newlines in the wrong place, content not truncated.

Were you using |format-table and |format-list (ft, fl) to explicitly format the text for console viewing, then redirecting that to a file?

Took a while to reproduce the actual case, and it's Select-String that's at fault. And it must be in "powershell" not "powershell ISE".

  'troll' * 1000 > troll.txt

.. works fine as you say

  Select-String -Path .\troll.txt -Pattern "t" > t.txt

.. wraps the output at the console width. Which means you can't use it as a grep replacement.

You can't use it as grep in bash, because it isn't that. It outputs [MatchInfo] objects, not text, and Out-File (>) formats complex objects for console viewing (for ??? reasons), which is one reason > isn't a great PS habit.

    Select-String -Path .\troll.txt -Pattern "t" | Set-Content t.txt

    or

    sls 't' .\troll.txt | sc t.txt

this ^^^

This is the main reason I still use cmd.exe

Related, running something like decrypt hugefile.dat | encode outfile.dat will usually result in powershell using up all your system ram and crashing.

On a somewhat related note, one of PowerShell's biggest weaknesses is memory utilization. And since they make it so simple to pass around hefty object collections it tends to bite you in the ass early on. Particularly when you're using cmdlets. When doing real scripting that actually needs to be run as a job I am surprised when I can get away with cmdlets and don't need to manage .net APIs/objects directly. Especially third party cmdlets.

One performance tip I picked up early on: If you're passing around large object sets you need to operate on try to keep them in a pipeline. Populating a variable and interacting with it is arguably more readable and maintainable, especially for junior people, but it has a heavy cost in terms of memory.

That said every version of posh is better than the last so I may be wrong or overestimating the impact for the newest versions. YMMV, don't believe everything you read on the internet, etc.

That is a good tip. There are certain product teams that produced ENORMOUS objects and your suggestion helps when dealing with that.

Jeffrey Snover [MSFT]

Upon reading that again maybe I should have said it's weakness is that it's easy for everyone involved to create objects that use a lot of memory. I'm not sure it's fair to say PowerShell itself is bad at utilizing or managing memory. It's more of a possible negative side effect of one of its great strengths.

That's great advice, man. Cheers.

Most likely because it loads it all into memory, rather than say streaming it.

That's how I would expect MSFT to build it.

This bit me badly when trying to pipe GPG output. I ended up building a command string and sending it to cmd. :|

That is odd, powershell shells out to cmd.exe if you try to pipe binary or text data. It goes back to powershell only if you use powershell commands in the pipeline.

If you don't pass the flags zo handle it as binary in those cases stuff breaks.

I found out yesterday that `Select-String`, PowerShell's "grep equivalent" expands tab characters to spaces.

Haven't tried in the preview build but this is still the case in the recent Windows 10 build I have.

Powershell drove me to Mac

Get-Content test.jpg -Encoding Byte -ReadCount 1000 | Set-Content test2.jpg -Encoding Byte

Piped binary data, not corrupted, since a long time before that blog post was written.

What you can't do is run native commands, and read their standard output as a byte stream without going through .Net and Start-Process.

Was super fun to learn about this when restoring the "backup" dumped via pipe in a powershell script.

While the first rule of backup is "do them", the second rule is "test them". If your backups are bad and you only notice it when it's time to restore, you only have yourself to blame.

"The backups fail to restore" is such a common occurrence, it's really scary.

Reply on news.ycombinator.com