Hacker News Reader

Comment by jolux | Hacker News Reader

I've been frequently derided for demoing it but I should write something about how to do similar tricks with PowerShell, as I have found it easier to learn than jq and an underrated tool in dealing with JSON.

The first example:

  ConvertFrom-Json $USERX | ConvertTo-Json | Set-Clipboard

Getting properties by name:

  ConvertFrom-Json $ORDER | select order*

View on HN

I haven't used pwsh in a few years, but does `ConvertTo-Json` still have that terrible `-Depth` parameter that you have to remember to set so that it doesn't mangle your JSON? A quick web search indicates that it does.

Edit: Yes, it does.

    '{ "a": { "b": { "c": { "d": { "e": 5 } } } } }' | ConvertFrom-Json | ConvertTo-Json

    {
      "a": {
        "b": {
          "c": "@{d=}"
        }
      }
    }

So you have to remember to set `-Depth 100` for every invocation of `ConvertTo-Json`. You also can't set it higher because it has a hard-coded limit of 100, so hopefully you never deal with JSON that has more nesting than that.

Yeesh. I like PS, but this one commandlet's design has always baffled me.

That API is being reworked now at least https://github.com/PowerShell/PowerShell/pull/11198

I was wondering about that, because the docs say it uses Json.NET, which while an excellent library is being replaced with System.Text.Json in the BCL/FCL as far as I know.

Nu shell is also really handy for these types of format conversions (and querying):

  > echo '{ "a": { "b": { "c": { "d": { "e": 5 } } } } }' | from json | to toml
  [a.b.c.d]
  e = 5

Yep. Love powershell, disliked this decision, but now understand it: objects returned from native powershell commands have a loooooooooooooooooooot of nesting. Try:

  gci | select -first 1 | ConvertTo-Json -Depth 2
  gci | select -first 1 | ConvertTo-Json -Depth 3
  gci | select -first 1 | ConvertTo-Json -Depth 4

one at a time.

In the words of Raymond Chen, adding a `-Depth` parameter is using a global solution to a local problem. If certain types should not be serialized beyond a certain depth, the config should be per type, not on the whole serialization.

Or leave it as it is and expect the user to map the complex values into hashtables. Eg in your example, make the user pipe the objects through `%{ @{ Name = $_.Name; Mode = $_.Mode; } }` first.

In any case, I don't know about other PS users, but all my command-lines that ended with ConvertTo-Json either started with ConvertFrom-Json (transforming an existing JSON file), or started with HashTables (building a JSON file from scratch) or a mix of the two. Therefore I always wanted everything to be serialized, and the `-Depth` parameter was always a nuisance.

Thankfully it looks like -Depth is going to default to 64 when the cmdlet is ported to System.Text.Json, per voxic11 downthread: https://github.com/PowerShell/PowerShell/pull/11198

True, but isn't it a good idea to have an "upper limit" global value too in case of badly behaved types?

Yes, a limit per type. Not for the whole serialization.

But what if a type which you need to serialize wrongly doesn't set the limit?

EDIT: I see your point now, thank you for explaining

Again, I'm not saying there shouldn't be a default limit. I'm saying the limit should be per type, not for the whole serialization.

It can be something as trivial as "PSHashTable is allowed to serialize without limit. Every other type is limited to N levels, unless the user sets `$SERIALIZATION_LIMIT[System.Type]` to some other value." Or make the `-Depth` parameter a `Dictionary<System.Type, int>`.

Why? Both JSON strings and their deserialized value's memory usage grows only linearly if you nest stuff (unless you're doing something dumb).

By that logic they should also add -MaxArrayLength to make the parser stop parsing long JSON arrays, -MaxProperties to make it only parse object properties up to a limit, -MaxStringLength to only make it parser strings up to a certain length...

It's completely pointless.

I suppose it is only a concern during serialization, not deserialization.

It turns an object hierarchy into flat text. References in the object hierarchy can be circular, and the Depth parameter limits the infinite loop a circular reference would cause.

This can't happen. JSON does not have references.

I believe he’s referring to an arbitrary object being converted to JSON.

From-JSON wouldn’t have this concern, but to-JSON would

Yes. My mistake.

>var obj = {}; obj.foo = obj; JSON.stringify(obj);

>Uncaught TypeError: cyclic object value

If browser JS engines can detect circular references, the PS serializer can too.

it CAN, but it DOESN'T.

Detecting circular structures (sometimes just called graphs), is a solved problem. The Lisp reader and printer even serialize them correctly: #1=(42 . #1#) makes a pair of two items. The first item is a 42 and the second is the pair itself.

It sounds like whoever implemented this just didn't know what they were doing.

Most of PowerShell is underrated when it comes to pipelines in shell.

I've completely rid myself of GNU core utils with just pwsh.

why? are you forced to work on windows?

Iirc powershell is cross platform.

PowerShell is supported on Linux and macOS as well [1]

I completely agree with the parent that PowerShell blows the existing shell options out of the water. Having pipelines with objects rather than just text makes everything so much easier. Instead of spending an hour futzing with awk or regex or applicatioons like jq to parse values from command results you can just access what you want directly and get on with your work.

1. https://docs.microsoft.com/en-us/powershell/scripting/instal...

I'm glad you're productive in your CLI tool usage but just because it takes you an hour futzing with basic *nix tools doesn't mean it's the same for everyone else. Powershell is extremely cumbersome for me due to the sheer verbosity of each command and the apparent MO of "let's come up with a new name for everything" and "sensible defaults? what's that?"

Yes, awk and jq “just work” for me: whether it’s because I’ve trained myself through constant usage or whatever, I’ve yet to understand the supposed benefits of a richer IPC mechanism than byte streams.

I just don't want to have to type that much physically, let alone the verbose commands and slightly odd syntax you often have to memorize. I know some love it, but I'm much happier with just using bash everywhere, even though msys/git bash on windows has its' own quirks, WSL ftw though.

You don't have to type the full commands. Most installs come with a full list of aliases: https://pastebin.com/eHsTUaRQ

For instance on my system:

   Get-ChildItem -> gci or ls


   Select-String -> grep

With tabcompletion you do not need to have aliases, but you need to name your commands in a sane way, having the meaningless words forst like powershell does is really cumbersome for tabbing. Examples: Where-, Get-, Remove-, Export-, From-, Select-, Invoke-

Nice list of aliases though.

Menu completion is even better if you haven't tried it!

FWIW I'm talking about CLI, I have little experience with menu completion, but how does menu completion solve the Get-<TAB> problem it will give too many hits and parsing that will take too long. Menu-completion in Intellij, and it's ilk, is more about discoverablity than fast typing then you can take your time parsing the output.

PowerShell is, in my opinion, quite discoverable and usable with a few commands to find your footing. I don't memorize a lot, because it's so well-structured and intuitive for the most part that once you get the basic structure it's easy to hack around with the docs.

Three commands that are useful to memorize, though, (particularly if you're having trouble remembering names) are Get-Command, Get-Alias (also with -Definition), and Get-Member. Get-Command gets you info about a command, like if it's an alias or not, and the path if it's a unix command. Get-Alias shows you all the active aliases, and Get-Alias -Definition shows you the active aliases for a given command. Get-Member shows you all the members of an object which can help you with Select-Object and Where-Object and so on.

No, but I'm a gamer at my core and I develop on the JVM so I have no need to interface with the kernel. Windows just works for me.

One difference in usage of PWSH I can't wrap my head around is when I want to do something akin to:

    grep -rl $pattern | $othercommand

So far I've got this (deliberately avoiding aliases for this example) in the above grep's command's stead:

    Get-ChildItem -Recurse -Path * | Select-String $pattern

But I just cannot fathom how to break from the table format to pass the lines to $othercommand.

How does one pass a pwsh list to a single command? Using %{}/For-Each {} wouldn't work because it would invoke $othercommand for each line.

Everything in powershell is an object, but sometimes the output format doesn't make it clear. Select-String basically returns a match object with some nested results, but the output formatting makes it awkward to see. Pipe it to

  | format-list *

to investigate it, and you'll see a bunch of members. In this case if what you want is just the matched string, you want to expand the `Matches` property of the object, then expand the value property of each of those. In this case the term 'expand' is key: if you just use `select` it'll return you the property, name and all. Use `-expand` to return the plain array. So something like:

  Get-ChildItem -Recurse -Path * | Select-String $pattern | select -expand matches | select -expand value

which you can then use how you like. An alternate form is

  (Get-ChildItem -Recurse -Path * | Select-String $pattern).Matches.Value

because the . notation for member access works on all items in an array (Matches in this case).

Since Powershell isn't working on text but on objects, you might just want to use grep. Anyways, this should reproduce the output of grep in powershell:

  Get-ChildItem -Recurse -File | Select-String "$pattern" -List -SimpleMatch -CaseSensitive | Select-Object -ExpandProperty path

You will want to only search in files for this to not ouput some lines of InputStream if there is a match in the actual path. If $pattern is an actual regex you will want to drop SimpleMatch. In case you want some proper powershell objects you might want to pipe this into Get-Item.

I don't think I entirely understand the question, is $othercommand doing something with all of the lines together?

Yes, for example other command may be (and often is) fzf followed by yet another command to exec on the chosen file.

If it only wasn't dog slow and chewed up memory when working with large files. Large files often force me to go back to cygwin.

Idk that's just not something I do with the shell. At work I process a lot of large files, but typically those require hand written code to use off heap or file/db backed collections (I work in the JVM).

You definitely should! Is it alright if I copy this comment onto my blog underneath this post in the "comments" section?

If you do write such a post make sure to let me know so I can link it from mine! You could write a response "here's how I'd do all the stuff in that post in Powershell with no extra tools" that would be very cool :)

I'll try to remember!

Reply on news.ycombinator.com