Hacker News Reader

I don't entirely agree. With traditional shells, you can still use objects, you just have to serialize them to text. jq is perfect for this, you can use it to store a serialized object in a variable and then pull things out of it for use with other commands. Example:

    example="{}"
    example=$(echo $example | jq .foobar=10)

Later on you can pass example.foobar into another program like so:

    grep $(echo $example | jq .foobar) ...

You can write simple helper functions that make this more concise. The beauty of it is that this works with any data format - plaintext, json, csv, etc - you just need a tool to serialize and deserialize it.

My understanding is that PowerShell is passing around specialized objects that have a few required methods, and some source specific ones. The beauty of this is that you can pass massive amounts of data without serializing and unserializing it at every step, and also that they can contain links to other information, such as links to parent objects which may or may not be part of the same list (I have no idea if this is allowed, but it seems a logical extension of the concept). Additionally, dates passed around can be complex types that don't have to lose relevant info such as time zone, regardless of what the default display format is.

The downside is that tools need to buy into this and work with this data stream rather than text, which makes the set of available tools necessarily much smaller.

powershell is a nice concept and a pretty nice shell, but its also complex due to this.

serializing to text just means "everyone must speak the same language" its also complex due to this.

And in fact jq does serialize. i'd argue jq is a "one tool, one job" and thus unixy. Its the way the "one job" is seen that differs.

After using powershell an bash/etc. for years, i'm still unconvinced which one is the most powerful in the real world. sometimes (most often) its faster/easier to do things with bash, sometimes the convenience of powershell objects is nice (that's where i'd basically use python on linux because of bash limitations).

As long as i can parse, grep, etc with jq, and pass it to another tool via a text stream output, im happy. as a human, i find it easy to deal with unstructured or very lightly structured text. Hard to dig through complex structures.

I understand this, I myself often trade off between bash and Perl. Sometimes I'm using a for loop on a shell expansion to tar directories, sometimes I'm piping an ls to xargs to parellelize a workload, but often I'm piping SQL output to Perl, selecting the records I care about and storing (and/or altering) the relevant fields in a data structure, and outputting the data in a formatted way (such as SQL update/insert statements) or reporting some aggregate information.

What I see in PowerShell, which to be truthful I never use (it's not in my common environment), is a less powerful/extensible version of Perl, but optimized for a specific workload, as there's less parsing and manipulation of data required if it's easily available in the formats you need already. Honestly, I still wouldn't use much it if it was installed by default on all UNIX boxes, because Perl fills that need for me, but I recognize what it's trying to do on a conceptual level, and it's similar to what I end up doing every day, so I support that.

You have it backwards. Even though they are similar on a conceptual level to what you do everyday, doing it in Powershell requires buying development tools. And there hasn't been a need to support that monetarily for a while.

What development tools do you need to buy for Powershell? (unless you mean Windows itself).

hmm i do powershell stuff with "just" windows amd gvim.exe

just windows

> My understanding is that PowerShell is passing around specialized objects that have a few required methods, and some source specific ones.

I don't believe that PS objects need to have any specific methods. Maybe only ToString() - but that is always auto implemented.

> The beauty of this is that you can pass massive amounts of data without serializing and unserializing it at every step, and also that they can contain links to other information, such as links to parent objects which may or may not be part of the same list (I have no idea if this is allowed, but it seems a logical extension of the concept).

Well, that's part of it. The fact that there is no need for serialization/deserialization between the commands has some obvious performance benefits (and a few drawbacks). But it is also what enables the perhaps most overlooked but most important characteristics of PowerShell: You can use it in-process to manipulate in-memory objects of the hosting process. This means that you can implement the management functions as PS cmdlets, and still build an admin GUI using those cmdlets but where the GUI is a rich process with in-memory objects that is just passed to cmdlets. Think how a task manager could be written to list the processes on the machine by invoking Get-Process. The objects returned represent the actual processes - complete with methods and properties. The GUI can hold on to those objects, and when you pick one and choose the "Stop" action (for instance) it could pass it as input to Stop-Process (kill). This is actually what a number of product-specific GUIs from Microsoft do, e.g. Exchange.

> The downside is that tools need to buy into this and work with this data stream rather than text, which makes the set of available tools necessarily much smaller.

Yes - it is a new model, and the commands need to support it to take full advantage. There is a "bridge" from the old world to the new in that the "old" command types are simply seen as commands that consume and produce sequences of strings. But yes - there still is some friction. The model works at it best when you can do all of your tasks using cmdlets.

At some level I suspect that PowerShell has been blessed with a rather poor CLI ecosystem on Windows preceding it. It didn't have to knock anything existing from the throne; rather it hit a vacuum and was almost immediately adopted. The same wouldn't have been the case on Linux/Unix.

There is an impressive number of cmdlets available now, however. On my machine "gcm|measure" tells me that there are 1484 commands available. Those are only those distributed from Microsoft, and the cover all sort of administration, firewall, sqlserver, file transfer, disk management, low level network managent, print management, start screen, user management etc. I don't think it is on par with what the Linux/Unix community has available now, but it is getting there, quickly.

While jarring when I first used PowerShell, once I realized what it was doing, I immediately understood it as a powerful concept. It allows shell pipeline processing to work in much the same way you can filter lists of hashes/dicts/arrays in more traditional scripting languages. I haven't used it much, and not at all in the last few years as I'm not often on Windows boxes, but I've always remained impressed.

If you want to see another approach, jshon is much more unixy. (disclaimer, I am the author)

homepage: http://kmkeen.com/jshon/

Jshon is also a year older. I've never been able to figure out why/how Jq got so much larger so much faster.

> I've never been able to figure out why/how Jq got so much larger so much faster.

I offer this in the sense of constructive criticism: for me, it's because your examples are scary. Here's the command you give for reading a URL of a photo uploaded to reddit (in file foo.json):

    $ jshon -e data -e children -e 0 -e data -e url < foo.json

Here's the jq equivalent:

    $ jq -r '.data.children[0].data.url' < foo.json

which looks an awful lot like every other tree parser I've used. Now suppose I want to get all the URLs with jq? That's just:

    $ jq -r '.data.children[].data.url' < foo.json

In Jshon, I have to rewrite the query as:

    $ jshon -e data -e children -a -e data -e url -u < foo.json

That is, replacing the "-e 0" (get item at index 0) with "-a" (do the rest for all items). The jq equivalent changes ".children[0]" to ".children[]", which to my poor muddled brain more clearly represents what I'm asking for. For me, jq's query language maps well onto how I think about the problem. Jshon's does not. That's why I use one and not the other.

Again, I offer this sincerely. Even if I don't personally use it, I love when cool ideas cross-pollinate from one project to another and I'm glad there's more than one player in the space. Thanks for your work!

Hey, thanks for jshon. I've been using it a lot lately!

I have only recently realized the effect a well designed project website can have on the users impression of your project.

I'm sure the code/design quality of jshon is great, but the website design/layout doesn't reflect this.

The current look of your website looks like a website from the late 90's, and may as well implicate that the project is no longer maintained.

> This cmdlet is introduced in Windows PowerShell 3.0.

So... I can only guarantee it's included on my customers' computers if they have Windows 8 or higher. I'm happy most of them moved from XP to 7. Maybe I can start shipping PowerShell instead of batch files in 2020.

I don't think that's really the point. The point is that the unix command line has some serious shortcomings in its fundamental design. By extension he may be suggesting that maybe we should get off our asses and do something about it because windows has actually managed to ship something that's not only good, but better than any of the comparable options on *nix.

I don't see how it's better. You can send objects over pipes if you want to (in, for example, json format).

If PowerShell wants to be the bloated Word-equivalent of shells that's certainly a design objective. I'm sure someone will eventually do a Mono or systemd of it, saving UNIX from the tragedy of not working like Windows. But it smells of Taligent.

Does it smell like people, places or things? ;)

Agreed, GP's post was about the technical merits of PS. PowerShell may be the greatest thing since virtual desktops, but that doesn't help me until PS 3.0 is included in the OS by default. Since my customer base is decidedly conservative on OS upgrades, it's going to be a while until I can try out any of those great PS features.

PowerShell 4 for Windows 7: http://www.microsoft.com/en-us/download/details.aspx?id=4085...

But it is not installed by default, which means you'll have to tell your users to find and install it first.

Package a bootstrapping batch script that installs the MSI, maybe? (I don't know your clients' requirements, obviously.)

But Powershell builds on top of .NET, which means you can't write cmdlets in non-.NET languages. And writing them even in C# is quite painful. Which is why there is virtually no ecosystem for Powershell cmdlets and there is a gigantic, ever expanding, ecosystem of Unix shell tools.

At the end of the day you have to choose a primitive and history has shown that text is the right one.

You can write them in PS itself, and it is quite simple and minimal.

PowerShell cmdlets can be written in PowerShell. PowerShell modules (the unit of distribution) can be written in entirely in PowerShell. You can also write them in a .NET language, like C#, F# or VB.NET.

I disagree that it is painful to write cmdlets in C#. A cmdlet is just a class, the parameters are just properties annotated with the [Parameter] attribute. The shell does all of the parameter parsing, matching and type conversion/coercion. If I wrote a traditional commandline tool I'd have to do that parsing myself. Yes, I could use a library function like getopt for that - but the code and complexity would still be there.

But my point was not to get anyone to use PowerShell on Unix/Linux. PowerShell draws much of it strength on Windows from the fact that there are already well-established object models on that platform (COM, .NET and WMI/CIM).

The point was that the age-old text-only view in command shells is beginning to show some shortcomings. XML and JSON (among other formats) challenge the tabular view of the world. Sometimes you have to experience the alternative to realize that you have been living in a restricted world. Like swallowing the red pill. Who know, maybe an object-oriented shell that mixes well with the Unix tradition will emerge.

> Who know, maybe an object-oriented shell that mixes well with the Unix tradition will emerge.

This doesn't make any sense though. Unix has a tradition of language diversity where Windows has a tradition of using whatever Microsoft says to use.

Unix has language-specific shells but like Powershell they haven't really caught on.

PowerShell's "pipeline" is largely analogous to chained methods in a scripting language's REPL, and therefore is not the same as Unix pipes, which are used to share data streams between processes.

Unix already has the object functionality you see in PowerShell: Python, Ruby, the JVM, etc, and all of these have existed for far longer than PowerShell. The only thing PowerShell has is a bunch of shortcuts for system operations, which Unix already has in the form of POSIX shells and utilities, and which many languages provide language-native abstractions for.

Depends how you look at it.

jq is a "Unixy" tool in the sense that it should do one thing and do it right - and that 'one thing' is parse json data to output it in the desired format.

Awk, arguably one of the oldest unix tools, also has its own language. Would you consider it non-Unixy or is it grandfathered in due to age?

Awk does one thing well, it runs Awk.

Awk can be used as a unix pipeline tool, or it can be used more as a stand-alone that you make scripts/programs in to do larger tasks.

Similarly, Perl falls under the same category, being an awk/sed replacement. But considering Perl just a unix pipeline tool is also severely underselling it.

I'd like it better if it was more like csvkit, with a separate jqgrep, jqfmt, jqsort... that you combine using the shell. Because of course everybody already knows the shell.

let's see:

jq does JSON parsing. And does it really well. Check.

jq does filtering jq does projection (. operator, .[] operator, select etc) jq builds objects (the {} operator) jq does arithmetic (addition, subtraction, division, multiplication) jq does database-like "lookups" in collections (keys, has, length etc) jq does pipelining of the in-tool "objects". jq does object generation of new objects like range operators etc. jq does data type conversions jq does sorting, grouping, extracts min/max etc. jq does string interpolation and string formatting jq does equals and relational comparisons, tests and other boolean tests.

awk does many of these too.

bash has some boolean tests, but typically very filesystem or envvar centric.

In general, when using shells and tools:

Sorting should be handled by sort. Except when it is handled by jq or by ls or by find or by ps or ....

Filtering should be handled by grep. Except when it is handled by ls or by find or by ps or by jq or by awk or ...

Formatting is always handled by each and every command. Which is understandable given that the format of each line is so important.

Now consider PowerShell where ConvertFrom-Json does one thing: Converts text from JSON to objects. Where-Object (aliases ?, where) does filtering. On any type of object. Sort-Object (alias sort) does sorting. On any type of object. Select-Object does projections. On any type of object. Format-Table, Format-List and other Format- cmdlets formats objects to human-readable output.

The text protocol does not scale well with structural complexity. It is really only good for lines of text. It has been somewhat enhanced to be able to delimit fields which then makes for an informal tabular-like protocol.

The object protocol scales with structural complexity. That's the difference.

It's "Unixy" in that it has a short cryptic name!

Some times "one thing" happens to be a big thing that you have to do 100% or not at all, and doing it well requires a hell of a lot of complexity.

Does a XSLT processor do "one thing": processing XML? Does a DSSSL processor for doing "one thing": processing SGML? (Hint: It has a Scheme interpreter built into it!)

You raise some interesting points.

As I said, jq is impressive. It is a JSON processor that implements it's own functional language, complete with variables, function definitions, arithmetic, recursion. There is no question that it is useful and will be the absolutely right tool for a number of jobs.

Likewise with awk. I really, really like awk. It is such an elegant solution to a common problem (text parsing and productions). It is also a functional language.

XSLT is also a functional language. An XSLT processor is expected to implement a standard (as opposed to jq and awk) - but that's a minor difference.

It is when you are required to blend the disciplines that the limitations sneak up on you. In powershell it works the same regardless of whether I process data from JSON, from XML, from line-oriented text, from CSV, fixed columns or or from database queries. Sorting, grouping, comparing, mapping, presenting, converting - it is the same skill set.

That is not so say that something like PowerShell can or should replace tools like awk, XSLT or jq. It should not. Period.

But for tasks where one has to use several formats, the lack of a sufficiently strong inter-command protocol causes the tools to bloat and include functionality that has nothing to do with the core task. Consider, for example, why jq needs a "@csv" function (called a "format string") - and @html, @sh, @base64!

My claim is that those functions have nothing to do with JSON processing.

The equivalent in PowerShell is handled by ConvertTo-Csv, ConvertTo-Html, Out-String or the [Convert] class.

It really comes down to a n*m problem. jq is just one tool that implements csv or html formatting (m formats). There are n tools (jq, awk, xslt, ...). Should they all use the same rationale as jq and implement csv formatting? Or should they stay away from that and just concentrate on the core task.

In PowerShell the equivalent of jq - the ConvertFrom-Json cmdlet - just converts from JSON. Regardless of whether the objects were converted from JSON, line-oriented text, database queries, fixed columns, XML - the output formatting is handled by tools that are optimized for that.

If I'm dissing anything it is the limited inter-command protocol of traditional shells. In this day and age we process complicated structures that are not easily expressed in line-oriented plain text.

PowerShell is a .NET CLI and is more comparable to a JVM CLI (or the Python, Ruby, or Node REPLs) than it is to POSIX shells.

Unix pipelines are for streaming data between processes using standard input and output. PowerShell pipelines are for sharing data between .NET classes called cmdlets and are more like chaining methods within a programming language. There are multiple programming languages with this functionality available in the *nix ecosystems.

Working with the PowerShell "pipeline" is like working in a CLI/REPL for Python, Ruby, Node, not like working with Unix pipelines.

I'm wondering how PowerShell's object model compares to other scripting languages. What's a good introduction to PowerShell's object model?

type

    help about_objects

(or go to the online version at https://technet.microsoft.com/en-us/library/hh847810.aspx)

Follow the pointers to the other topics from there.

well, JSON is a serialization format -- so I'm genuinely curious what you mean. Can you better describe the dichotomy you're talking about (text vs objects) and give an sample problem that demonstrates a weakness of text ?

The http protocol is just text when you look at it. But I cannot just send "text" to a web browser or server. I have to follow a protocol. When I do so, I can anticipate how the receiver will understand the "text" - and plan for it.

The same way with shell tools - they input/output byte streams - but by convention we expect those to be (mostly) text. But there is no *protocol there. One tool cannot build a text stream in anticipation of how the next tool will understand it.

Hence you can only standardize on a very basic level. In traditional shells that basic level is text lines and sometimes space or tab delimited fields of a line.

There is no common way do describe a tree of nodes or a graph of objects that you can expect all of the tools to understand.

Sure, you can use xml - that's just text. Or you can use json - that's also just text. At least they can describe compound values and trees. But other common tools simply do not understand those formats.

PowerShell outputs objects - i.e. the pipelines are delimited so that the elements are objects. Each object can be arbitrarily complex. Each tool understands that it may receive complex objects - but can carry out their task regardless. Example: select (projection akin to the map function), group, sort operates on objects. Even if you sort the objects - the output will be the same objects. If you select properties using the select cmdlet, the objects that are the properties are passed on - and they can be arbitrarily complex as well.

To sum up: PowerShell has a higher-level protocol for pipelines and standardizes an object model (duck typing) so that tools can produce and consume trees, graphs, dates, customers, services etc.

PowerShell is cool, I wish it was more popular. I also wish that XProc was more popular too, but XML is a big no-no on the Web nowadays.