Hacker News Reader

Thanks for this, a very interesting insight into the Windows ecosystem.

I won't be able to speak with any authority on Windows systems and I certainly don't want to get into a language/OS war but I would like to point out a few issues that raise red flags for me:

* "Powershell is more UNIX than UNIX"

Again, I don't have enough knowledge to really speak on PowerShell specifically but I will point out that the shell is one aspect of a Unix system, even if it's one of the bigger ones. The concept of files, filesystem, memory, in addition to the ecosystem of tools, are part of Unix. Obviously Windows has solved these aspects in some way shape or form but, at least from my perspective of the outside looking in, Microsoft has re-invented a lot of the Unix philosophy in order to make a Windows box behave more like a Unix box.

* The mess of 'simple' tools vs. a cohesive philosophy in Windows

While it's true a lot of the tools in the Unix ecosystem have arbitrary names and non uniform output syntax, the diversity of tools allows for a large range of solutions to a given problem. As you point out, the hodge-podge nature is most likely a historical artifact of the Unix evolution but I would argue this is kind of a natural by-product of systems that are heavily used and improved.

I have no doubt that there are some cleaner interfaces in PowerShell but my concern is the reason for the clean interface is because it's a walled garden with Microsoft as the gate keeper. I think this gets into the old conversation about quality of code for FOSS projects vs. commercial projects.

It's my belief that, after a certain threshold, having a quantity of software to choose from will lead to better quality software overall compared to software that has less quantity but higher proportion of quality. I think this might be a contentious view but my opinion is that it's better to let many "amateurs" experiment, with the best tools from those experiments filtering to the top, than to rely on a few experts to solve the same problems.

Maybe there's a stronger economic argument as it relates to FOSS, I don't know.

I will admit that I have a lot of sunk-cost/first-language-bias that makes me stick with Unix. If Microsoft ever open sources their operating system, I might reconsider that position though.

> I have no doubt that there are some cleaner interfaces in PowerShell but my concern is the reason for the clean interface is because it's a walled garden with Microsoft as the gate keeper

This is simply not true. Many third-parties have powershell modules to interact with their product. Anyone can write them.

You can't say that the "hodge-podge nature is most likely a historical artifact of the Unix evolution" and then say that it is "quality". You can't have it both ways. Random, unpredictable, historical quirks do not add up to quality. They add up to a mess.

It's like pointing to a shanty town and saying it is "quality housing" because it has a high population density.

I'm reminded of a recent interview with Jim Keller, who's famous for being one of the architects of the x86 instruction set, Apple's ARM CPUs, AMD's Zen architecture, and Tesla' AI chip: https://www.youtube.com/watch?v=Nb2tebYAaOA

An interesting throwaway quote, but actually a deep insight, is that he thinks a lot of things should be "redesigned every 5 years or so", but unfortunately are redesigned only every decade or longer.

I totally agree. PowerShell was a clean-slate design, and was called "Monad Shell" originally. It's elegant, and is based on a cohesive concept. It's now 10 years old and starting to accumulate... inconsistencies. Warts. It's in need of a clean-slate design again.

Bash and the GNU tools are the most random mess imaginable that can still function. Its components can trace their roots back to the 1960s. Ancient code that should have been rewritten from scratch a dozen times over. Like you said, it's "evolution", but evolution gave us the trigeminal nerve and the inverted retina. I want a designed system that doesn't take the scenic route and have things back to front because it was "always done that way and is too hard to change now".

If you got sat down and told to come up with a set of composable command-line tools for a shell, the end result would have nothing at all in common with the bash/GNU tools. There is nothing there worth repeating.

I've seen that interview before and that quote in particular resonates with me but at the same time there are basic economics to consider.

I want to be clear, what I meant by 'quality' above is functioning systems. If there's a hodge-podge solution that gives you a 10x improvement at the cost of 1/2x the productivity vs. an "elegant" framework that gives you a 2x improvement at 1/10x the productivity by requiring to re-invent all the tools, the 'hodge-podge' solution wins.

To me, this is the same basic argument of Dvorak vs qwerty keyboards. Dvorak might be a 1-2x improvement over qwerty but qwerty wins because of cultural momentum. The improvement isn't worth uprooting the infrastructure already in place.

I think programming languages suffer the same fate, where the size of code base, libraries, etc. secures it as the lingua franca even if there are other languages that are marginally better.

As a rule of thumb, I think a replacement needs to be at 10x the improvement before it has a chance of uprooting entrenched systems.

The Unix system might be chaotic but it's had a lot of time to adapt to different needs. There's also a culture of freedom, sharing and experimentation that isn't present in the Windows world. These all have real world implications on what tools are available for experimentation, use and improvement.

> If you got sat down and told to come up with a set of composable command-line tools for a shell, the end result would have nothing at all in common with the bash/GNU tools. There is nothing there worth repeating.

Sorry, no. This sounds like an emotional plea rather than anything rooted in critical thinking. Sorting text files isn't worth repeating? Searching text files isn't worth repeating? Piping text from one process to another isn't worth repeating? I've tried to be even keeled in my responses but this is ridiculous.

> UNIX commands are more an accident of history than a cohesive, coherent, composable design. PowerShell was designed. It was designed by one person, in one go, and it is beautiful.

I'm curious about that design and would love to learn about it.

Note, that I'm not interested in learning how to actually use powershell as a developer, but understand the abstraction design choices, that went into it.

Every time I tried to learn about that, I'd find recommendations about Books with 1000+ pages and similarly time consuming resources.

Do you have an recommended read for understanding the design choices, that went into powershell, while not learning the details, that a everyday windows would need?

Things I care about: -How do Lamdas work? -Arrays are Objects. How exactly? -Typesystem -Are there native wrappers for the kernel APIs how do they fit into the objectmodel? -Since to my understanding no shortflags exist, how is verbosity avoided? -How thight are interfaces defined. If a tool provides a new interface in a new version, can old tools still work with it?

There are no good references other than a handful of blogs from over a decade ago. Google "Monad Shell", which was the original name during which the formative ideas were still congealing.

Generally, PowerShell simply embraces the UNIX concept, but with a clean slate. Being based on the .NET runtime, it inherits much of its low-level functionality, such as being able to call into WMI, COM+, and Win32 APIs.

But what it's really about is keeping things in strongly typed objects until the last second when they are formatted for display. The original UNIX pipes formatted output immediately, because C is not object oriented. So everything ends up as a byte/char stream, and then to glue commands together you often have to parse & split the output. There's ways around this, but it's inconsistent and not at all general.

So whereas in bash you would see only 2 or 3 commands in a row with a pipe symbol in between them, in PowerShell you would often be able to chain 10 or even 20 commands in a row in a single pipe. You build up pipelines almost like SQL queries, which is easy because the data passing between each step is structured.

The other big win that I see over legacy shells that got built up over decades is the discoverability and consistency. The naming is consistent, and all the commands publish detailed metadata that integrates with both the help system and tab complete.

Issues like verbosity are simply sidestepped. PowerShell is verbose to read, which makes it readable. Typing it is fast, because you can tab complete not just the command names, but the parameter names, and even the parameter values in most cases. No other shell can do this. It has aliases, so if you prefer 'ps', you can use that instead of the canonical 'Get-Process' command name.

Generally all commands simply extend a .NET class called "Cmdlet", which defines functions such as Begin(), Process(), and End() that handle pipeline input. Declarative parameters can be defined, which the shell engine hooks up for you automatically. For calling external APIs, you can do whatever .NET can do, which is pretty much everything in Windows and Linux with only a handful of exceptions (no inline assembly language).

In terms of interfaces, it's a little bit too weakly typed for my taste. When binding parameters or pipelines, the engine goes through a complex process of figuring out what you meant. It can go wrong. The classic example is that some search or query commands can either return a single object or an array, and making this consistent is a bit of a pain.

It supports lambdas, they're simply a block of code with squiggly brackets around them, e.g.: { echo "foo" }. You can do all sorts of fun things with them, like capture a closure with simply: { ... }.GetNewClosure()

Install PowerShell 7 on Linux, type in {}. and press tab to cycle through the list of functions you can call on an expression block. I like toying around with things like .Ast. The engine exposes some crazy powerful things directly, such as being able to generate code for remote RPC wrappers automatically, etc...

Yes, it's not particularly difficult either, as long as you have the appropriate tools installed.

    csvjson path/to/infile.csv \
        | jq -c ".[]" \
        | while read LINE; do
            UNAME="$(echo "$LINE" | jq -r ".uname")";
            CNAME="$(echo "$LINE" | jq -r ".cname")";
            if id -u "$UNAME" &> /dev/null; then
                # Find processes with this command name belonging to this user
                comm -12 <(ps -C "$CNAME" -o "pid=" | sort) <(ps -U "$UNAME" -o "pid=" | sort) | \
                    while read PID; do
                        # rss=Resident Set Size, i.e. the amount of memory this process is currently
                        # using, not including swap. You might want vsize instead, depending on your needs.
                        RSS="$(ps -o "rss=" "$PID")";
                        STARTISODATE="$(date --date="$(ps -o "lstart=" "$PID")" --iso-8601="seconds")";
                        # If you actually plan to use this you need to make sure pid isn't anything important
                        kill "$PID";
                        jq -n -c \
                            --argjson "pid" "$PID" \
                            --argjson "rss" "$RSS" \
                            --arg "start" "$STARTISODATE" \
                            --arg "command" "$CNAME" \
                            --arg "username" "$UNAME" \
                            '{"pid":$pid,"memory_kb":$rss,"start":$start,"command":$command,"username":$username}';
                    done
            else
                # Print an error to stdout if the user does not exist
                echo "warning: user $UNAME does not exist" >&2
            fi
        done \
        | jq -s 'sort_by(.memory_kb)' \ # sort_by(-.memory_kb) if you want it in descending order
        | in2csv --format=json \
        > path/to/outfile.csv

I'm guessing you meant "parse the CSV correctly using only bash", and I agree that's difficult, which is why you should not do that. Instead, you should use a tool that has been specifically designed to do that one task, and do it well (in this case, that task is "convert a csv into a format that's easier to work with, namely json"). This is the unix way.

How extensible is PowerShell? With bash, you can write a new command that takes text input in a certain format and produces text output in a certain format, in whatever language you want, and plug it in in the appropriate place in the pipeline. Is the same true of PowerShell?

I am properly impressed. Wow!

Most people I give this challenge to fail to account for the complexities of even a simple format like CSV, and typically simply use "grep" to find processes, which would cause havoc with simple user names like "Bob Ash = bash".

For comparison, the equivalent in PowerShell is this:

    #require -RunAsAdministrator
    $i = ConvertFrom-Csv @'
    UserName,ProcessName
    DOMAIN\BobJones,explorer
    DOMAIN\Bash,notepad
    '@
    
    $ps = Get-Process -IncludeUserName
    Compare-Object $ps $i -PassThru -Property UserName,ProcessName -ExcludeDifferent -IncludeEqual  `
        | Stop-Process -PassThru `
        | Sort-Object PeakWorkingSet64 -Descending `
        | Select-Object `
            ProcessName, `
            PeakWorkingSet64,`
            @{ Name='StartDate'; Expression={ '{0:o}' -f $_.StartTime } } `
        | ConvertTo-Csv

Note that it's easy to substitute "Import-Csv" and "Export-Csv" to deal with CSV files instead of CSV formatted text.

It's also a trivial extension to convert this snippet into a "ps1" script that can take formal parameters as input from anything that has those two columns, not just CSVs. And those parameters can be invoked manually, validated by the engine (not code!), and so forth.

What I hope this PowerShell version shows off is just how "neat" it is compared to bash. It's just a pipeline of little processing commands: "get,compare,stop,sort,select,convert". As impressive as your bash solution is, it's... messy. You were forced to write manual loops, use JSON for processing CSV, and do all sorts of admittedly pretty fancy trickery. Ask yourself: Could a junior tech make changes to your script and not break it? Could they even read it? I know I certainly couldn't, and I know more than a little bash!

As to how extensible PowerShell is: More than bash!

The "shell API" is just a byte stream in, and a bunch of byte streams out. That's it.

The PowerShell API is very rich, but it's also very easy to get started. You can simply write scripts or write a "binary module" in any .NET language such as C#. Simply derive from the System.Management.Automation.Cmdlet class, override the processing methods as required, and start writing code. The PowerShell engine will take care of things for you like hooking up input parameters, matching wild cards, expanding file paths, validating inputs, generating error messages in the user's OS language, etc...

In my experience, I can write a "proper" command line tool with all the bells & whistles using C/C++ in about a few thousand lines of code. With modern command-line parsing libraries, it's about 500 lines if I'm lucky. Meanwhile, it's possible to build the equivalent using PowerShell and C# in about 50 lines, of which 40 is just generic C# class boilerplate.

Yeah, the bash ecosystem looks fairly different with regards to dealing with structured data since jq became popular. Basically jq means that as long as you can safely get your input from whatever custom format into json, and from json to whatever output format, you can safely work with structured data.

Incidentally, I think it would have been possible to use only `csvkit` commands and skip the format munging to and from json, but I have an allergy to working with csvs since the format is a bit underspecified and there exist "csv" files that break even the few rules the format is supposed to have. You would be horrified at some of the things I have seen that called themselves "csv".

> In my experience, I can write a "proper" command line tool with all the bells & whistles using C/C++ in about a few thousand lines of code

Yeah, in my experience trying to write command line tools in low level languages is an exercise in pain. I usually write new tools in Python, where similar to your experience writing a simple command is ~20 lines of code, ~10 of which is boilerplate that is the same for every command.

That said, powershell does look pretty cool. And I see that it's possible to install on linux as well[1], and as far as I can tell even to write my commands in python[2], so it almost looks like "a unix-style shell but with types" which would be useful.

[1] https://docs.microsoft.com/en-us/powershell/scripting/instal... [2] http://pythonnet.github.io/

This thread was a joy to read because you guys didn't come into fight :)

It is a stylistic thing, of course, but I tend to put the vertical bars at the ends of the preceding lines, which eliminates the need for escaped newlines.

One of the things not often mentioned about PowerShell is its embeddability. I have explored this since writing https://news.ycombinator.com/item?id=22849569 .

PowerShell 7 fixes it so you can put pipes on the new line, without needing backtick escape on the previous line. e.g. in Windows Terminal ctrl+enter a newline and write

    gci
    | select name

> One of the things not often mentioned about PowerShell is its embeddability

Not mentioned enough; Exchange 2007 GUI was built with PowerShell integration such that you could click through with a mouse, and it would show you the powershell code for you to copy/paste and reuse it yourself without a GUI. This is a very underused, under-discussed idea, IMHO.