Getting work done with PowerShell on Linux
deusinmachina.net
deusinmachina.net
This is for me the key value of shells like PowerShell.
Uncountable billions of person hours have been wasted parsing the text output of common *nix commands. It’s time to move away from the “everything is text with random formatting” philosophy.
In my experience, there are often commands which just output text, and I need to parse them somehow, which leads to an awkward flood of Out-String | { $_ | Select-String -Pattern "etc." } type commands which are hard for me to get right.
Perhaps that's just a reflection of my lack of experience with PowerShell, but assuming commands have a stable CLI interface (which most commands do in the *nix world), it's much easier for me to simply parse things with sed/awk/grep.
Everything returned by the built-in commands is. If you are calling some executable, yeah, then you'll need to wrap it. I do recommend wrapping commonly used executables with things that return PowerShell objects. Tedious at first but as you build up over time you find yourself straying into string parsing less and less.
I'd better spend my time writing a different shell. This sounds like a life-long with ever increasing number of bugs nightmare to support kind of project.
It would give me tools like sed/grep/awk/tr/cut/etc., and allow easy capturing of output into variables and performing bash substitutions. IMHO, that's much more powerful than PowerShell's Select-String.
The big difference is that you don't have to use them for every goddamned command.
Really? What are the equivalents?
Also, you can use linux tools in PowerShell, but that is not the point, it is back to stone age moment.
Shell scripts shouldn't be other name for text parsing.
You cannot use linux tools in PowerShell in most cases, since they aren't installed by default.
> Shell scripts shouldn't be other name for text parsing.
That may be true, but like any other ideal - it's just that, and ideal. In the real world a lot of shell scripting requires parsing unstructured text from various command outputs and files. PowerShell works great in its own microcosm of objects and commands that manipulate objects, but as soon as you step out of that comfort zone it becomes awkward to use.
> You cannot use linux tools in PowerShell in most cases, since they aren't installed by default.
The same as jq and specific variants of tools you mention. Not like you can't install them in single line: winget/choco install awk,grep,jq,less...
> In the real world a lot of shell scripting requires parsing unstructured text from various command outputs and files.
No it doesn't. I do shell scripting for more then decade in PowerShell and I can count how many times I parsed the text. The most used one is simple split like with `$paths = $Env:PATH -split ';|:'`, but other then that no. I used select-string only couple of times in decade, on all OSes.
> but as soon as you step out of that comfort zone it becomes awkward to use.
That is true for literary anything. Actually pwsh does great with 'native' tools too, and there is even MS module to help with "native app wrapping". See [1] and [2]
Today its more or less regular to have json output which can be trivially read by pwsh ConvertFrom-Json
[1] https://devblogs.microsoft.com/powershell/native-commands-in....
[2] https://redmondmag.com/articles/2020/12/11/powershell-cresce...
I really couldn't care less about plain text. It doesn't bother me. What PoverShell "gives me" is another spoonful of sugar in my already sweet tee. It's really unnecessary / would just make my life more miserable.
That is funny and shows tremendous lack of understading of shell complexity. Perhaps your own OS next ?
Another thought I have is re-designing the work with processes and their output in the way similar to Erlang's ports. I'm not sure this can be done in all instances, but I would definitely give it a try.
There are other minor evils, like return vs exit that needs to be solved... but, I really don't have a plan for making my own shell, so all the above is just based on particular pain points I had with it over the years, so, I don't really plan on making my own shell. But, equally, and even more so, I have absolutely no intention to use PowerShell, especially because I'd have to maintain hundreds of unrelated pieces of code, each with its own versioning history and a huge compatibility matrix. That just sounds like insanely boring and unproductive way to spend my life.
Obviously.
If you create pwsh script that is longer then equivalent bash script and more confusing, then you are doing it wrong. One almost never parses string in pwsh, unless string parsing is your primary task itself. Contrary, in bash, you always parse text, no matter what you want to do, which is defocusing and hard to maintain, and has cross-OS problems even in *nix land (solaris vs red had comes to mind and their respective ls/stat output which differs).
Assuming that there are object-based commands that do everything that you need, which is quite a generous assumption.
I'm certainly not saying PowerShell is bad to use but if you aren't on Windows you're probably already scripting in Ruby or Python or something like that.
Then if your script becomes more of a serious thing, all the stuff you're interacting with probably has a Python SDK and you'll never find that with PowerShell.
Working with text output is easier because it's uniform, conceptually simple and easy to debug.
Working with hierarchies of objects is a lot more complicated. Comparing things becomes a nightmare. Working with unknown hierarchies becomes a nightmare. It's a lot harder to work with objects. But, in some situations it's more rewarding because it eliminates certain classes of errors. I.e. you put more work in, and get arguably better results.
The problem with PowerShell approach is that it's attempting to do it on the wrong level. The operating system and the programs it runs have to understand structured input and produce structured output. The system also needs to provide serialization protocol so that multiple systems can communicate with each other not breaking the illusion that they are using objects. However, object RPC is not a solved problem...
In other words, PowerShell is a pretension. It solves some easy problems and if you don't give it enough thought, you may think that it will solve harder problems too... but it turns out there's some fundamental problem with this approach. So, it's OK to use it as a gimmick, but I would never consider using it as a replacement of Unix shell (no matter how much I hate it).
If it is so uniform, why does every command need a dozen switches for modifying their output? Why do we need to use complicated tools like awk to reformat it for input into other commands?
Everything you cite as a problem with structured IO is a problem with text too, it just happens to also suffer from in-band signaling issues and be human readable. Oh and of course it doesn't have a standardized structure at all.
It is uniform. It is an immutable string.
> why does every command need a dozen switches for modifying their output?
This question is unrelated to uniformity... Depends on the command I guess? Some do, some don't. Another argument is about how much information you want printed... I mean, when you work with SQL clients you don't just select the entire table every time you want to do something with it, but it doesn't mean that tables are unstructured uniform strings... quite the opposite.
> Everything you cite as a problem with structured IO is a problem with text too
Can you explain this? I don't see the connection. You make no argument to support this what so ever...
PowerShell objects are uniform too then. They're just a blob of binary.
> Another argument is about how much information you want printed
That's the thing, though, when I'm passing it to another program for further processing that is entirely different than formatting the output. PowerShell has commandlets you tack on the end of a pipeline for formatting the output. In between commands they have standardized objects.
> Can you explain this?
> Working with hierarchies of objects is a lot more complicated.
Hierarchies exist in textural output too! And they are indeed a pain in the ass to deal with. In fact, much more so since they aren't adhering to a standardized structure and use in-band signaling.
> The operating system and the programs it runs have to understand structured input and produce structured output.
Same with text, it's just that the "structure" is completely ad-hoc with no standardization and using in-band signaling that often requires a lot of error-prone manipulation to adapt between commands.
> Hierarchies exist in textural output too!
Dealing with immutable strings instead of hierarchies allows you to ignore the fact that there's a hierarchy encoded in text. Maybe it's there, maybe not. I don't care. My tools allow me not to care. Your's don't. That's the point.
> Same with text, it's just that the "structure"
No, it's not the same. Read again.
That anyone is seriously promoting this as a benefit boggles my mind. I don't think we live in the same reality so this discussion is pointless.
Yes, writing powershell tends to be more verbose. But reading scripts is so much better, and from my experience larger scripts are much more maintainable. Powershell can enfore types in function arguments, for example.
> But reading scripts is so much better, and from my experience larger scripts are much more maintainable.
What kind of experience do you have maintaining shell scripts?
I maintain a management product that is installable on systems all the way back to RHEL6 (and family), and similarly SLES12, Ubuntu18. The overall matrix of systems and product versions I have to maintain is 20 different OS x product combinations. Most of my maintenance work goes into things like "the release of system X broke the systemd interface to LVM by means of starting udev service too early, now we need to make an exception for this system by altering its configuration somewhere". In order to reduce the burden of maintenance going forward, I need to write scripts that are as generic as possible. If ntpd service was replace by chrony in some distro, I want to have as little code to modify as possible to adjust to the new reality. I don't want to write adapter and factory and other "architecture" for each such change. If I did that, the product cost would skyrocket, while development time plummeted. This would, basically, spell the end for this product. Customers know the risks and accept them. And that is the unfortunate reality of it...
But, if I had to change anything about it, I would never choose objects. This is just replacing one problem with another with very little benefit. What I would like is something like Erlang: records with a handful of predefined atomic types. Objects will bring too much of unnecessary problems with them.
> Powershell can enfore types in function arguments,
I know. This is awful. Especially for maintenance.
Semantics. Nobody uses bash without those, and having different variants on them with different variants of regex is most troubling. Contrary to that, in pwsh it is recognized that those should be part of the shell that will behave the same on any OS. U ofcourse left out tone of those that come OTB in pwsh (like html/markdown/json/csv/sql readers and exporters vs jq,pandoc,usql,whatever)
> But, if I had to change anything about it, I would never choose objects. This is just replacing one problem with another with very little benefit.
You keep saying that without any evidence whatsoever. Its just a personal taste and you should declare it as such.
> Powershell can enfore types in function arguments
>> I know. This is awful. Especially for maintenance.
It looks to me that you dont actually have programming background, and that is the reason you say things like that. Your bash scripts the way you described them would probably be heavily reviewed and massaged where I work, for them to be accepted.
No. It's not semantics...
> Nobody uses bash without those
Boy you are so wrong... Of course many do. And those are the people who care the most about what shell can do and how it does it. People you describe are the people who need shell for other reasons, who could probably do away with Perl / Python / Ruby etc. instead. I don't care about that audience. Actually, it would be probably even better if Shell isn't used there at all. Shell has way too many restrictions for that context.
> You keep saying that without any evidence whatsoever.
Please, read again. I listed the problems earlier: identity (comparison), serialization, methods overload, mutable state, dependency through hierarchies s.a. inheritance. I don't want any of that in a shell, because, someone will definitely use it, and in the most unnecessary place, and it will throw a wrench into the Rube Goldberg machine of Linux shell scripting that's falling apart in many hilarious ways as it is.
> It looks to me that you dont actually have programming background
Look. An expert! You just discovered my programming backround (or lack of it) by reading a post on the Internet. How quaint!
> Your bash scripts the way you described them would probably be heavily reviewed and massaged where I work, for them to be accepted.
Lol. You sound like a Web developer with 5 years of experience, who decided that they've seen it all by now, and got a title of "senior developer" which made them feel important and overly confident. Nope. You aren't there yet. The road ahead is long and bumpy. You better curb your enthusiasm of judging others for a while.
PowerShell objects are flat, 99.99% of the time.
And when you say compare, you don't compare objects, also 99.99% of the time. You compare attributes.
If I wanted that, I'd be using Java.
Compared to a flat list of related strings? No, its not. Its like saying that categorisation is worst then keeping everything in large bin.
> Comparing things becomes a nightmare.
Comparing objects deeply can be done in one line. If anything is nightmare, its people who judge without RTFM.
> So, it's OK to use it as a gimmick
I am amazed about lack of recognition on amount of work that MS and community have put into it (during two decades of gimmicking) :)
I'm amazed by how irrelevant all MS products are without a fault when it comes to catering to people who are on the more experienced and knowledgeable end of the spectrum. Their products seem to always completely misunderstand the target audience, but bruteforce their way in nevertheless due to the huge investment in marketing and "grease" to secure deals.
In a way, it's very similar to that last showing of Tesla trucks, where it became apparent to truck drivers that whoever designed them simply didn't know what truck drivers value or need. So, Tesla trucks are no joke to design. Definitely, a lot of effort went into making them, and yet they fail on very simple stuff like having enough room to put your dirty shoes away.
In a very similar way, every product by MS I ever touched was usually decently invested in (outside of things like MS Outlook, or MS Word original file format etc.), decently tested, decently documented and... completely irrelevant in any context I could think of applying it.
I use various shells daily for a huge number of tasks. My shell history just for a single terminal may be couple hundreds lines a day (but usually I have several tmux sessions in several ascii-term buffers in Emacs open with typically 6-10 buffers per tmux session). I do write some reusable Shell code too, but less often. As a general infra / system person I have to touch every aspect of the system, be it storage, virtualization, networking, user management... and there simply isn't any place in any of that where I'd say "well, PowerShell seems like a good fit". There's always an obvious way to do things better without it.
That's not to say I don't want my tools to improve -- far from it. There are plenty of things that Unix Shell gets wrong. None of those problems are addressed by PowerShell. Like I said, it's a gimmick for MS Windows users. Maybe it makes the experience of using MS Windows more palatable -- I don't even want to know. On a very rare case, like when a relative asks me to help them with their computer, I see the horror show that MS Windows is, and I want none of that stuff.
This deserves a good laugh. How do they keep making programmers so naive every new iteration?
Seriously? Some famous Java dude wrote a whole book about how Java got object comparison wrong. And how seven or so versions and twenty or so years later they still had problems with sorting because of that stupid dysfunctional comparison. And you solved it. In a single line? I sense I might be talking to the next Turing award nominee!
> libxo - A Library for Generating Text, XML, JSON, and HTML Output
* https://github.com/Juniper/libxo/
* https://wiki.freebsd.org/LibXo
Want structure? Ask for JSON or XML and parse. Otherwise it's the regular text output.
If you spend most of your time in the Windows ecosystem, then there's no reason not to learn it. If I had cut my chops with PowerShell instead of bash I'm sure I'd be clamoring for some kind of LSW system. There's a reason why things went the other way, though. I don't think we're going to get Rust bindings for C, either.
This was the reason why I learned PowerShell. It is the only shell that works on Windows/Linux/OS X (to those who don't know, it's open source now: https://github.com/PowerShell/PowerShell ).
Every team I join is a mix of Windows and Mac users with the occaisional Linux user. This is the only shell that you can use to across all three without compromise. Otherwise I find that whatever platform is dominant writes the helper scripts and the others get shrugs and "figure it out". Now, you could try to write them in Python, which isn't the worst idea, but it's not a shell.
It's intentionally different than UNIX shells. If you are expecting UNIX shell behavior you will not like it.
I have to use it at work for Windows stuff. It is amazing. But I would never install it on my home computer because I don't trust all the other Microsoft stuff that comes along with it.
There is one big complaint that I do have with it - working with massive (multiple gigabyte) files is a pain. It wants to load the entire file before doing anything; most likely because it wants to treat everything as an object - therefore it has to build the objects before it can work with them.
Can it finally manage Office 365 and the like from Linux?
I tried it once (1-2 years ago) to run some command that couldn't be done from the web interface. The docs said, "this should work". As I expected, it didn't. Some further digging turned up that it didn't support OAuth2 or something related to authentication.
What? POSIX sh
jupyter notebooks have filled this void for me, especially with vscode integration and ipywidgets. I want to learn and like powershell, but my junk drawer of automation helpers is too full to start from scratch.
It's a huge minus. The fact that all shell tools produce text is a basic tenet of shell programming which underlies the composability and interoperability of them all.
> the main benefit of bash is the portability
No, see above. Also, you probably mean ubiquity rather than portability.
> If you spend most of your time in the Windows ecosystem, then there's no reason not to learn it.
The reason is that it will make you only work with other tools by Microsoft, or at best some Microsoft-centric companies which write such tools. Those tools might gain some benefit from working with non-textual objects, but the detriment in variety and composability (even ignoring multi-platform work) is massive.
Another detriment would be how it's non-shell-like work. Part of the idea of shell scripting is that it's work that you might have done "yourself", manually, manipulating the text. With something like PowerShell - you need a debugger, data viewers etc. In fact, one could argue that PowerShell is barely a shell.
PowerShell is essentially a cut down .NET environment with some nice syntax-sugar for typical shell activities. The objects in PowerShell are marshalled to/from text the same as in any other programming language.
So you can still write your tools as normal. Any PowerShell specific objects would just be a wrapper, or extra metadata. If we're being honest we already have that for other shells to make them more convenient: autocompletions. PowerShell objects give you autocompletions, type information, and interoperability with .NET libraries _but for data in addition to commands_.
The shell to programming language continuum has enough space to permit richer interpretations of the base data tools emit. We don't lose composability.
> With something like PowerShell - you need a debugger, data viewers etc.
Oh, what I would give for better shell script debugging. This is right up there with better CMake debugging for me.
$ ll /bin/dash
-rwxr-xr-x 1 root root 85368 Jan 5 09:52 /bin/dash
That shell can go places that Powershell will never, ever see, because Powershell is too big.There really isn't any way that Powershell could change this.
For me the main benefit of PowerShell is portability. I now just have one bootstrapping script for my cross-platform project, which works on Powershell Core on Windows, Powershell Core on Linux and if I wanted also Windows Powershell or MacOS.
I mean maybe. My big thing is single responsibility.
My suspicion is if you wrote 10,000 lines of shell script that should probably be like 20+ individual scripts and you've written something with way too many responsibilities.
A script should do a single thing.
Redis is 70k lines of code and yet only "does a single thing" - runs an in-memory data store.
IMO it's much more useful to just consider which mode of code organization increases readability and maintainability. Those are things you can actually describe and examine.
I’m not switching all of my workflows to Powershell. That would be silly. I’m honestly not particularly inclined to use Powershell unless I have no other option. But it’s a valuable tool in the box. As are Python and (lately for me) zsh. And as with anything else, it’s worth knowing how to get the most out of it.
This article isn’t bad. I also found Powershell in a Month of Lunches to be helpful at demystifying it.
um...Really?
1 - if you feel like disputing the security record of Azure, they're literally the only big public cloud with more than 0 (and multiple at that!) critical cross-tenant security vulnerabilities, at least some of which were extremely trivial to exploit.
PowerShell is great, MickeySofts own modules, not.
> A couple of years ago and senior exec told me that PowerShell was THE technology that enabled Microsoft to move to the cloud.
> He said that Exchange lead the way and that it could never have done it if weren’t for PowerShell.
> I had never heard that perspective. But liked it.
Maybe I'm misinterpreting it, or he is stroking his own ego, but there at least is one source for it
But of course, the syntax stays the same. And, to me, one of the worst things is how laggy it is (on an 11th gen i7 with some kind of nvme drive).
pwsh:
> write-error Oh-Oh
Write-Error: Oh-Oh
Windows Powershell: > write-error lol
write-error lol : lol
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorExceptionDid a search, and seems he was eventually promoted:
https://www.spiceworks.com/tech/tech-general/news/jeffrey-sn...
He was a technical fellow at MS for the past number of years, which is their technical track equivalent of a Senior VP.
To take an example from the article, here is the marcel code to read penguins.csv and get unique species values, sorted:
read -c penguins.csv | map (species, *: species) | unique | sort
Or to find species = 'gentoo': read -c penguins.csv | select (species, *: species == 'gentoo')
"read -c penguins.csv" reads the file, but instead of returning strings, -c causes each line to be parsed into a Python tuple.In both cases, parens delimit a Python lambda (you can omit or include "lambda"). "species, *" binds species to the first value of a field, and * to a tuple of the remaining fields. So "map (species, *: species)" maps the row to just the value of the species column. "select (species, *: species == 'gentoo')" uses a Python expression to select rows with species 'gentoo'.
You can also do grouping and aggregation in marcel. For example, this command locates files recursively, obtains the extension and size of each, groups by extension, and then sums sizes for each extension:
ls -fr | map (file: (file.suffix, file.size)) | red . +
"ls -fr" lists everything recursively (-r), yielding only files (-f). A stream of File objects is piped to map, which maps each File to an (extension, size) tuple. "red . +" does reduction (i.e., grouping and aggregation. The "." specifies that the first part of the input tuple, the extension, is used for grouping, and the second part, the file size, is aggregated using +.It feels kinda weird at first, but then you install the LSP server for it in your editor, and suddenly things start feeling way more solid and reliable than bash.
I’ve come to appreciate it more than I expected.
set-location ~
> On PowerShell in Linux and… cd ~
> on Windows both send you to whatever your Home directory is.Is not it vice versa?
I mean, on Linux it is `cd ~` and on Windows `set-location ~`, no?
As for me, it would be better to clarify as next:
> Under Linux and Windows both commands `cd ~` and `set-location ~` in PowerShell send you to whatever your Home directory is.
You can use both in pwsh on any OS.