I went from having no knowledge at all to having written a web scraping function and checked 7000 sites for a specific phrase in less than two hours. It was the most intuitive piece of technology I've ever used.
If you're thinking "but... it's all so obvious. What is this guy talking about", ask yourself how long ago you first typed a shell command. Can you even remember how hard it was to learn? For me it was something like 15 years and I actually can remember because it made such an impression on me how terrible a system it was. (That and X86Config if you remember that insanity.)
Even though it was awful to learn, it feels totally easy and 'intuitive' to me now. But that's just because I use it so much. It's not intuitive. Don't fall into the old curmudgeon trap of thinking the thing that you find trivial through experience is actually good.
/rant
No you don't; whether ls is a builtin is mostly irrelevant, but
ls *1*.png
dir *1*.png
https://blogs.msdn.microsoft.com/oldnewthing/20071217-00/?p=...
...would effectively produce a pattern matching all files ending in '.png'.
ls doesn't do glob matching. ls '*' would try to fstatat (FreeBSD) that character.
EDIT: formatting
It's very common to think of the first thing you learnt as more intuitive. Learning the second thing you have baggage and expectations of how something is supposed to work.
From what I recall, it was easy. Other than a few small differences ("mkdir" instead of "md", needs a space between the "cd" and the "..", and so on), it was similar to what I used every day. The rest I learned gradually, by reading shell scripts written by other people, and the man and info pages.
As for XF86Config, I always used the XF86Config generators which came with the distributions, so it was never much of a problem for me.
I can remember when I first learned bash/sh and Unix systems in general (by reading a book) and it was not really hard. However, I saw your sentiment a lot among the students I used to teach, and my response has always been "you're approaching it the wrong way." Bash, PowerShell, etc. are effectively programming languages, and my advice for learning programming languages applies: it's a language.
To elucidate, programming languages have their own vocabulary and rules just like a human language (and often, the rules are far more consistent and the vocabulary smaller) --- to ask questions like "Why not `list`?" is like asking "Why is the plural of sheep, sheep? Why is it called a mouton in French? Why not sheepe?"
Of course, you can ask such etymological questions when learning about programming languages and use them to help you, because they will have far more logical answers than the same questions about human languages; but in general, learning a language as a language is the best way to progress.
In fact I'd say super-verbose and somewhat convoluted languages like PowerShell (and C#, which I don't fancy much either --- nor its Java-ish ancestry, for that matter) may only appear to be more intuitive at first, but are actually hiding some quite nonintuitive complexity. The verboseness may make it easier to get started, but becomes a hindrance thereafter.
/rant
Some historical syntax ancestry aside, state-of-the-art C# is too classy and elegant to be in the same sentence with Java.
But unlike normal languages, software languages are designed. So asking why questions makes a lot more sense.
I wanted to do floating point math. The recommended answer is to pipe your data into a better programming language. You can't even do math in bash. On another occasion I wanted to make a simple program that opens a random file. The recommended answer breaks in the simple case where files have spaces in them. The correct answer I found is this horrible mess of weird characters and it's completely incomprehensible. The best answer someone submitted was just to call one line of python from bash...
And why is it so wrong for the shell to be a decent programming language? It's not impossible. It doesn't need to make anything harder or more complicated. And it adds a ton of functionality. There are tons of people programming in Bash and powershell anyway.
You shouldn't, certainly, but, as to 'can't', I think that `expr` (or, as I just learned from ABS, `$(( ))`) covers a lot of basic uses: http://www.tldp.org/LDP/abs/html/arithexp.html .
Someone who tries to learn Chinese may have the same thought after having been exposed to nothing but English. The keyword is "language".
The quoting rules in bash/sh are not difficult to memorise, and definitely a must-learn.
I wanted to do floating point math.
Maths is not a strong point of shell languages because they were designed more for string manipulation and gluing other programs together. Try doing pipelines and redirection in a "real programming language", for a contrasting example.
"The right tool for the job".
mv and ls are not shell commands, they are programs. They existed long before bash and were developed at a time of 110 baud teletype connections when short, highly abbreviated command names were worth the inconvenience.
If you don't like them you're free to alias "rename" to "mv" and "list" to "ls"
"How do you end this block? Maybe... I type 'end'? Nope! You write the opening statement backwards! Ha ha get it? Backwards lol!"
done?
Also, there's a story behind all these commands, their naming, the symbols, reasons for the brevity. But I've never not found Unix intuitive, even before I was aware of the reasoning behind everything. That said, Unix is definitely an environment for the more savvy. But what's intuitive for me is not necessarily intuitive for someone less savvy.
It redirects standard error to standard out? That's what it does in powershell anyway. Frex, I got this line in my $PROFILE:
$p2 = & $env:Python_2/python.exe --version 2>&1
The real problem is that, for some reason, python.exe --version outputs to standard _error_. That, I don't really get. I'm sure there's a good reason for it but it's not obvious and that's what's making the "2>&1" at the end hard to figure out.
I miss-remembered the number of sited scraped but it's all the same. There is no parallelism (I really really am a complete non-techie) but actually in this case I didn't want to hammer away as the sites were all under the same domain and I imagine there might be trouble (which I why I also played it very safe and added the pause).
I know there are better and smarter ways of doing things and I did further modifications and experiments after my thread, but in any case, I had a task and was able to quickly complete it in PowerShell starting from absolute 0.
There's a bunch of 3rd-party modules out there that do a great job of offering parallelism[1][2][3], and we also support the notion of "PowerShell Jobs"[4] (which one or two of those third-party implementations actually build on to achieve what they're doing).
However, we decided that strong parallelism support is important enough to build into the language, so we're mulling on a first-party design.
If you're interested, we've got an RFC out where we're looking for feedback on the parallelism/concurrency design.[5]
[1]: https://www.powershellgallery.com/packages/PSParallel/
[2]: https://www.powershellgallery.com/packages/SplitPipeline
[3]: https://github.com/RamblingCookieMonster/Invoke-Parallel
[4]: https://msdn.microsoft.com/en-us/powershell/reference/5.1/mi...
I've come to the conclusion that its just better to write a small c# app than try and use powershell
The whole "I can run this script on 100 different machines at once super easy" is fine, but lets see you try to write some rigorous code in PS and the come back and tell me how it's easier than C#.
They both have their place.
When I took over it, I wrote a PS script to automate the whole build. The script contains a description of each library's dependencies, so it knows what order to build all the libraries in. Each library is built in its own "job" using Start-Job, so the builds are also maximally parallelized; the script is essentially a loop of two functions: `for each library -> if library's dependencies have all been built -> start parallel job to build library` and `for each job -> if the job has finished -> mark the library as built`.
I would not write it in C#. Most of the build steps for each library involve running other processes, so using a shell DSL rather than multiple Process.Start()+Process.WaitForExit() is convenient. Even things like copying files is more convenient using Copy-Item than C#, especially since the PS versions support globs.
I would hope a build tool counts as both rigorous code as well as justifies PS as a programming language.
Aside: The repository was forked by some other people who did not like / could not read PS, so they rewrote it in Python. The Python version is several times larger than the PS version, though to be fair to it it's also more generic and has more features. A lot of that verbosity though does come from the more verbose code for spawning processes from Python compared to PS.
Well, it puts Arnavion in good company: https://en.wikipedia.org/wiki/List_of_build_automation_softw... .
and I should mention, I feel like that's a solved problem. I don't know the specifics of the project, but it's hard for me to buy that you needed to write dependency checker in PS.
But the worst thing I came across with Powershell is this. The fricking return semantics. http://stackoverflow.com/questions/10286164/function-return-...
Where does tcl fit into things for you?
I'm definitely using 'shell' and 'command' in a narrow sense. I meant to add contrast between PowerShell and C#, by calling one a command shell and the other a programming language. But both words have been used with much broader meanings. So I'll try to be a little more careful and say that what I'm really talking about is a system shell and system commands, where a system command means direct access to executable files on the system, and system shell means an interpreter that provides system commands.
Even though Tcl stands for 'command language', it's not talking about system commands using the same distinction I made above. Tcl's use of "command" is referring to programmer-defined functions & API, and it is not referring to system commands.
Tclsh is a shell, but I wouldn't call that a system shell like PowerShell or bash. It is an interactive tcl interpreter, but not really a system shell. Tclsh isn't something sysadmins would typically use, right?
Perhaps the defining characteristic of a system shell (as opposed to a programming language) is system commands; running executable files on the system can be done by typing the bare name of the file, without any other syntax in the way. That is true in powershell and bash, and not true in C#, tcl, python, etc. To run a system command in tcl, you have to use 'exec' or 'open', you have to use 'glob' to get file completion, and there's required programming language syntax frequently involved in passing arguments. In tcl you have to use $env(var) or $::env(var) to get an environment variable rather than $var. Those are the kinds of things that shells provide with little or no syntax, and exactly the kind of stuff that begins to compromise language syntax and language design in favor of promoting system commands to first class status.
BTW, you got me curious about tcl again. I never used it enough to get a sense of what it's best used for. What kinds of things would you use tcl for yourself? Do you like to use it for system shell tasks instead of bash? (I know there are certainly legit reasons to do that.) What kinds of programs would you start in tcl?
they definitely should have gone with some variation of C#.
Similarly, we're trying to make some improvements to debugging right now with PowerShell 6.0 [1] (the one that's cross-plat and built on .NET Core), and I'm very interested in hearing your feedback on how we can do a better job there.
Even if it's "fix your docs" or "too hard to learn", that's great info, especially coming from a C# guru.
[1]: https://github.com/powershell/powershell/issues?utf8=%E2%9C%...
It doesn't seem that I can currently debug scripts that are contained in a module folder, but not directly the script being called.
If I have a module file that looks like this:
Get-ChildItem -Path $PSScriptRoot\.ps1 | ForEach-Object{ .$_.FullName } # source the script files Export-ModuleMember -function -* # export those sourced scripts
I am not sure how it works exactly, but if I am writing a new cmdlet, Get-HackerNews.ps1, the breakpoints aren't hit in ISE. It would be super cool if it did.
------
Also is there guidance on module structure? Right now it looks like the docs indicate that all the cmdlets should be placed in the .psm1 file, but to me that sounds like it would get rather unmaintainable very quickly. Just looking at the one my dev team has created, we have nearly 50 exported functions. Maybe our our module could be broken up, but that would still leave us ~10 cmdlets per module.
In 2007 I joined the powershell enthusiasts excited by the notion of a bash-like capability treating .NET constructs as a first-class citizen. I invested time into learning common use cases. Certain things you would THINK would be easy like loading a DLL as well as it's corresponding config file were actually quite difficult in practice which was a shame because the primary vision for powershell was to be flexible glue between compiled assemblies. Have you ever tried to invoke ms-test with powershell on a DLL which has a config xml?
I had a touch-and-go relationship with poweshell. I returned to use it a few years later to find that many of my powershell V1 scripts were obseleted by the powershell V2 runtime. Writing C#/.NET code to execute on many versions of windows this is rarely an issue. Writing powershell code to execute on many versions of windows you have a serious problem on your hands as the commands syntax and general practices vary significantly between versions.
To this day googling for examples on how to accomplish something basic such as running a dos command line call with parameters constructed within powershell is not only highly vexing for the occasional casual user such as myself but many of the answers I see in blogs or stackoverflow are completely temporal and you need to be careful to look for results dating around 2013 for powershell v2 because anything before that wont work, but not too new because v4 has a new approach and that would require a service pack not all my users can currently procure. Not to mention everyone writes powershell in a different style. You often end up with a handful of different code examples that look wildly different each with their own issues as well.
Sure there are expert powershell users out there with muscle memory that can write elegant solutions for version incompatibility issues and common gotchas, however for the casual seldom user the overall experience that too much frustration, less than intuitive approach and 10 ways to solve a common problem. It feels sort of like running perl in a dos prompt that is constantly changing. Compare this with zen of python where [There should be one-- and preferably only one --obvious way to do it.] I find myself installing python runtime and avoiding powershell when I'm required to make something of significant complexity and high compatibility.
If PowerShell could support this, it would allow fully automated cert replacement for those of us who use Let's Encrypt on Windows (ducks)
Compared to what cygwin’s bash does, it’s like an entirely different world.
- One which stops IIS app pools before a deployment (30 of them). I wanted to stop them all in parallel but found this far more difficult than executing a list of Tasks in c#. So I now just run c# console app to do it
- one which runs msdeploy commands against a web server and had many issues dealing with passing parameters down to msdeploy. I can't quite recall but I think it was either quotes or spaces related.
The ISE as it's called is very very basic. Not what I'd expect from MS. I'd like a full debugger.
There also seems to be a potential security risk of having editable text file scripts on a server Vs a compiled exe.
1) Powershell mingles "process stdout" and "function return" to be one and the same. While this kind of makes sense, unless I'm extremely fastidious about redirecting everything I call to $null, chances are I'm not returning what I expect to return. The end result is I shy away from doing anything that needs to return values (rather than output) in Powershell, which is kind of important for anything nontrivial. I tend to set globals like I've gone back to the Apple 2 and started programming in BASIC again as a result, which gets messy real quick. Processing stdout/stderr in C# is a little annoying by comparison, but easily fixed.
2) Syntax changes in updates. When I installed powershell tools for VS, my scripts broke - to fix them I had to replace e.g.:
info "$vmGuid: has BuildMatrix tags: $tags"
info "${vmGuid}: has BuildMatrix tags: $tags"
logCmd {& $VBM -- clonevm $OriginalVmGuid --register --name $NewCloneId}
logCmd {& $VBM -- modifyvm $NewCloneId --natpf1 "ssh,tcp,$SshHost,$SshPort,,22"}
logCmd {& $VBM -- startvm $NewCloneId --type headless}
waitUpTo 60 3 -Sensitive $true {& $VBM -- guestcontrol $NewCloneId run --exe "/bin/sh" --username "$Username" --password "$Password" -- '/bin/sh' -c "mkdir ~/.ssh; echo '$PubKey' >> ~/.ssh/authorized_keys"}
waitUpTo 60 3 {& $Ssh -- -o NoHostAuthenticationForLocalhost=yes -o ConnectTimeout=60 -o BatchMode=yes -p $SshPort $Username@$SshHost -- '/bin/sh' -c "echo Server ready" 2>&1}
logCmd {& $VBM clonevm $OriginalVmGuid --register --name $NewCloneId}
logCmd {& $VBM modifyvm $NewCloneId --natpf1 "ssh,tcp,$SshHost,$SshPort,,22"}
logCmd {& $VBM startvm $NewCloneId --type headless}
waitUpTo 60 3 -Sensitive $true {& $VBM guestcontrol $NewCloneId run --exe "/bin/sh" --username "$Username" --password "$Password" -- '/bin/sh' -c "mkdir ~/.ssh; echo '$PubKey' >> ~/.ssh/authorized_keys"}
waitUpTo 60 3 {& $Ssh -o NoHostAuthenticationForLocalhost=yes -o ConnectTimeout=60 -o BatchMode=yes -p $SshPort $Username@$SshHost -- '/bin/sh' -c "echo Server ready" 2>&1}
3) Default installs lag way behind - e.g. Windows 7 only comes with PowerShell 2 by default, so you don't even have things like Invoke-WebRequest, forcing you to write your own or get your IT department to install an updated version for everyone. C# just feels like it has more "batteries included" even for older .NET targets.
4) Figuring out exactly what types are expected vs what I happen to be using is a general muddle in Powershell for any sort of collection in my experience. C#'s static typing, clear errors, and intellisense in general is just extremely hard to beat.
* Referencing dependencies (if you're used to "add reference" and "using" it's awkward having to write a bunch of Assembly.Load that then needs to be maintained redundantly with the C#; it would be nice if PS could somehow work with VS/MSBuild projects)
* Calling extension methods (ties in with above since in C# they depend on "using")
* Calling generic methods
* Calling overloaded methods
* Calling async methods
There are workarounds for all the above but put together they definitely make PS less pleasant/useful for me.
Also, much of the above also applies to interop with WinRT which I think is becoming important (as much of the value of PS comes from good access to the underlying system, and more and more Windows functionality is being exposed through WinRT APIs).
The other problem I've had is performance - for example, PS could be useful for some of the same text processing tasks for which Bash is often used (and I'd prefer it for its stronger programming language/object system), but in my experience is dramatically and prohibitively slower when dealing with large amounts of text.
Try getting an experienced C# developer to write a PS function that:
- Initializes an IDisposable resource and deterministically disposes of it when the function terminates (successfully, with an error, or due to ctrl+C). - Honors the caller's $ErrorActionPreference/-ErrorAction. - Understands that catches won't catch "non-terminating" errors. - Throws errors without trashing the ErrorRecord so that the caller gets a meaningful stacktrace. - Copes with the difference between process working directory and PS working directory if a .NET method and a relative path are involved. - Doesn't blow up on a square bracket because someone didn't realize they needed -LiteralPath. - Doesn't accidentally write an unintended object to the pipeline. - Does something reasonable with pipeline input. - Doesn't get tripped up by type coercion magic - $a = @(); $a -eq $null - Uses multiple ParameterSets without creating ambiguity problems. - Doesn't get tripped up by PS silently swallowing exceptions thrown by property getters. - Uses some appropriate subset of language features and modules available across different versions of the OS and WMF. The OS based limitations are understandable. The hard dependencies that Microsoft's own products have on specific PS versions are not. - Is not super-slow.
It's not impossible, but it sure is hard (and takes a ton of boilerplate), and there aren't many resources out there to teach you how. It's all quick and dirty "how to do X" recipes for sysadmins that don't care about more than what they need to paste into a console.
To be clear, I actually mostly like PS I'm just trying to express a set of pain points here.
At this point I should also plug our tooling improvements. First, our VS Code Extension[2] is AWESOME, debugging there has massively improved, there's snippet support for boilerplate with best practices, it integrates directly with PSScriptAnalzyer, and even enables some "quick fix" of linter warnings. We've also got something called Plaster[3] that does templating to encourage best practices (think of it like a "Yeoman for PowerShell").
But yeah, as we craft new docs for writing the best "cross-CLR, cross-platform" (aka universal, aka portable) PowerShell modules, I want to make sure we tick all these boxes.
Oh, and we believe we addressed the whole "different singleton version on different versions of Windows/WMF" thing by making sure that PowerShell 6[4] is x-copyable, fully side-by-side, and supported downlevel to Windows 7/2008 R2 (though Win7 support is currently busted, we're working through fixing it). If you've got a workload or an application that depends on a new feature of PowerShell, you can include PowerShell 6 app-local, or you can distribute it to machines in an environment through either MSI or a ZIP-based file copy.
As for Microsoft products that support specific versions: again, I totally hear you. The best thing you can do is push on those products their respective feedback mechanisms (almost everything is on the Feedback Hub or some product-specific UserVoice now) to support all versions of PowerShell. In the meantime, the side-by-side nature of PowerShell 6 means you can install the latest version without fear that you'll void your support or break existing scripts/workloads.
[1] https://github.com/PowerShell/PSScriptAnalyzer
[2] https://github.com/PowerShell/vscode-powershell
There are crossover points (varying by person) where a scripting language is preferable to a shell, and where a strongly-typed language is preferable to a dynamically-typed language.
PowerShell isn't 10x harder to use than C#, how do I know that? Because many people use PowerShell regularly in their day to day jobs, and do so quite successfully. Most people who actually use PowerShell don't complain about how hard it is.
It is harder than C# to debug, probably more than twice as hard, actually. Does that make it impossible to program in? Hardly. There are tons of other languages that are roughly equally difficult to debug, especially many of the dynamic languages: PHP, Python, etc. People have been building things of tremendous value with hard to debug languages for ages. The fact that C# has such a good debugging story is great, but it's not the end-all be-all of a language. Also, PowerShell has some significantly advanced debugging tools compared to a lot of other dynamic languages.
I can say that every programming language that I grew to love, I liked almost immediately. The ones that make no sense from the get-go, I don't spend much time on.
Surely the second sentence is the reason for the first? If you don't give languages that make a bad first impression a second chance, then you have no idea whether you would have grown to love one of them. (Certainly you aren't obliged to find out, but to regard this as evidence that you wouldn't have liked them I think is reversing causality.)
Especially the fact that I was able to accomplish what I set out to do in a very reasonable amount of time.
$sites = @('https://www.google.com/', 'https://www.example.com/')
$searchString = 'oogle'
$sites | %{ if ((iwr $_).Content -like "*$searchString*") { "$_;OK" } else { "$_;NOT OK" } } > C:\temp\output.csv