Hacker News Reader

Comment by jerf | Hacker News Reader

"So the biggest change is that Oil will be based on strings (and arrays of strings) like shell"

I would advise the author have a look at the internals of Tcl, not because you should copy all the sophisticated features immediately, but because it demonstrates a way to have both "everything is a string" and "advanced data structures" in a relatively principled manner. You don't have to start out with all the optimizations from day one but I always find it helpful to have a look down the road in cases like this, and make sure that if this sort of thing interests you that you don't accidentally make it impossible later.

View on HN

(author here) Yeah I appreciate a lot of things about Tcl, which is basically Lisp + shell (I think Ousterhout even said that?) I read most of the thick book about it even though I don't regularly program in Tcl.

A bunch of people have brough up Tcl, and as noted here [1], Oil looks superficially like it because it has proc and set keywords.

The biggest difference is that Oil isn't homoiconic in the sense that Tcl is. I believe that implies a lot of dynamic parsing which I'm staying away from [2].

And of course Oil is a compatible Unix shell, which imposes a bunch of constraints. Namely the 'proc' is "isomorphic" to a process -- it takes argv and returns an exit code, and it can be transparently put in pipelines. I wrote 3 years ago about this powerful form of composition [3] and I wish I had finished that :-/

[1] https://www.reddit.com/r/oilshell/comments/esqyaw/ambitions_...

[2] http://www.oilshell.org/blog/2019/02/07.html

[3] http://www.oilshell.org/blog/tags.html?tag=shell-the-good-pa...

You can get Tcl-level homoiconicity with fexprs, Lisp-style macros, Smalltalk-like lightweight block syntax, or Algol-style call-by-name lazy evaluation rather than strings. Lisp moved from fexprs to macros in the 1970s for reasons that may not apply to Oil. You may also be interested in my comments in https://news.ycombinator.com/item?id=22150740 about typing; I regret having taken so long to respond to your thoughtful comments.

Thanks for pointing them out. And no problem, I realize I wrote a big mass of text :) I will try to respond without going a big tangent!

I also invite anyone interested in language design to come chat on Zulip [1] or send me mail. I had called for participation in the Oil language design [2]. I got a bit of feedback but am interested in more, especially from people who have used many languages.

I view shell as kind of the lowest common denominator "bridge" language. e.g. Python programmers use it, OCaml programmers, C/C++ programmers, etc. but a lot of them don't view it as a language! I view that as a big missed opportunities, for reasons I will hopefully explain in the upcoming blog posts -- e.g. the one about uftrace I mentioned in this thread. e.g. Shell Helps You Write Better Native Code

[1] https://oilshell.zulipchat.com

[2] http://www.oilshell.org/blog/2019/10/04.html

> I wrote 3 years ago about this ...

> But I realize that, in the best case, this will take decades. That's OK ...

Wow. I’ve never really thought about how much time and dedication is put into the languages and tools I’m surrounded by. Thank you for what you’re doing, Andy. You’re honestly an inspiration!

Yes, take a look at the appendix for my realization about Python from 1991. The comments about Clang and LLVM are also related -- with the Linux kernel and distros and distros in the early stages of migration 15-20 years later.

But to clarify, I don't want implementing Oil to take decades :) I was talking more about adoption. Mass adoption will take decades, just as bash actually wasn't that popular until 15 years ago, but it's also 30 years old. The adoption curve is very long just as it was for Linux, Python, sqlite, etc.

A major reason I write the blog is so that if I decide to stop working on Oil, future maintainers can pick up the knowledge. But I'm also saying it will be embarrassing if I quit before Oil replaces bash for my own use cases. Replacing bash for other people will take a lot longer.

I thought I was the last person who learned Python in 2003, but no, it turns out that several million people were still yet to hear of it :)

What are your thoughts on MSFT PowerShell?

Not the author, but I've used a lot of Unix and Powershell.

Powershell is all objects which has some significant consequences. The first is that many operations will only be reasonably performant on tiny files. If you just want to grep a string from a medium sized file (Ex: 80 MB) and then sort it, the Powershell command is trivial, but can take minutes to run as every row becomes an object. To get around this, you have to generally write a fair amount of straight .NET code like C# which defeats the purpose of using Powershell in the first place. The second consequence is convenience. Because everything is an object, when you get files, you have so much metadata at your fingertips. This is definitely harder in shell and even a pain for me in Python.

Edit: on one of my older accounts I complained about the impossibility of parsing files with reasonable speed in Powershell despite me testing 5 different methods. Someone replied back with another few methods I've never seen before that were much faster, but still slow. If anyone can find that comment I'd be much obliged (hopefully the former poster remembers and can look back).

I really really want Powershell to be more performant with text work as I do a lot of that and everything else in Powershell is so darned convenient. As it stands, it is more of an IT Admin language and possibly DevOps more than anything.

(hopefully the former poster remembers and can look back).

I do, and can: https://news.ycombinator.com/item?id=20724899

[edit: I didn't mention `get-content -ReadCount 1000` but it might be good for streaming-reading a large file if you don't want it all at once. It outputs blocks of N lines at a time as a single string, which you have to split on newlines by yourself.]

There is a related issue in the PowerShell GitHub asking to standardise on a "just give me the data, no metadata" option which would be get-content returning plain strings: https://github.com/PowerShell/PowerShell/issues/7855

I spent 1/2 hour looking for that and failed on two different occasions, so thank you kind internet stranger and better PS user than I. I wish they covered this in "Powershell in Action" instead of just the cmdlets for IO.

I honestly can't believe that they don't have a cmdlet or cmdlet flag for get-content that does as you say and just returns plain text. I don't want metadata anyway when I'm not messing with files and directories.

Do you know of https://hn.algolia.com (change dropdown to search comments not stories). Way better than a Google search. Although it helps that I knew "if it was my comment about faster file reading, I know what was probably in it" to search for "powershell ${C:\" and there it was.

I didn't...I accidently log myself out of my account every now and then and have to make a new one. The only problem is I can't remember all of them :)

Since a few people have asked, I might write a FAQ about this, but the short answer is that I view PowerShell as one of the projects called "shell" that is least like a Unix shell.

Part of that is on purpose, because it's designed for Windows. And what works on Windows is very different than what works on Unix.

Windows is based around COM objects and binary formats like the registry, while Unix is based around text files. That makes the design of an appropriate shell extremely different.

I would say bash, zsh, and fish are pretty close (with the latter concentrating on interactivity), but PowerShell is way out there.

PowerShell also appears to be coupled to the .NET VM and is pretty inefficient. Some links and sharp comments here:

https://lobste.rs/s/16djrz/type_safeness_shell#c_gtmqcl

I think if you want a C# like programming language with shell-like syntax, that integrates with Windows, then PowerShell is for you. But most bash users don't want that! As mentioned, shell is used on tons of embedded systems now (usually busybox ash, which is growing towards bash.)

While I don't disagree with most of your points, NuShell's recent popularity seems to suggest that there is a demand for shells that operate on structured data – even in the Unix world. I dislike much of PowerShell's UX (syntax, tooling, etc.) but I love that aspect of it.

I see from the blog post that you've decided that Oil will operate on strings instead – I'm really looking forward to the blog posts explaining that, I'm sure it's not a decision you took lightly.

Here's some more color on that: https://news.ycombinator.com/item?id=22157587

The summary is that I "left space" for types, but they won't be Python types, for several subtle/philosophical reasons.

* Some links and sharp comments here: https://lobste.rs/s/16djrz/type_safeness_shell#c_gtmqcl*

> "I gave up on PowerShell when I found out that the return value of a function is the console output."

The reason its return values are like that, is that the PowerShell developers copied the behaviour from Unix shells. Cat a file in bash, in a function, assign the result of the function call to a variable, the variable contains the "console output". (Although PowerShell has a pipeline output and separate "console" outputs as separate streams, so this reason for giving up makes no sense - the function could write verbose or informational console output and return something else through the pipeline, as the return value. (They aren't console outputs because PowerShell can be run with no console, e.g. remotely or as an automation engine)).

> It’s like they the cargo-culted a bunch of bad design decisions like the -eq operator without understanding that syntax isn’t why people use shell.

It's like they can now have a consistent operator style which doesn't clash with > and >> and < and | and &, and allows non-math operators like -contains and -match and -f string formatting. I've not seen anyone who wants -equ to be == explain what they'd do with -ieq and -ceq (case insensitive forced, and case sensitive) variants. A mess of inconsistent operators would be even worse, and `-` allows tab completion, which is a bonus.

What none of the comments there address is PowerShell as an extension language - the Active Directory Administrative Center GUI is built on a PowerShell backend. When you are a point-and-clicky Windows Admin doing things in the GUI, you can show "Windows PowerShell History" pane and that shows you the PowerShell commands to run to do the same thing you just did, because that's what it did internally[1]. Any C# (or other .Net language) can load the PowerShell automation dll and run PowerShell commands through it. Any Enterprise C# developers could expose the behaviour of their tools to PowerShell scripting by making a simple C# class with a few attributes. The (perfectly valid) complaints about exit codes and piping binary data from .exe to .exe slightly miss the mountain for obsessing over the pebbles. What PowerShell is to a Windows ecosystem, is like Linux people missing what Remote Desktop is by saying "just use VNC", or what Active Directory is by saying "just use Kerberos and an LDAP directory". It's not just the top three bullet points of what it does, it's how it connects into Windows world.

[1] https://biztechmagazine.com/sites/default/files/tiny-uploads... - random google result, but imagine you're a 20 year Windows admin dragging yourself towards scripting, and on one side Linux users are mocking you, and on the other side Microsoft is giving you human-friendly tools you can click through and then copy the script code out for next time. Imagine you're a competent scripter and just don't know your way around which cmdlets do what, and what parameters they can take.

Reply on news.ycombinator.com