I think your thoughts here are definitely coherent, and in fact you're getting to the heart of the issue. This exact question is pretty central in motivating many next-generation shells. I'll refer to the documentation of a few of them below. Feel free to follow the associated link to tug on any of the threads and see if the overall argument holds in your opinion.
> Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on.
> Traditional shells use strings for all kinds of data. They can be stored in variables, used as function arguments, written to output and read from input. Strings are very simple to use, but they fall short if your data has an inherent structure. A common solution is using pseudo-structures of “each line representing a record, and each (whitespace-separated) field represents a property”, which is fine as long as your data do not contain whitespaces. If they do, you will quickly run into problems with escaping and quotation and find yourself doing black magics with strings. […] Elvish offers first-class support for data structures such as lists and maps.
> bash does not meet any modern expectations for syntax, error handling nor has ability to work with structured data (beyond arrays and associative arrays which can not be nested). Let it go. You are not usually coding in assembly, FORTRAN, C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your life harder than it should be. Let it go. (Let’s not make it a blanket statement. Use your own judgement when to make an exception).
>
> Python along with many other languages are general purpose programming languages which were not intended to solve specifically Ops problems. The consequence is longer and less readable scripts when dealing with files or running external programs, which are both pretty common for Ops. For example, try to check every status code of every program you run, see how your code looks like. Sure you can import 3rd party library for that. Is that as convenient as having automatic checking by default + list of known programs which don’t return zero + convenient syntax for specifying/overriding expected exit code? I guess not.
For me, the great thing about a shell language is that it's a programming language I get to _live_ in. Scripts can just grow organically from commands I chain together at my shell prompt, including simple loops and blocks and stuff. The better I get at the programming language, the better I get at navigating my computer and performing routine tasks. The better I get at navigating my computer and performing routine tasks, the better I get at the programming language. This is a really wonderful kind of thing imo, but its usefulness is undermined when one dimension of the shell (interactivity or programming capability) holds back the combination of the two.
The idea with these new shells is to try to find a way to enhance the programming capabilities of shells without making them any less convenient for navigating the filesystem and performing simple tasks with external programs. There definitely are shells that (imo) have tried to improve programmability (usually be embedding in a full-fledged programming language) in a way that undermines the simplicity and freeform character of shell languages, for example Rush (embedded in Ruby), Ammonite (embedded in Scala) and Xonsh (embedded in Python).
But other next-generation shells, mostly following the examples of PowerShell and Fish (and lots of programming languages), (imo) do a remarkable job of improving programmability without making simple, everyday shell navigation workflows feel any weirder or more laborious. In particular, Elvish, Nushell, and Oil Shell strike me as very thoughtful efforts whose very small teams of contributors have been hacking away at them for quite a long time.
One important thing to keep in mind for this particular breed of next-gen shells is that their novelty is mostly in the synthesis of a collection of minor and conservative innovations— things that we've seen in some shells (like PowerShell) and some programming languages (especially Lisps) for a long time. Each of these shells is a substantial leap forward beyond shells like Bash only because the language designs of shells like Bash have been largely frozen for decades. For a concrete example of this perspective:
> Nu takes cues from a lot of familiar territory: traditional shells like bash, advanced shells like PowerShell, functional programming, systems programming, and more. But rather than trying to be the jack of all trades, Nu focuses its energy on doing a few things well:
>
> * Create a flexible cross-platform shell with a modern feel
> * Allow you to mix and match commandline applications with a shell that understands the structure of your data
> * Have the level of UX polish that modern CLI apps provide
That phrase at the end of the second bullet point is key: ‘a shell that understands the structure of your data’. We can see a clear call for shells like that in the growing crop of fancy new CLI tools designed for chopping up, querying, and modifying ‘structured’ (not merely textual or stringly-typed) data. Tools like that frequently show up here on HN: jq for JSON, yq for YAML, xsv for CSV, etc. Developers and sysadmins want to be able to pull in structured data from whatever source and query it and manipulate it right there inside their shells. But each format gets its own utilities for that kind of querying and manipulation, and they don't necessarily know how to talk to each other. The pipelines we use to plug various tools like this into each other, meanwhile, only know about raw text/byte streams. The natural thing to want, when you're faced with this, is to be able to just ingest data of all of these forms into variables in your shell which retain that structure, and query them in a uniform way (that your shell may even be able to assist you with as you type, with some static checking!).
The Elvish shell has a nice example of using this to query the latest Elvish issues from GitHub on their website:
curl -s https://api.github.com/repos/elves/elvish/issues |
from-json | all (one) |
each [issue]{ echo $issue[number]: $issue[title] } |
head -n 11
1396: Exception control: “try” should rethrow flow control exceptions
1391: Document forking behaviour
1385: Feature request: command to find location of rc.elv file
1381: `use edit` creates a crash situation for elvish
1380: Issues running `wal -i`
1377: exit builtin does not trigger program's defer statements
1376: buildin cmd source !
1374: Feed stdin to all code blocks in run-parallel
1373: Recent performance/speed benchmarks?
1372: Octal Format Specifier
1371: "autoview", tables, new data formats...