A few things you said stuck out to me:
> For me, the great thing about a shell language is that it's a programming language I get to _live_ in.
> The idea with these new shells is to try to find a way to enhance the programming capabilities of shells without making them any less convenient for navigating the filesystem and performing simple tasks with external programs.
Generally, I agree, and I actually use `eshell` in Emacs for this kind of thing a lot. Being in a pseudo-shell LISP layer allows me to get arbitrarily manipulate text I get back from the command line with elisp and allows for some really flexible workflows. With that said, eshell does of course have many trade offs and limitations, which I won't get into here.
I think where these things always fall apart for me though is in the Elvish shell example you provided. To me, that is not functionally better than using Bash and jq, it's just different. Maybe it's better, I'm not sure, but my first impression is that it's neither more concise or more readable, it's just different syntax.
I think there is a credible argument to be made for "batteries included" type shells, with something like native json parsing, but at what point does it then tip into being like Powershell or Python, where once again you've gotten away from the native/accessible experience because you need to support `n` kinds of structured data inputs/outputs on `n` platforms?
As I was reading your comment, the thought that occurred to me was: Why not just make a python library that can be loaded into the REPL that abstracts over some of the more cumbersome parts of interacting with the OS/filesystem in a shell like way? That seems to be the best of both worlds, and from what I gather that seems like what Rush and Xonsh are doing? I need to look into them further.
> Developers and sysadmins want to be able to pull in structured data from whatever source and query it and manipulate it right there inside their shells. But each format gets its own utilities for that kind of querying and manipulation, and they don't necessarily know how to talk to each other.
I agree re: structured data manipulation, but I think the solution that has naturally emerged exists for a reason. I can use pipes, and programs like jq to push and pull data into and out of whatever format I need, and each layer of that ecosystem can be maintained in parallel, adapting to changes in the overall landscape. In a sentence, it's the core of the Unix philosophy. One thing and one thing well and all that.
I dunno. I've thought a lot about it while typing this comment out. I think it sounds like I'm disagreeing with you, but I'm not. I don't even really think we're debating. I think these new ideas for shells are good things, and I will definitely investigate them more and see if they make my life easier. I just can't seem to shake this feeling that we can't have our cake and eat it too. Maybe I'm looking at it the wrong way though.
Maybe in 20 years time shells like Oil and Elvish will be the norm, and we'll be complaining about how they don't handle quantum data structures well without lots of pipes and fd's :D
Either way, this has been a very interesting digression. Thanks again for your thoughtful comment and insight.