ilyash

Welcome to (shameless plug) Next Generation Shell. It's a modern programming language for DevOps. It is both concise for running external programs (uses small subset of bash syntax) and it is a "real" programming language with data structures and sane error handling among other things.

Compared to general purpose programming languages, it was built for "DevOps"y use cases in mind. See https://github.com/ngs-lang/ngs/wiki/Use-Cases

Ability to run external programs is high on the priorities list. In NGS it means having its own short syntax and handling exit codes among other things. NGS throws exceptions on "bad" exit codes. Hint: not for all tools non zero exit code is "bad". Did not see equivalent exit code handling anywhere else.

Compared to bash... It's not fair comparison even. Another era, another reality, other expectations. Small example: When APIs return structured data, well... you better be able to handle it. See https://ilya-sher.org/2018/09/10/jq-is-a-symptom/

Compared to other modern shells, I would say, NGS is programming language first as opposed to shell first.

It's always the same trajectory with declarative programming. It starts with "it's just configuration, we need something simple". Then users come with use cases which are more complex. Then you have programming language on top of configuration language syntax.

* https://ilya-sher.org/2018/06/30/terraform-becomes-a-program...

* https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a...

> successful

As in not throwing exception or as in evaluates to true?

> exit code is > 0

You refused to get into specific exit codes for specific programs citing potentially surprising behavior (which I agree will sometimes be an issue).

> STDERR in ('false', 'no', 'off', 'fail', 'failed')

I correlated this with the following source code:

> if len(s) == 0 || s == "null" || s == "0" || s == "false" || s == "no" || s == "off" || s == "fail" || s == "failed" || s == "disabled" { return false

... which also has a potential for surprising behavior. (side note: appears to work on stdout, not stderr)

> or []byte(STDERR) > []byte(STDOUT)

Does that mean len(stderr) > len(stdout) ?

... also has a potential for surprising behavior. I can easily imagine a program with lots of debug/warnings/log on stderr and truthy/non-error output.

> STDERR messages are useful for functions that return a message rather than exit code.

Might be problematic because typically exit codes are used for that.

> where `if` is evaluating the result of `which` and you don't need to do more complex tests like you would in Bash to test the output of `which`.

Mmm.. Never did this. In which situation looking just at exit code of `which` is not good enough?

> if { which: foobar } else { err: "foobar is not installed }

Just to show off :) in NGS that would be:

    if not(Program('foobar')) {
        error("foobar is not installed")
    }

or sometimes just

    assert(Program('foobar'))

> there isn't really an entry point in murex scripts

I have a nice trick in NGS for that. Under the idea that "small scripts should not suffer", script is running top to bottom without "entry point". However, if the script has defined main() function, it is invoked (with command line arguments passed).

Example - https://github.com/ngs-lang/nsd/blob/afe0cad5e506ec4ee2fa924...

> `args` still contains more boilerplate code than I'm happy with

Is there anything preventing you to have exactly the same functionality but with syntactic sugar that it looks like parameters declaration? (Just to be clear, keeping all the ARGV machinery).

Something like (assuming local variables are supported; if not, it could still be $args[Flags] etc):

  function hippo(name:str, hungry:bool) {
    if { $hungry } then {
      out: "$name is hungry"
    } else {
      out: "$name is not hungry"
    } 
  }

> I don't see how that's surprising because it's literally the same thing you were describing with NGS (if I understood you correctly).

Yes it is almost the same in NGS with regards to exit codes, which you preferred not to do in Murex. On the other hand, checking stderr looks very similar to knowing about exit codes and here you decided to go for it. I'm puzzled why. It is somewhat fragile, like knowing about exit codes.

> It was me who was unclear as I assumed you meant it was the same error handling for builtins and functions too (as is also the case with murex). I just meant the "reality of handling forked executables" as an example rather than as a commentary about the scope of your error handling :)

Here I lost you completely but I hope it's fine :)

In NGS, exception handling is used throughout the language, consistently, well at least I hope it is.

From what I understood - same in Murex.

Somehow I noticed more footguns in bash than in utilities.

> the mystery behind function overloading

In NGS the common case for multiple dispatch would be to define your own type and then add methods to existing multimethod to handle that type; additionally I've simplified the dispatch algorithm exactly for this reason - unclear answer to "what's going to be called actually?".

> function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority.

I don't think it is called overloading. That's very different from having two methods with the same name and the "right one" is called based on types of passed arguments.

> interpreting an indexed argument into an named arguments.

In NGS it happens in exactly one place, when main() is called. At that point command line arguments are automatically parsed based on main() parameters and passed to main().

I don't think I understand your reasoning behind not including named parameters.

> NGS likes to expose its smart features whereas in murex they're abstracted away a little

Sounds about right. Power to the people!

> Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided.

I can see why. NGS prefers to do the right thing in most cases and have shorter and cleaner code. Yes adding some risk, which I see as not that big - exception when an exit code was "ok" for unknown program (unknown external programs default to exception on non-zero exit code).

I also dream about moving the hard coded information about programs into separate schema, which could be re-used between different shells. That would be similar to externally provided typescript definitions for existing code, described at https://www.typescriptlang.org/docs/handbook/declaration-fil...

> differ between Linux, BSD, Windows,

"detect program variant" feature in the schema I mentioned above

> hard coding executable behaviour

Yep, smells a bit

> causes more potential issues than it solves.

I think in particular situation with exit codes the risk is low.

> users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions

Yes. Downside. That's the kind of surprises that suck. Need to make sure at least the docs are clearly warning about this.

> So instead I've relied on having foundational logic that is consistent.

Totally understandable.

> STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across.

Sounds also like potentially surprising, somewhat similar to handling exit codes. Yes, it's not per program and that's why it's better but still.

> Additionally STDERR is highlighted red by default

As it should! Yes!

> mini-stack trace

Yes, please!

> testing framework baked into the shell language.

Same in NGS but https://github.com/ngs-lang/ngs/issues/494 :)

> So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too.

I guess. Differently of course :)

> Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables.

Sounds like I was not clear enough as if something is not uniform in this regard in NGS. Exceptions are not only for external programs. It just happens that external program can be called inside "if" condition; an exception might occur there.

Wow, that's a bunch of reading to do :)

> I've not heard of the term "multiple dispatch"

https://en.wikipedia.org/wiki/Multiple_dispatch (includes example from NGS too)

In NGS, multiple dispatch is based on CLOS model described at https://en.wikipedia.org/wiki/Common_Lisp_Object_System , with my own simplifications.

> Powershell does support this with classes

Somewhat similar. Since methods do not live in classes in NGS, I would argue the mechanism in NGS is simpler (more elegant?). You can just define your_method(c1:Class1, c2:Class2).

> murex does it's overloading at the API level[1].

mmm. I see the support but looking at documentation at https://murex.rocks/docs/apis/Unmarshal.html , I see it's not exposed into the Murex language, it's in Go. Is this correct?

> where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way

That is good and what I would expect.

( sorry, running out of time, to be continued :) )

> being programmer-first is not always a desirable feature.

programming-language-first allows convenient scripting. I could not think of a way to make anything shell-first be convenient for scripting (anything beyond tiny scale).

> higher level language REPL generally makes for a shit daily shell.

I think it proves that general purpose languages, where using them as a shell was afterthought "makes for a shit daily shell".

NGS, on the other hand, has somewhat-bash-like (read: easily run external programs) syntax at the top level, which should be good for CLI. Want to use more advanced features that are typically associated with "real" programming languages? OK, pay the price, switch syntax with { ... } and have full blown programming language at your disposal, in the shell.

> NGS isn’t unique in that regard.

I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments?

> Murex has methods, Powershell has methods.

Just looked at Murex docs. No multiple dispatch mentioned. I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html

There is a huge difference in having methods and multiple dispatch.

> not unique these days

NGS is a mixture of "borrowed" features and unique ones. Multiple dispatch is an old concept and has been implemented in other programming languages. Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes.

Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code.

Edit: and some other features around handling exit codes such as short syntax to provide expected exit code (any other exit code throws exception)

Edit: clarification - NGS knows that some external programs have exit code 1 which does not signify an error.

NGS is programming-language-first while other modern shells are typically shell-first. Multiple dispatch would probably be the most prominent manifestation of this approach in NGS.

Sorry. "elegant" here triggered.

https://ilya-sher.org/2021/03/19/running-elegant-bash-on-sim...

> writing shell scripts in Python to be cumbersome because whatever flavor of os.system() I end up using just doesn't work well syntactically.

This is exactly how I felt. Bash can't handle structured data well. Python (being general purpose programming language) can't handle calling external programs well because it's not the "focus" of the language. My shameless plug solution is a "real" programming language that can do both well (along with proper handling of exit codes and more goodies for "devops"y scripting).

https://github.com/ngs-lang/ngs

More about this feeling when you don't have a well fitting language for "devops"y scripting - https://ilya-sher.org/2017/10/10/why-i-have-no-favorite-prog...

> Not all thoughts are concise and straightforward. That's not a problem that can be solved.

My observation is that inexperienced programmers have this kind of mess in their head. No programming language can help with that. Hard to tell because I'm not sure what exactly you mean.

> Concision by itself is not something to optimize for,

Concision is one of the aspects. Too much and the code is unreadable. Not enough and code is too long. In both cases mental effort grows and maintainability suffers.

I see what you mean. I am more focused on personal perspective (all languages suck). It's good to be more aware about the perspective. I'll also try to look from the suggested perspective too. Thanks!

Resonates. I typically say that my programming language should just suck less than others for intended use cases.

> specific way in which it sucks.

This

I don't have objective data but it seems that I'm reaching the "they did what?" pretty quickly when I read about yet another language. Don't need to start using it even.

I'm working on my own programming language since 2013 so I am thinking about programming quite a bit. So here is the summary of those years:

* All programming languages suck. We are not there yet as humanity. I can not think outside of the box enough to make the breakthrough that should be made, at least as of now. While looking at other programming languages, I see the same issue: inability to express thoughts in concise and straightforward manner. Unsolved issue.

* The only idea in programming that I consider definitely good is pattern matching. Note that I'm talking about the concept here. No particular implementation strikes me as "yep, this is it": be it limitation to strings only (regex) or syntax-only implementation (preventing composability of the patterns) or some other sh*t.

Have a nice weekend!

Shameless plug follows.

I think shells are horrible not only from (today's) programming perspective but also from UI/UX perspective. In response to commands, shells dump text to your screen: from stdout, from stderr, from different programs, humanly unreadable amounts - they just don't care, it's "not the shell's job" apparently.

Here is my take: the shell's job is to do everything for my productivity. Imagine programmers arguing that "notepad" is the best, IDEs are not needed, we need to keep text editing pure.

How about shells output objects on the screen and these can be interacted with? Amazing for shells, right? This exists for decades everywhere else.

My vision for UI in NGS - https://github.com/ngs-lang/ngs/wiki/UI-Design

Looks pretty close :)

What happens in ABS if curl fails in this example? Or JSON parsing fails? In NGS, it's an exception with a stack trace.

> fun part

NGS is mostly use-cases oriented - https://github.com/ngs-lang/ngs/wiki/Use-Cases

Decorators are not implemented yet because there are other things higher up the list. Currently working on deeply nested data structures because it's needed for our use case for FHIR - https://github.com/ngs-lang/ngs/wiki/Deep-Data-Structures-Ma...

> cumbersome writing commands

In NGS, there are two syntaxes: "commands" and "expressions".

Commands is bash-like. It's for running external programs.

Need some fancy stuff? Switch to expressions and pay the price.

I didn't see (and still don't see) any way to make a "real" programming language based on bash-like syntax.

> A little collaboration is fine, .... Bezos structured Amazon so that teams were as independent as possible.

Maybe too much independent? It looks like AWS had roughly zero collaboration between teams. I wonder how they even get IAM to work with all the services.

* Basic things like how you call pagination fields is f%cked up. https://gist.github.com/ilyash/f845d050df5c7c0805183790b28ac... . That precludes straightforward implementation of wrappers and what not.

* Code being published of quality which says (99% sure) it was never reviewed (at least not by someone proficient in programming and the language).

* CloudFormation lags supporting new features of other services.

Hello. Author of Next Generation Shell [1] here. I've skimmed ABS documentation and was surprised by the amount of similarities in thinking and reasoning between ABS and Next Generation Shell. Differences that popped into my head in limited amount of time follow:

* ABS has CLI and NGS does not yet, CLI of NGS will be written in NGS itself, so it takes time. The CLI in NGS will be radically different [3].

* ABS runs on Windows.

* ABS has decorators, NGS does not yet, still thinking how to do it right and more generic.

* ABS has some functions that NGS doesn't and vice versa.

* Found inconsistently named functions in ABS (json() vs tsv() - json() parses JSON while tsv() generates tsv); given more eyeballs on NGS, such inconsistencies should come up.

* NGS has user defined types.

* NGS has many predefined types, ABS has few.

* NGS has multi-methods and multiple dispatch.

* NGS has exceptions.

* NGS has (partial but already very usable implementation of) Universal Pattern Matching [2]

Translating example in the docs:

    # ABS

    res = `curl -s 'https://api.ipify.org?format=json'`

    if !res.ok {
      exit(1, "An error occurred: %s".fmt(res))
    }

    ip = res.json().ip
    total = ip.split(".").map(int).sum()
    if total > 100 {
        echo("The sum of [$ip] is a large number, $total.")
    }

    # NGS

    ip = ``curl -s 'https://api.ipify.org?format=json'``.ip
    total = ip.split('.').map(Int).sum()
    if total > 100 {
        echo("The sum of [$ip] is a large number, $total.")
    }

In NGS the double backtick syntax above runs external program and parses the output. The check of `ok` is not needed because on error exception will be thrown.

[1] https://github.com/ngs-lang/ngs

[2] https://github.com/ngs-lang/ngs/wiki/UPM-Design

[3] https://github.com/ngs-lang/ngs/wiki/UI-Design

Before there are 10 people with "there is a library for that in Python" (because they are not and can not be ergonomic enough):

Next Generation Shell is a "real" programming language. It also has syntax for calling external programs. It handles exit codes properly. It has facility for constructing command line arguments.

Project - https://github.com/ngs-lang/ngs

Motivation related to the above - https://ilya-sher.org/2017/10/10/why-i-have-no-favorite-prog...

edit: ah, yes, NGS also supports command line arguments automatic parsing when main() is called; augmenting this facility would be the sane way to solve the problem.

My rant about “elegant” and “bash” in the same sentence:

https://ilya-sher.org/2021/03/19/running-elegant-bash-on-sim...

This is how you "integrate" `jc` with Next Generation Shell ( https://github.com/ngs-lang/ngs ):

    data = ``jc PROGRAM ARGS ...``

The double-backtick syntax runs the external program, `jc` in our case, and parses the output. It means that the "integration" is not `jc` specific.

`data` is now structured data that comes from the parsed JSON.

Example (run from your shell):

    ngs -pl '``jc ifconfig``.filter({"name": /docker/}).ipv4_addr'

Will print IPs of all docker interfaces, one IP per line

Opinion: Today's shells should have structured data; `jq` is a symptom of a shell that can't handle structured data.

https://ilya-sher.org/2018/09/10/jq-is-a-symptom/

My bias: I'm the author of Next Generation Shell.

I have considered different constructs like "where:" to be placed in code blocks (such as function) when working on Next Generation Shell. I have decided against because these constructs will not be executed at runtime (when the function is called) and therefore I assume it will be at least somewhat confusing.

For tests, NGS has "TEST" keyword which can be placed at beginning of any line. The test is everything till end of that line. I typically place these below functions that they test.

In that case, welcome to Next Generation Shell (I'm the author), which actually cares about errors when running external programs. Yes, I mean much more than naive "non-zero is error" because that's just not true in many cases.

https://ilya-sher.org/2017/01/28/ngs-unique-features-exit-co...

... and hope that $FILENAME does not start with a dash... just in case you don't have enough landmines to step on.

> name the pieces

Solved without splitting into functions.

https://ilya-sher.org/2019/10/21/section-syntax-next-generat...

> I'm 97% sure you're going to stick with your own language

Agree with the estimation :)

> but for others seriously give Ruby a look

Of course. Python is not out of the question either. I think it's mostly about alignment of how you think and what is possible to express in the language in a straightforward manner.

> It's so easy and elegant to shell out

That's the area where I think NGS has better facility than pretty much anything else I've seen out there. In NGS, it's the "domain". From the top of my head:

  * Short syntax for run-the-command-and-parse-output (currently auto detects JSON but customizable) - like backticks in bash but double-backtick on each side.
  * backticks syntax like in bash (but don't strip the last newline)
  * Argv facility for constructing command line arguments (if the command is complex)
  * Automatic handling of exit codes. Errors throw exceptions and no, not any non-zero is an error.
  * ok: option to specify non-error exit codes for particular run of a program
  * log: option to log the command being run

More at https://github.com/ngs-lang/ngs/wiki/Use-Cases#use-case-ops-...

> Worth noting too is that Ruby also has a fantastic one-liner command line syntax like perl and awk for example.

NGS is somewhat there and somewhat getting there