A statically typed scripting language that transpiles to Posix sh | Hacker News Reader

A statically typed scripting language that transpiles to Posix sh

github.com

251 pointsTaKO8Ki92 comments

This seems neat. I love that there are examples front-and-center on the README page. This is exactly what I want and the first thing I would think to put when demonstrating a new language, but sadly this is rare.

I'm not sure how I'd use something like this, but I've often found myself wanting simple statically typed scripting languages for generating YAML/HCL/etc configs. Dhall seems close, but config is something that every engineer has to touch and the burden of making everyone learn its syntax and functional programming idioms probably negates the advantages of using static typing. I've also looked into CUE, but it seems like it's largely neglecting the problem that I'm trying to solve DRY configs in favor of some problem I didn't know I had (unifying a bunch of layers of mostly-static configs?). This seems even worse than Dhall with respect to burden:reward ratio. I haven't yet checked out Nickel (the statically typed Nix language)--would be curious to hear from those who have.

EDIT: just skimmed Nickel. It seems like it's trying to hew closely to Nix syntax and functional programming conventions, which suggests it probably shares many of the same problems as Dhall. Too bad. It seems like no one understands the assignment.

> I'm not sure how I'd use something like this.

It seems that you could use this to program embedded systems that have a shell, yet which don't support any other usable language (due to flash space concerns, product management approval or whatever).

What I miss in the README page is the output of the transpilation.

I guess the output is not really meant to be readable in any way

if it transpiles to sh then the output should be an sh script. that should be at least somewhat readable.

Yes although it might be readable in the same way that autotools configure scripts are readable -- technically you can, but you probably don't want to, and if you need to then something has gone badly wrong.

yeah, that's kind of what i meant. especially if the script has external commands you should be able to at least recognize something.

I mean yes. Things do go wrong, especially with bleeding-edge projects such as this.

You'll have to look at that output if you want to debug things, understand error messages.

As such, I'd absolutely see it as a quality measure for a transpiler how readable the code it produces is.

I think you can liken this to web assembly. The output might be text but it’s not intended to be human readable.

That's not a given, think typescript compiling to minified javascript. Some compilers like coffeescript do put in some effort to have the output be readable but I wouldn't assert that it should be a goal for everyone

> typescript compiling to minified javascript

This is an odd example. TypeScript compiles to readable JavaScript for nearly everything except the odd runtime behavior (eg enums) and “downleveling”, which is increasingly unnecessary. Minification is still a separate step AFAIK (if you’re using tsc; obviously ESBuild and SWC are another story, as will be Bun).

You caught me, I've never written typescript. I don't think that changes my claim though

I think I have varying and stronger feelings about it depending on context. Personally if I were to use this or any compile-to-shell thing, I’d probably prefer it produces unreadable output. I don’t want to ever read bash if I can help it. If I have to, it’s a good sign I’m already beyond my preferred scope for using bash in any way.

Whereas if I’m compiling TypeScript/CoffeeScript/anything to JS, it’s pretty likely I already have JS as a built in constraint and I probably want to be able to read the output so I can debug it.

It's possible to express many common features and functions of other languages in sh fairly well, so it's probably possible for a tool like this to output something not too bad.

But even so, I would never expect any transpilation output to be more than technically readable.

The part with the high level meaning is the part the human wrote. That's the entire point of any language.

The output can not be considered source, regardless that it's possible to also write source in that same syntax.

The only reason to even bother with transpiling instead of just requiring the necessary interpreter, is to ship something that the target already supports. It's very very little different than shipping a binary.

The differences are just that a .sh is at least possible to read and hack vs not-possible. Not convenient or good, merely possible vs not-possible. And a .sh is more portable than a binary in that it doesn't care what the cpu or os is.

If you want anything more high level meaningful instead of some 90's web page generator html output, you have to write that yourself. An automated tool cannot reverse engineer some code into an elegant expression of the original intent of that code. In that way a compiler or transpiler is not much different from a decompiler.

Though, I admit there is room for levels. I don't know just how crude this tool's output is, but I know there is a range of what's possible, it does't have to be

Also, source maps a la JavaScript.

I, too, missed that in the README. I was curious enough try it out in a vm. Here's the output for fib.li: https://termbin.com/pagk Or for something more complex, raytracing.sh: https://termbin.com/4rpd

I was mostly wondering how/if it does any type checks for function input; it does not. Also raytracing.sh unfortunately fails to run on my system as it is.

I love that there are examples front-and-center on the README page

except that those examples are to simple. for something that transpiles to sh i would like to use it to write better shell scripts. so i'd like to see examples that resemble actual shell scripts in that they call lots of external commands, do file operations, terminal interaction and other things commonly done with sh.

Felt the same. Most complete example seems to be the installer for the language which is written in it. This covers some of the things you mentioned

https://github.com/cotowali/konryu/blob/main/konryu.li

and the generated shell script

https://raw.githubusercontent.com/cotowali/konryu/dist/konry...

There is also the website, heavily under construction, but has some more documentation on the language than the readme

https://cotowali.org/

So this 366-line script compiles into 588 lines of shell, 2,229 lines if we count the parts that seem to come from the imported modules. That includes a lot of auto-generated names like `_cotowali_tmp_166` or `first8b04d5e3775d298e78455efc5ca404d5` and replacements for built-in shell functions like `eprintln` or `rm_r`. In fact that script seems to use `eprint`, `eprintln`, `println`, `print`, AND `echo` which seems a little much.

It would have been nice to see something more transparent, e.g. a type-checked extension to shell, like TypeScript is for JavaScript. Something to make it easier to write correct shell scripts, but would generate understandable, editable shell scripts.

> simple statically typed scripting languages for generating YAML/HCL/etc configs

Do you actually want a statically typed _imperative_ language for generating configs?

Or do you want a sane, statically typed, declarative language for configs?

I haven’t seen a sane declarative language. I’m pretty sure what I want is an imperative language, especially since that’s already well-understood by the target audience. I’m open to persuasion, but I gave Cue an honest shot but it seemed to regard reuse as an afterthought at best rather than the main problem to solve (I think reuse was possible, but it wasn’t obvious and it seemed like it was only supported by accident).

You might find Cue interesting: https://cuelang.org/

It's a bit overambitious IMO, but it tries taking a relatively different approach to the problem that I find to be quite interesting.

Parent already mentioned their investigation of CUE.

Alright, here's the hot take: The assignment is for any developer without basic knowledge of FP beyond .map and .filter to complete their intermediate-level programming education.

All problem domains that involve declarative programming, such as configuration (Terraform, Kubernetes, Nix, many build systems), dynamic HTML, relational databases, and pretty much anything that is manipulating data in a pipeline fits functional programming like a glove and is hard to get right in a strictly traditionally imperative fashion. As an industry we should stop pretending FP isn't a necessary part of a professional programmer's education and that we can solve the problem by coming up with the umpteenth "we promise it's not FP" abstraction with its own take on fooling developers into writing functional code through an imperative-looking layer full of foot guns that subtly break imperative developer expectations, like React.

I think it's a super hot take because the real world has demonstrated (over and over again) that a huge portion of professional software developers have no interest in learning functional paradigms just to write some config files.

Just like a huge portion of professional software developers have no interest in learning RTL or digital logic just to program an FPGA.

HLS is a massively complex system to solve this problem. People will also build massively complex systems to solve the problem of "No FP here, please".

I agree with your point here, though imo it's worth drawing a distinction between the FP concepts the language designers / implementors must understand and the FP concepts the language users must understand.

Ideally languages prevent bug classes like null deref, use after free while still being familiar enough for most programmers to use.

It's analogous to the spectrum of SQL knowledge folks have. For instance knowing what a group by does vs how a group by reshuffles data on a single node / across nodes. Ideally the language design hides the hard details.

A very tall order admittedly.

That'd be fine if it was just config files, but that's not the problem now isn't it?. It's an enormous amount of problem domains that FP handles trivially and is a complete pain to manage imperatively. We've had entire groups of people coming up with ridiculous ideas like replacing relational databases with key-value stores for relational data just because they find SQL "icky" because it fits a mental model they never developed (syntax issues with SQL aside) and it isn't until they're hand rolling data aggregation that they realize the mess they got themselves into.

Which is why I stand by what I said: this knowledge is not optional, and engineers not willing to learn them are going to be stuck with sub-optimal tools, dragging the whole industry down with them. While I can't be confident this is the definitive solution, I believe that the best way to move forward is to start treating basic FP (purity, data transformation pipelines, immutability, favoring data structures over code) as a baseline that everyone needs to know to work as a developer, and stop treating it as if it is on a superior level of difficulty compared to learning whatever bullshit combination of half-templated server side rendering, 4D DOMs and incantations needed to upgrade Webpack we need to know to be "up to speed" nowadays.

> (over and over again)

Eh. I'm not saying one way or the other that FP is good or bad, but the software industry is incredibly young and changes a lot. Even in a relatively short period - 10 years, let's say - things are pretty difference. 20 years, still a short time, and things are radically different.

And on top of that, it is still the case that a lot of stuff takes a long time to get adopted. It's hard to build and popularize a programming language, regardless of how good or bad that language is, because they tend to lack interoperability with other languages if they're very different, which is usually a requirement if they want to be considered worth looking at.

So I'd just as easily buy that people haven't used a lot of FP due to inertia, a lack of the right external inventives, etc, and not by any virtue of the paradigm.

Do you have resources to this end that you'd recommend?

The easiest hands-on approach I know of is to go through the Elm tutorial and build a simple CRUD app. The rest should flow from there.

After that, https://fsharpforfunandprofit.com/series/why-use-fsharp/ and pretty much anything made by Scott Wlaschin is very accessible, and to the point.

Missed opportunity: fsharpforfunandfunds

I mean, maybe. I’ve dabbled in FP a fair amount, but I was never convinced that it’s categorically better for developing software than imperative programming. Certainly it has its strengths and I enjoy it. But ultimately it doesn’t matter because the reality is that imperative languages dominate the industry, and I doubt that will change for the next decade if only for inertia.

so what then is the friendliest functional programming language for those of us with lots of... non-fp experience but very little fp exposure?

Imo, Elm is the friendliest pure functional language. It's a frontend language though.

I'm a backend developer by trade and Elm made me fall in love with frontend development, which I've rather hated before.

The ML family is the most reasonable IMO. Languages such as OCAML and ReasonML feel like they’re designed to solve real world problems rather than adhering to type manipulation pedantry.

"Standard" Scala (i.e. without the fancy monadic libraries like cats or zio) and Rust are both mixed paradigm languages with good support for many FP constructs.

In practice you can write imperative code using either of them but the focus on immutable data structures, and (in Rust more than Scala) functional error handling means that many manipulations are easier to do with FP constructs, so they appear quite naturally (Option, Result/Try, iterator maps etc.)

As a bonus IME they are very similar in many respects so experience in one translates well to the other.

The functional stuff in Nickel is really pretty vanilla, and the syntax seems much more inspired by ML than Nix. There's some info in the repo's `RATIONALE.md` that specifically answers "how is this different/better than Dhall/JSonnet/Cue/Starlark".

> making everyone learn its syntax and functional programming idioms probably negates the advantages of using static typing

I think this is an over-exaggeration of the complexity involved because Dhall/Nickel/Nix are just JSON with variables and functions. If you're familiar with JSON and any general purpose programming language, say Python, you can learn the aforementioned languages in less than an hour.

> Too bad. It seems like no one understands the assignment.

You mean all those language designers for static configuration languages have no idea what they're doing? I think there's a better explanation. For complex configurations that warrant the kind of high level configuration languages we're talking about, declarative syntax seems to a better fit. Tracking mutation and state for large scale configuration would put the burden on everyone, beginners and experts alike.

I'm surprised to see that the compiler is written in V (https://vlang.io/). Last I heard, V was vaporware, and a lot of its claims about performance etc. ended up being "todo" items rather than facts about the language as it actually existed. But that was several years ago; has anything changed?

Otherwise, fun project! I could see this actually being useful in some niche cases, e.g. generating scripts that need to work on some exotic architecture that has a POSIX Sh implementation.

It might also be useful for writing interactive shell "plugins" that set up things like a fancy PS1 prompt.

I would love to hear more about that. Does someone here have an experience with V lang and can tell us if its claims are at least close to truth?

This was my impression too and it looks like they've made really impressive progress since I last saw them on HN. Their bug list [0] looks kind of scary, but that's maybe par for the course for a language? I remember the bad old days of Go's 32-bit GC not being wonderful for example.

[0]: https://github.com/vlang/v/issues?q=is%3Aissue+is%3Aopen+lab...

Maybe? Some of those sound like unit tests should have caught them.

looks very much like Rust's issue tracker to me

Agreed! Also... I was just going through smhasher. Thank you for it!

I generally prefer h https://h.christine.website/

h is the core framework language of my global infrastructure

Would be good to go check out their documentation and/or modules, to get a feel for the language: https://github.com/vlang/v/blob/master/doc/docs.md https://modules.vlang.io/

The documentation and modules are truthful. What they have claimed to set out to do, is what they are doing and where they are going.

V is a beautiful language, but got tons of flak for being overambitious if you're generous, a scam if you're being suspicious. They've been called both on this very site.

I follow it, and love their ambition...but there are a ton of open issues that convince me it's not even alpha quality yet. If anyone has experience otherwise, would love to hear.

Rust, Go, Zig, etc... have a huge list of issues. Talking they have thousands of issues... Which is plain to see on GitHub. I think people have to be clear to separate the hate (or sneak hating) that can be spewed out by advocates of competing languages versus the reality.

Nice, a language written a pretty obscure language (V) and using a pretty obscure build tool (z). I have a fondness for people using unusual tools for their hobby projects.

POSIX shell is a good compilation target if you want to write a non-trivial build system, but be able to run it on some old or otherwise challenging systems.

I see 2 possible rationales to transpile to posix sh:

1: To remove bashisms/kshisms/zshisms to create output that works on ash/dash/busybox/xenix/sun/... In which case, the more useful source language is bash or ksh etc, not a new invented one.

2: To generate a portable output like above from a better language than any *sh (*). In which case the source language should be something already common like python or perl or ruby etc (even if just a subset).

In either of those two use cases, I don't think it makes a great deal of sense to make a new language, and especially not to write the compiler for it in V of all things.

That relegates the whole project to being just a curiosity or a purely academic excercise.

(*) I use every trick bash and ksh offer, so I know it's possible to do all kinds of stuff in some *sh's, but a lot of the features in them are encoded into practically inscrutable special syntax and rules and tricks, where more purpose built programming languages make all the features more explicit and spelled out -> readable. bash is a powerful enough language, but not a readable one, except for rairly stupid scripts that either don't do much, or do everything the hard way.

What about, to gain saner semantics (static typing) while retaining the portability and versatility of shell scripts?

Typical shell programming idioms are quite dynamic. You can give saner semantics, allowing a fair amount of static checking, without abandoning this dynamicity.

What Andy Chu has been doing with Oil Shell is promising in this respect. It supports optional type annotations. Cf. https://www.oilshell.org/blog/2021/01/why-a-new-shell.html

"What about, to gain saner semantics (static typing) while retaining the portability and versatility of shell scripts?"

I think I was insufficiently critical in a misguided intention to be nice. Let me correct that a little :)

If the need is great enough for a DSL, then the need is great enough for python or c or any of 100 other fully fledged universally available languages, and either require it's interpreter or ship a binary.

However annoying it may be to require a ruby interpreter or something, there is essentially no case where the most sensible answer is this "Vala-for-sh".

It's the worst of all worlds. Taking the whole life cycle into account, it's not saner or portable or versatile, it reduces all of those.

Do you know of any "common language"-like (e.g., Python-like subset/DSL) that compiles to sh, to enable `curl -sSL https://tool.example.org/install | sh` use-case.

Neat!

A really cool enhancement to this would be if the compiler could target a polyglot script that is both Shell and Batch. For example: https://stackoverflow.com/a/17623721

Then you'd get a single script that would automatically be portable and executable on pretty much every conceivable platform.

It's pretty nice! I'd love to see the compiled output of the examples though!

Wow this is a great advert for V (the programming language it was written in). Having never written a line of V before I can still read and understand the code.

I had exactly the same reaction. Extremely digestible syntax. Didn’t even notice until after looking it over that it had been written in a language I don’t know.

This is amazingly great for how simple the idea is, going to have to play with the output a bit to poke around for strange edge cases (it is sh, after all) but given all the random devices out there with Busybox or similar this could end up being significantly useful for configuration management, etc.

Also seeing quick, readable examples right on the github landing page is such a small but wonderful thing.

Would be nice if the language supported a #! bang line in order to make it a bit easier to execute, rather than manually typing "lic run X". Although of course on Linux you can always use https://en.wikipedia.org/wiki/Binfmt_misc without needing the bang line.

As I understand shebangs, one could use something like this:

#!/usr/bin/lic run

looks like the task runner part [1] has unfortunate name overlap with the family of programs that include z.lua[2] or zoxide [3] or z.sh [4].

Not really that big of a deal, I just like tracking command line apps naming because it is an interesting exercise of being short, intuitive, and unique. When you decide a single letter is your name, surely you are going to run into this.

1. https://github.com/zakuro9715/z 2. https://github.com/skywind3000/z.lua 3. https://github.com/ajeetdsouza/zoxide 4. https://github.com/rupa/z

What’s the main use case? I guess build scripts?

Build scripts are one of the assumed use cases. Build scripts are one of the assumed use cases. The other use case is installation scripts. cotowali's installer is written in cotowali itself. https://github.com/cotowali/konryu

I don't see any other comments about this so I'll ask: what other options are there for similar use cases? I.e. "statically typed scripting languages that transpile to Posix sh".

Definitely a niche that needs a solution - cool to see one!

Let me preface by saying this is pretty cool.

I feel that this seems like a good idea on paper, and a bad idea in the wild.

Last time I looked into shell syntax and such, I found out that most shells don't. They vary so greatly that very large context-free grammars have to be used to cover most of their context-sensitive rules. I don't think being POSIX compliant changes that.

I suspect that if this got popular usage, the bugs would be quite numerous.

I wonder whether there's a subset of shell syntax that's stable enough and powerful enough that it could be used as the compilation target whilst avoiding the incompatibilities you mention?

I also wonder if targeting bash which has things like pipefail and arrays could make things easier in the implementation side?

Unless I'm misinterpreting the example, it seems rather unfortunate that you need to create a sentential value before a `read` call to update some `&n` location. The `twice` function really could be just `return read() * 2` no?

I guess I should look into how this is actually compiled to sh. I agree the README would benefit from adding some examples of this.

Neat idea, but it seems to me like a critical part of making a new language of any kind is documenting it so that others can learn it and use it. On that note, can anyone explain the syntax that seems to use pipes in a function declaration (i.e. every function in the readme example other than fib)?

It appears to me those are the types the functions expect to consume and output when used in a pipeline.

Does this provide real data structures like arrays, hash maps, etc? And can you pass them around?

So it's APE (actually portable executable) but for POSIX systems which instead of binary uses scripting.

nice

Back in the day i had a similar idea, and i even started implementing it, but never had the resilience to finish. It waa too difficult for me at that time (probably even now). I am glad that finally someone had the courage to do it.

Wouldn't it just be better to make a new shell that has this syntax?

Better in that you can use it directly, not necessarily great for interop. I love Fish shell, but it can be a pain when trying to use tools that expect posix syntax to work.

There's already a plethora of shells that fit that domain:

  + Nu Shell
  + NGS
  + Powershell
  + Oil Shell
  + Elvish
  + Murex (which is one I'd written)

Not to mention all the LISP, Python, and other such high level language REPLs come shells.

But they solve a different problem. This seems to be more like what Typescript is to Javascript: a alternative "safer" language where your end point is limited to supporting a different language. So you need to compile down to a known portable base language.

Would appear that V isn't as vaporware as I initially thought.

Going to give this a fairly through once-over because I think Posix sh is an under-appreciated transpilation target.

I wonder how many complaints shellcheck would have about the resulting script :)

Could this be done with LLVM?

Why not just build a binary? What's the benefit of transpiling to sh?

Binary files are different for each architecture, but shell script is not so. Most scripting languages require a interpreter to be installed. Output of Cotowali is shell script. so no need to install any interpreter.

nice. I do a lot of bash/sh things when need to automate some task, now they can be programmed in a propper lenguaje( sh can be painful sometimes) and just transpile it to sh

There are no examples which show that the source language is good for shell scripting tasks. Making shell programming tasks nicer doesn't appear to be a goal.

Although not yet in the sample, Cotwali is also intended to use for shell programming tasks.

Awesome idea, but "cotowali" is a truly terrible gibberish sounding name, maybe even worse than klubernetes.

So a project with primarily Japanese support with what looks like a Japanese lead dev (https://za.kuro.red/) should use easy English words? K.

Is the audience intended to go beyond people who are proficient in Japanese?

V is such a nice and simple language, it's Go for system programming

Might as well check out my POSIX sh static site generator https://mkws.sh/ if you're into this stuff.

Reply on news.ycombinator.com