Show HN: Esy – fast and simple package management for native ReasonML/OCaml

> - Is there any issues with sharing the same config file with another package manager? Ie. `package.json`?

esy can install JS dependencies just fine but we are supporting plug'n'play installations only (see docs https://esy.sh/docs/en/node-compatibility.html for more info on that).

> - On a similar vein, why JSON, and not a config format that supports comments, like TOML?

Yeah, we thought of that but decided to use JSON for now as it's easy to publish esy projects to npm - you already have package.json ready for publish! We might add support for comments to package.json soon though.

> - Will it be able to do Coq projects as well? I've not had a great experience trying to use Opam with it, and had to resort to using submodules. D:

I think it's possible now (I only use it install Coq itself to work through Software Foundations) but I'm not sure about the details yet.

But considering that Coq ecosystem is moving to dune (the build system of choice in OCaml/Reason native ecosystem) and esy supports dune very well I think Coq is going to have a first class treatment soon in esy!

EDIT: re JS compat: you can esy.json instead of package.json and then esy will use it instead. That way you can use package.json with yarn/npm.

> I can use Esy to write native OCaml/Reason, but has no use in a compile-to-js environment - there it is npm/yarn.

If you use js_of_ocaml (a compiler from OCaml/Reason to JS) then esy works with it very well (with the help from dune). With Bucklescript (another OCaml/Reason to JS compiler) it's more complicated (see below).

> With Esy I will be using Dune as my build tool, but instead of the opam binary I can use Esy itself which can pull packages from the opam repository. It takes care of both packages and compiler environment just like opam.

This is correct!

But not necessarily dune (though this is a very good choice), you can use almost any build system you want.

> Is there anything in Esy that would make porting native OCaml packages (like say https://github.com/c-cube/qcheck) to Bucklescript?

Unfortunately not, esy works on level higher than build systems, it doesn't built code itself but rather calls an actual build system to do the workd (dune, make, cmake, ...). Bucklescript is unique as it tries to own the whole stack and thus you have to make code compatible with its build system (bsb) manually.

One thing which esy could help though right now with Bucklescript is providing native tools like ppx'es (preprocessors for Reason and OCaml code) or tools for editors like Merlin (autocompletion and etc.).

> It would also be wonderful to see it in action through a short video tutorial.

Agreed. Hope to have something soon.

Hi Ben,

> It looks like a nice project, but I'm still a little confused about its purpose. You mention your JS background, so is it all about familiarity to npm?

I think familiarity with npm is important here!

I mentioned that npm allows you to have all dependencies of the project local.

Now if we want the same experience for a native project it gets more complex: we can't just install OCaml/Reason/C/C++ code into node_modules, it's not going to work.

So what esy does is it constructs for each package a "perfectly" isolated build environment based on dependencies declared in package.json, which is then used to produce build output (executable or library). Such environment contains things like `$PATH`, `$OCAMLPATH`, `$PKG_CONFIG_PATH` and etc, so these are all environment variable which C/OCaml/Reason/... software can understand.

Now it looks like this (super simplified!):

    esy build: package.json -> isolated build environment -> build output

Now because such isolated build environment depends only on the package.json (I simplify a bit here... later on that) and the build output depends only on its build environment we can easily compute a cache key for each package build output just by looking at package.json of the package.

So esy has smart cache for builds which makes each package build work like this:

      let KEY = hash(package.json) # this is very cheap
      if ~/.esy/store/KEY
        done
      else
        build(package.json, ~/.esy/store/KEY) # this is not cheap!

It means in practice that if you have completely separate esy projects which depend on the same version of `ocaml` (or `gcc` or any other package) then esy will build that version of `ocaml` just once. So those "every local" dependencies are super cheap once you have esy cache warm.

(the key isn't really hash(package.json) but more like hash(package.json + esy.lock) as we need to resolve dependency constraints to concrete versions first using "esy install" command)

> Or are there other things you gain when you run esy on top of dune?

The correct comparison is to dune on top of opam, I think (opam, the command line program, not opam, the registry as esy supports opam registry too!):

- with opam you manage switches manually (switches are like installed package sets), if you need a switch per project - you create a local switch per project

- with opam every time you create switch you need to compile everything from scratch, with esy we try to reuse as much as possible from the build cache

- you use dune with esy the same way you use dune with opam but you need to prefix dune invocations with esy: "esy dune ...", so that way dune sees esy build environment

- with esy you can pull packages from npm/opam/github/tarballs/...

- there's mechanism called `link:`[2] which allows to develop multiple packages as parts of the same project (so called monorepo workflow).

- ...

Some of these things are possible with opam too (not sure about smart cache though and I think it's pretty important) but with esy we've tried to provide the same experience as when you develop with npm/yarn (of course we wanted to solve gigantic node_modules issue and thus we have pnp, and flaky dependency constraints and thus we have a proper SAT solver, the same used by opam). So you might consider this a different UX to develop native code which is focused on Reason/OCaml.

> And my other question? It doesn't run with bucklescript - right? Are there specific reasons?

It doesn't integrate with bucklescript very well now but you still can use esy with bucklescript projects so you can install Merlin (IDE assistant) or other tools from Reason / OCaml ecosystem.

The way how bucklescript works right now isn't compatible with esy - for example we use pnp[1] installation mechanism (same as yarn) so we don't produce node_modules and thus we are fast and package sources are cached globally - for now bucklescript doesn't understand how resolve package sources from pnp.js runtime and thus it can't work with esy directly.

If bucklescript implements support with pnp then it will be perfectly usable with esy but for now you can use package.json for npm/yarn for bucklescript dependencies and esy.json (esy looks for esy.json before package.json) for native dependencies.

[1]: https://github.com/yarnpkg/rfcs/pull/101 [2]: https://esy.sh/docs/en/linking-workflow.html

And how is the build env recorded for timestamps and env variables? Is all of this standardized?

>we do only integrity checks now for packages downloaded from npm and via opam

Which is fine, but esy should be capable of accepting signatures and checking them against a given key.