450 karma · joined May 12, 2011
paul.chiusano@unison.cloud
https://share.unison-lang.org/
It's open source, you can create a free account with GitHub OAuth, and you can push projects there and collaborate on them, open PRs, publish releases, etc. It's very quick to pick up if you're already familiar with GitHub.
The tooling takes a little getting used to but it’s extremely powerful. Here are a few benefits you’ll see -
UCM keeps a perfect incremental compilation cache as part of its codebase format, so you’re generally never waiting for code to build. When you pull from remote, there’s nothing to build either.
Pure tests are automatically cached rather than being run over and over.
Switching branches is instantaneous and doesn’t require recompiling.
Renaming is instantaneous, doesn’t break downstream usages, and doesn’t generate a huge text diff.
All code (and code diffs) are hyperlinked when rendered, supporting click through to definition.
I don’t know if you saw these getting started guides, they might be helpful -
https://www.unison-lang.org/docs/quickstart/
And then this tour -
https://www.unison-lang.org/docs/tour/
You can come by the Discord (https://unison-lang.org/discord) if you have any questions as you’re getting going! I hope you will give it a shot and sorry for the trouble getting started. There are a lot of new ideas in Unison and it’s been tricky to find the best way to get folks up to speed.
The Unison website and docs are all open source btw -
I am not 100% sure the origin of the idea but I do remember being influenced by git and Nix. Basically "what if we took git but gave individual definitions a hash, rather than whole working trees?" Then later I learned that Joe Armstrong had thought about the same thing - I met Joe and talked with him about Unison a while back - https://news.ycombinator.com/item?id=46050943
Independent of the distributed systems stuff, I think it's a good idea. For instance, one can imagine build tools and a language-agnostic version of Unison Share that use this per-definition hashing idea to achieve perfect incremental compilation and hyperlinked code, instant find usages, search by type, etc. It feels like every language could benefit from this.
We ourselves make use of this sort of thing since (for instance) Unison Cloud is implemented mostly in Unison but uses Haskell for a few things.
Related:
https://news.ycombinator.com/item?id=46051750 (you can use Unison like any other open source general-purpose language)
https://news.ycombinator.com/item?id=46051939 (you can integrate Unison's tooling with git)
There can be enormous value in creating multiple pieces of tech all designed to work really well together. We've done that for Unison where it made sense while also keeping an eye on ease of integration with other tech.
> sending code over the network to be executed elsewhere feels like a security risk to me?
I left out many details in my explanation and was just describing the core code syncing capability the language gives you. You can take a look at [2] to see what the core language primitives are - you can serialize values and code, ask their dependencies, deserialize them, and load them dynamically.
To turn that into a more industrial strength distributed computing platform, there are more pieces to it. For instance, you don't want to accept computations from anyone on the internet, only people who are authenticated. And you want sandboxing that lets you restrict the set of operations that dynamically loaded computations can use.
Within an app backend / deployed service, it is very useful to be able to fork computations onto other nodes and have that just work. But you likely won't directly expose this capability to the outside world, you instead expose services with a more limited API and which can only be used in safe ways.
[1] Though we might support Unison compiling to the browser and there have already been efforts in that direction - https://share.unison-lang.org/@dfreeman/warp This would allow a Unison front end and back end to talk very seamlessly, without manual serialization or networking
[2] https://share.unison-lang.org/@unison/base/code/releases/7.4...
I'll mention a couple things that might be relevant - you could have the git repo reference a branch or an immutable namespace hash on Unison Share. And as part of your git repo's CI, pull the Unison code and compile and/or deploy it or whatever you need to do.
There's support for webhooks on Unison Share as well, so you can do things like "open a PR to bump the dependency on the git repo whenever a new commit is pushed to branch XYZ on Unison Share".
Basically, with webhooks on GH and/or Unison Share and a bit of scripting you can set up whatever workflow you want.
Feel free to come by the Discord https://unison-lang.org/discord if you're wanting to try out Unison but not sure how best to integrate with an existing git repo.
The cloud product is totally separate and optional.
Maybe we'll have a page or a reference somewhere to make the lines more clear.
It's very recent and we'll be adding more types to it soon, this first PR was just focused on the core machinery. This is in the 1.0 release, btw.
Let us know if you give it a whirl.
> crypto.hash Sha3_256 (x -> x + 1)
⧩
0xs704e9cc41e9aa0beb70432cff0038753d07ebb7f5b4de236a7a0a53eec3fdbb5
We only do this caching for pure tests (which are deterministic and don't need to be re-run over and over), enforced by the type system. You can have regular I/O tests as well, and these are run every time. Projects typically have a mix of both kinds of tests.
It is true that you can only hash things which are "closed" / have no free variables. You might instead hash a function which takes its free variables as parameters.
Overall I think Unison would be a nice implementation language for really anything that needs to make interesting use of hashing, since it's just there and always available.
[1]: https://share.unison-lang.org/@unison/base/code/releases/7.4... [2]: https://share.unison-lang.org/@unison/base/code/releases/7.4...
Re: distributed computing, the main thing that the content-adressed code buys you is the ability to move computations around at runtime, deploying any missing dependencies on the fly. I can send you the expression `factorial 4` and what I'm actually sending is a bytecode tree with a hash of the factorial function. You then look this up in your local code cache - if you already have it, then you're good to go, if not, you ask me to send the code for that hash and I send it and you cache it for next time.
The upshot of this is that you can have programs that just transparently deploy themselves as they execute across a cluster of machines, with no setup needed in advance. This is a really powerful building block for creating distributed systems.
In Erlang, you can send a message to a remote actor, but it's not really advisable to send a message that is or contains a function since you don't know if the recipient has that function's implementation. Of course, you can set up an Erlang cluster so everyone has the same implementation (analogous to setting up a Spark cluster to have the same version of all dependencies everywhere), but this involves setup in advance and it can get pretty fragile as you start thinking about how these dependencies will evolve over time.
A lot of Erlang's ideas around fault tolerance carry over to Unison as well, though they play out differently due to differences in the core language and libraries.
For interesting usage - we built Unison Cloud (a distributed computing platform) with the Unison language and also more recently an "AWS Kinesis over object storage" product. It's nice for distributed systems, though you can also use it like any other general-purpose language, of course.
In terms of core language features, the effect system / algebraic effects implementation is something you may not have seen before. A lot of languages have special cases of this (like for async I/O, say, or generators), but algebraic effects are the uber-feature that can express all of these and more.
Here are a few places where "multi-resumable" stacks are still useful, even outside of nondeterminism:
* For instance, in a workflow engine a la Temporal, a `sleep` primitive might serialize and store the continuation in a distributed priority queue. The workarounds of not having access to the continuation are all not nearly as good.
* A pure interpreter of a structured concurrency ability is quite useful for testing, since it can test different interleavings of threads and produce tests that fail or pass deterministically. For Unison Cloud's distributed programming API, we have an (in-progress) chaos monkey interpreter that you can use for local testing of distributed systems.
* You can implement a simple debugger... as a library. It lets you set breakpoints and go forwards and backwards in time. Here's an example: https://share.unison-lang.org/@pchiusano/stepwise
Basically, any time you want to stash and do something interesting with the continuation, even if you only end up ultimately using it once, you still need the more general form of algebraic effects.
And then there are various nondeterminism effects that do call the continuation more than once. I'd say these are somewhat niche, but when you need them, you need them, and the code comes out much nicer. I especially like it for testing. You generally want tests to just be the logic, not a bunch of looping code or map/flatMap.
Some links:
* https://www.linkedin.com/posts/pchiusano_dan-doel-has-been-d... has some details on the optimization we do in Unison
* https://dolio.unison-services.cloud/s/blog/posts/optimizing-... is Dan's blog post on optimizing affine handlers
* https://www.linkedin.com/posts/pchiusano_kestrel-is-a-higher... is a typed query DSL that uses nondeterminism in an interesting way. For a declarative query DSL, it's nice to avoid explicit looping, similar to what SQL does.
allocationPolicy = 23484
-- two usages of allocationPolicy
foo = allocationPolicy + 1
bar = allocationPolicy + 99
move.term allocationPolicy defaultAllocationPolicy
Next if (say) you wanted `foo` to reference a different definition, you'd `edit foo`, and introduce:
fooAllocationPolicy = 283
foo = fooAllocationPolicy + 1
A couple other notes -
* It's very rare for independent implementations to end up with the same hash. It probably only happens for some very simple defintions that exist in base. (Like the identity function, say)
* If a hash has multiple names in your project because you've used `alias.term` to do so explicitly, the pretty-printer picks one using a deterministic rule (it prefers names you've given that hash in your project, then it consults library dependencies). If you really want to give two definitions different hashes even though they are functionally the same, you can introduce a minor change, like an unused binding.
* The type of a definition is part of its hash, so sometimes you might specialize a more generic function with a more narrow type signature, and this gets its own hash.
* The OP is slightly out of date re: patches. We use something simpler now for updates and merges. When you update, we compare the new and old namespace to obtain a diff, which is applied to the ASTs in the namespace. If the result typechecks, you're done. If not, we make a minimal scratch file for you to get typechecking - it will contain the minimal transitive dependents of the change.
When the source of truth is instead a database, content-addressed by hash, cache invalidation is simple - if the hash has changed, a cached result is invalid and needs recomputing. If the hash is the same, you're good. We use this approach in many places throughout Unison and it's quite robust.
There's some interesting subtleties with FFI in a distributed programming language, so I'll ramble about that here in case it's interesting to you. :)
So, in Unison, all values are serializable, including functions and their dependencies. This is a key superpower that enables a lot of the neat stuff we do. As long as those functions are written in pure Unison, we can easily serialize them, deploy them on the fly, etc. But when we add an FFI, the story changes - the sender may have some C library in their environment, and that library may not exist at the recipient node. The two nodes could be different platforms, one Mac, one Linux, and the library may have been written specifically for Linux, say!
So when we add FFI, we will likely be doing it in a different way than most languages. Functions that use FFI will have this tracked in the their type, using our effect system. You'll be able to use whatever C libraries you want in your local computations, but if you want to sent those values around, you need to be sending them to a place that supports that same set of FFI effects, since the C library and its dependencies can't literally be deployed on the fly in the same way as pure Unison code. In our cloud platform, nodes are typed based on what effects they support and we'll probably add a way to create new node pools that have access to whatever C libraries you want.
In "regular" programming, we're not used to thinking about "the execution environment" as a thing that's represented explicitly within the program. Instead, there's an assumed execution environment (which includes the set of native libraries, etc) and you get runtime errors if you run a program and some of the assumed execution environment is missing (like a shared library, say). For the most part, people have been okay with this, but it's already somewhat of a problem for languages that target the browser and the backend. The language may be statically typed, but now the type system is not tracking some key information - namely, is this a function I can call here (if I'm expecting this code to compile to JS) or is it a function that only can be called for backend code? In a distributed setting with heterogeneous nodes with different capabilities, this problem is even more pronounced, which is why we track this information in the types and plan to do so once we add FFI.
Hope that was interesting!
[1]: https://www.unison-lang.org/blog/jit-announce/ is an early progress report, and I think we're finally shipping something in the next month!
It is true that tech exists that try to make all 3 of those items easier. YMMV, but having used these technologies myself and now having used Unison + Unison Cloud, all I can say is that the Unison experience is quite different overall.
The details matter. A bicycle and a motorcycle share some common principles but that doesn't mean they're "about the same". The fine details of execution and polish can matter too: Slack is different than IRC, Dropbox was different than the million other backup services. Also, bringing a number of things in a cohesive way can lead to big improvements in the experience when it's done well.
Getting into specifics a bit, I don't think deployment with a function call is well handled by existing technologies, because of the reasons discussed in the post. In the absence of Unison's features, there is inevitably some sort of out-of-band packaging step or "setting up the environment with the right dependencies" as a precondition, instead of calling a function and having it Just Work.
Re: RPC, Unison remote calls can pass around arbitrary values, including functions and values containing functions. This Just Works. There's also no generated code which needs to be somehow integrated into your build and no boilerplate converting from the "wire format" objects to your actual domain objects you want to work with.
My experience with ORMs is they are overly opinionated / magical and provide insufficient control for many projects. So every project I've worked on ends up not using ORMs in favor of a layer of boilerplate for talking to the storage layer... which still can't store functions or even sum types properly! And it's not typechecked, either. Using our cloud's storage, I get to use whatever data structures I want, can write my own easily if needed, and I can store any value at all including functions and again it Just Works. And access is typechecked. It's pretty great!
When you put all these things together in a single cohesive programming environment, with a common type system and language, uniform composition, a set of tools all meant to work well together, you really start to see how different it is! It already feels like a huge step up, and will only keep getting better and better as we build out Unison and our cloud platform.
All that said, I'm kind of doubtful that abstract arguments like this will be convincing. Instead, I'd just try Unison out for a low-risk project and decide for yourself if the details are making a big difference for you.
Hope that is helpful! :) If you do decide to play around with it, feel free to come by the Discord https://unison-lang.org/discord to get help, ask silly questions, etc. We are here to help and it's a nice community.
> Interesting. Whats it like upgrading and managing dependencies in that code? I'd assume it gets more complex when it's not just the Unison system but 3rd party plugins (stuff interacting with the OS or other libs).
In Unison, there's an all-in-one tool we call the Unison Codebase Manager (UCM) which can typecheck and run your code and talk to the code database (we use SQLite for this). The workflow is that you have your text editor / VS code open, and UCM in another terminal, watching for changes.
So if you want to edit a definition, say, here's the workflow -
1. `edit blah` brings code into a scratch file, pretty-printed. You make your changes and get that compiling.
2. You type `update` in UCM, and it tries to propagate this change throughout your project. If it can, you're done. If it can't (say because you've changed a type signature), UCM puts the minimum set of definitions in your scratch file. You get this compiling, then do `update` again and you're done. It's quite nice! The scratch files are very ephemeral and not the source of truth.
For library dependency upgrades the process is similar: you fetch the new version, then use `upgrade` to say "I want my project to exclusively use the new version". If everything's compatible, you're done. If there's incompatible changes, UCM creates a scratch file with the minimum set of things to get compiling.
One interesting benefit is you can have multiple versions of the same library in use in your project. Unison doesn't care if you do this (though it can get confusing so people tend to consolidate). But there are cases where we've made good use of the ability to reference multiple "incompatible" library versions within a project.
This post https://www.unison.cloud/our-approach/ talks more about why such radical changes were necessary to achieve what we wanted. (In particular check out the "3 requirements of the dream" section, which walks through what the programming language needs to support to be able to do things like "deploy with a function call.")
My general take on "when and where to innovate" is: if you can get a 10x or more improvement in some important dimension by doing things differently, it can absolutely be worth it. This is the philosophy we've applied in developing Unison over the years. I am generally happy to learn something new if I know that I'll be getting something substantial out of it. Of course it can be hard to tell from the outside if the benefits really are worth the changes. I'm not sure what to say about that, other than try it out with something low risk and decide for yourself.
Besides the distributed programming / cloud stuff, I'll give a couple other examples where we gain advantages by doing things differently: by storing Unison code in a database, keyed by the hash of that code, we gain a perfect incremental compilation cache which is shared among all developers of a project. This is an absolutely WILD feature, but it's fantastic and hard to go back once you've experienced it. I am basically never waiting around for my code to compile - once code has been parsed and typechecked once, by anyone, it's not touched again until it's changed. This has saved me countless hours compared to other static languages. And I don't have to give up static typing to get this.
This sort of content-addressed caching also plays out for testing - for pure tests (which are deterministic), Unison has a test result cache keyed by the hash of the test code. This also saves countless hours - imagine never needing to rerun the same tests over and over when nothing's changed! (And having certainty that the cache invalidation is perfect so you don't need to do a "clean build just to be sure")
Also replied here re: self-hosting https://news.ycombinator.com/item?id=39293568
The "real" cloud platform providing the fancy distributed compute and storage fabric, deployment with a function call, etc, isn't open source - selling this product in various forms is how we are sustainable as a business.
If you're at a company and want to deploy "Unison Cloud in a box" on your own infra or in your own VPC then that's something that's doable and I'd love to talk more - feel free to email hello@unison.cloud.
If you're just an individual wanting to do cloud stuff at small scale on your own infra, that's probably harder for us to support right now. I'd recommend just using the free tier or starter tier of our public cloud. Even if we had some sort of free self-hosting option for cloud, there's economies of scale you'd miss out on so it could easily be more expensive anyway!
Not to mention, time is valuable https://www.unison-lang.org/blog/developer-productivity-real... We've built a nicely managed public cloud that eliminates huge swaths of tedious work. If you're happy to pay for that, then it's a good fit. If you prefer to self-host all the things, even for personal-use scale, then I totally understand that but Unison Cloud probably isn't the best fit right now.
Hope that helps!
We have a few things still in Haskell which we'll probably move into Unison eventually.
[1]: Just to clarify, our storage layer wraps DynamoDB in an interesting way to provide the transactional API we wanted - we didn't literally implement our own cloud database on top of just the file system and some VMs. :)
[1] https://github.com/unisonweb/unison/issues/499
... that said, the language semantics and libraries are still going to be different, so even if we have a python-ish or typescript-y syntax, there'll still be new things to learn. :)
http://pchiusano.github.io/2019-07-06/learning-without-a-gra...
http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
I think you’re right that the area is underexplored.
The responses on this thread are kind of discouraging and missing nuance. It’s turned into “should you tell the hard truth or pretend the world is a fairy tale” (a false dichotomy) when it’s more like “don’t be a jerk in this moment”.
To me that sounds like overhead. Rather than just computing 1 + 1 as a single assembly language instruction you’re sending the 1 + 1 expression tree to a thread’s work queue or something? If I’m misunderstanding can you clarify?
> Compiling a function in isolation is fine, why wouldn't it be? Not sure I get it.
It seems like optimal evaluation is a whole program optimization - you need the whole program available in order to do it, you can’t just compile one function to assembly language in isolation, and then link that against other precompiled functions. Do I have the wrong idea here?
> Just because I used integers as an example, which happens to be optimized, it means an entire optimization technique isn't interesting? This applies to every data structure that can be defined algebraically.
I do think it sounds cool as I said, but I want to see it on an example I care about, not Peano arithmetic. Especially since deforestation is already an optimization that is used by languages that don’t do optimal evaluation. So I want to see how HVM does it better, if that’s the case. And not just you telling me how it’s theoretically better. :)
Lastly, a while ago I remember you mentioning that you couldn’t compile arbitrary lambda calculus terms. Is that still the case and if so can you better describe what can’t be compiled?