Elixir Livebook is a secret weapon for documentation | Hacker News Reader

Elixir Livebook is a secret weapon for documentation

fly.io

312 pointsbehnamoh71 comments

To ensure you do not miss this: LiveBook comes with a Vega Lite integration (https://livebook.dev/integrations -> https://livebook.dev/integrations/vega-lite/), which means you get access to a lot of visualisations out of the box, should you need that (https://vega.github.io/vega-lite/).

In the same "standing on giant's shoulders" stance, you can use Explorer (see example LiveBook at https://github.com/elixir-explorer/explorer/blob/main/notebo...), which leverages Polars (https://www.pola.rs), a very fast DataFrame library and now a company (https://www.pola.rs/posts/company-announcement/) with 4M$ seed.

The vega-lite wrapper is very slim too, basically just converts your slithery_snake keys into humpyCamels and corrals between different data sources, so if the docs on the elixir page seem sparse its because most of the "how do I build a graph like x" is just on the vega docs. You can easily dump it to json to share too, etc.

Good note :-) The wrapper can by the way be used outside of LiveBook (e.g. https://twitter.com/thibaut_barrere/status/16885335909659033...) to provide charting in an Elixir app.

It's a shame because elixir documentation is gorgeous

Oh I needed vega-lite badly in my life, thanks so much!

This concept sounds quite a bit like the usage of .mdx files in OCaml. Those are markdown files which can load and execute OCaml code in them.

They are useful for unit testing code like libraries, providing usage examples for end-users (as well as the text format markdown usually supports) and testing it at the same time. Definitely a great idea for documentation, and one which I should use more of.

https://github.com/realworldocaml/mdx

Apart from running code inside a "markdown" file, livebook can do much more. You have Smart cells to show charts, run sql queries against a db, run Neural Network tasks such as Image-To-Text generation using Bumblebee[1], etc. It is collaborative as well.

[1] https://github.com/elixir-nx/bumblebee

After reading your comment, I watched the first half of this video: https://youtu.be/4hVIxyHxwK8

Wow!

Yes, more uptake of mdx in general. E.g. I use Astro static site generator and they offer it too.

https://docs.astro.build/en/guides/markdown-content/

While I'm glad the idea of tying documentation to tests is catching on, I think markdown documentation -> create tests is the wrong way around.

I built this tool to let people generate evergreen markdown documentation from annotated type-safe YAML integration tests:

https://github.com/hitchdev/hitchstory

This is incredible work.

To anyone curious, I highly recommend:

- https://hitchdev.com/hitchstory/approach/

- https://hitchdev.com/hitchstory/why-not/

From the overall RDD/BDD type home page:

- https://hitchdev.com/hitchstory/

The entire product site is a thing of richly informative beauty.

---

My only question was whether the generated 'docs' snippets would add value over just reading the story in your DASL. Any markdown site generator (such as the chosen Material for MKDocs) can just embed the ```yaml anyway. But then I realized what was generating e.g. …

- https://hitchdev.com/hitchstory/using/engine/rewrite-story/

… and how superior that is to typical docs, especially typical docstring or swagger factories.

Thank you!

The ability to execute sample code during documentation generation seems invaluable. Instead of being subject to rot, documentation turns into an executable test suite.

I've been working on something like this for Kotlin using a compiler plugin that allows code to access the source text of lambdas, functions and classes being executed. You write code that spits out markdown and captures its own source into code blocks.

https://github.com/mfwgenerics/kapshot

A related Elixir feature you might also find interesting is Doctests: https://elixir-lang.org/getting-started/mix-otp/docs-tests-a...

You can write inline examples in your docstrings and they’re automatically turned into unit tests. So again you get that documentation as an executable test suite not subject to rot, but at a per-function level. Same kind of idea but different levels of the documentation stack.

I still don't understand why more languages don't have doc tests.

I think there's two main things:

- Elixir's scripting makes it super easy to write a one-liner that tests a function, that'd be a lot more verbose in many other languages

- You need the docs tool and unit tests tools to 'speak the same language' to make it work, and for whatever reason Elixir devs don't seem as intent on reinventing tools like those as some other languages

This is neat! Cheers

That's fantastic. We use Kotlin to build Taxi (taxilang.org) and orbitalhq.com. this would be an awesome addition to our docs.

However, do I understand it correctly that this allows me to output verified, working code in my test suite, but its not a tool that I can embed in my docs to create runnable code samples? (which is what the OP is showing).

super valuable either way, just trying to validate understanding.

That sounds about right. The approach I'm working on won't allow you to embed runnable code in your docs. Instead, it requires you to write your documentation in Kotlin e.g. using a Markdown DSL where source code from the codebase can be included in the output. I'm working on a separate project to programmatically generate file trees and Docusaurus doc sites in code.

I'm not aware of anything exactly like Livebook in the Kotlin ecosystem but you might be interested in kotlinx-knit which takers a more incremental approach to executable documentation:

https://github.com/Kotlin/kotlinx-knit

Neither of these approaches will result in interactive documentation.

I haven't used Livebook, but I think Kotlin Notebook is very similar - https://www.youtube.com/watch?v=2PLYlDJPelQ

I would suggest to anyone to download Livebook and test it out, even if you don't know Elixir or Erlang. You can do so much with it. I even use it for things outside of programming due to it being pretty much a markdown notebook that you can run code inside of.

It's one of my favorite tools.

For linux the easiest way to download livebook is with docker [1] or maybe with fly.io [2].

Installing it directly is a bit complicated if you are not familiar with it. I wish linux also had Desktop app like mac.

[1]: https://github.com/livebook-dev/livebook#docker

[2]: https://fly.io/launch/livebook

I managed to install it with devbox (nix under the hood), works very well !

That's really cool! Was it as simple as running `devbox add livebook@latest`? Or did you need to configure it in any way?

To do so much with it you need to know Elixir or Erlang ;)

You actually don't. You can use smart cells which lets you do a lot without actually coding. You can transform data, build charts, run AI model and more all without writing code.

You might have to Google how to read a file in Elixir, but you don't actually need to know Elixir.

In the future, you won't even need to know that to use it.

Ah. I never actually looked into smart cells.

Mind blown.gif

Fwiw, I’ve been using Elixir professionally for 5 or so years and even created a few open source libraries. Never once had to touch Erlang.

Erlang is very nice. I prefer it for writing libraries nowadays. It’s much easier to use Erlang from Elixir than the other way around.

Or, not and. Livebook just added Erlang cells.

If you're interested in this hybrid of docs and live coding, I'm building something similar for the Typescript / Javascript / React ecosystem @ https://www.typecell.org (open source). See https://twitter.com/YousefED/status/1677342430545301504 for a preview of the next version which will be more "wysiwyg" than the current notebook environment

Today, in a very round-about way, one could use a livebook as if it were a blog post: download-install Livebook on your machive, download the blog post livemd file, run it locally.

Here's an example. Note in the first paragraph a link to the post as a livemd file: https://genericjam.com/blog/image-processing

If you go through the steps, you'll notice that the livemd isn't stylized as a blog post. This is where the next opportunity lies: creating beautiful blog posts as livebooks, and without the need to install and run a livebook server locally.

Is there a way to render the livebook as read only until they try a change? I tried to open the link and it failed. I could download it and run it but I just want to see it as you intended.

There's a link on that render page to download the livebook file and run locally on your own livebook server

One small note: It looks like the 'Mix standalone' runtime option was removed late last year. Instead, you can use `Mix.install/2` pointing to your project directory:

  Mix.install(
    [
      {:my_app, path: Path.join(__DIR__, ".."), env: :dev}
    ],
    config_path: :my_app,
    lockfile: :my_app
  )

(The above snippet assumes a notebook located within a `notebooks/` directory, as per the article.)

Alternatively, you can connect to a running instance, e.g. a Phoenix project:

  $ elixir --sname my-app --cookie cookie -S mix phx.server
  $ LIVEBOOK_DEFAULT_RUNTIME=attached:my-app:cookie livebook server

Sorry for asking a dumb question, but how is this different from using Jupyter to write your interactive documentation/runbooks and interactive tests?

While Livebook will mostly have an appeal for people already working with erlang/elixir, it does have a few features that are pretty nice.

- it’s collaborative (think Google docs for code) when several people are working on the same instance of a livebook;

- it’s easy to extend with so-called smart cells (which are essentially pieces of gui you can inject in your document https://news.livebook.dev/v0.6-automate-and-learn-with-smart... ). Smart cells are available for various tasks (db connection/ interaction, data frame exploration, ML tasks, maps), and building your own is relatively easy;

- you can turn a notebook into a web app ( https://news.livebook.dev/deploy-notebooks-as-apps-quality-o... )

- you can run your code an a remote elixir node by attaching to it (although this requires some knowledge of distributed elixir/erlang )

thanks for the detailed answer of the differences. The interactive (I assume using CRDT's) is a good feature, jupyter server does not have this and is at a time a bit of a hassle. Also deploying it directly as an app is pretty sweet.

Livebook actually uses state reducers as described here[1]. Except for cell content. There it uses Operational Transformation. The source code[2] is really instructional and easy to understand. I highly recommend to look into it :)

[1] https://github.com/jonatanklosko/notebooks/blob/main/article... [2] https://github.com/livebook-dev/livebook/blob/main/lib/liveb...

A great read, thanks so much!

> how is this different from using Jupyter

It requires you to run a far less common tech stack and limits your hiring to a vanishingly small subset of developers.

Paradoxically, that may be a good thing.

This is just, wrong. There's actually a huge number of Elixirists, but after this long in the game, we're used to cynical elitists...

Love it! yes, it might be better. We have found paying above market average and being tighter on interviews has a similar outcome.

Elixir market is already more high paying than python.

According to SO’s survey, Erlang is the highest paying technology or something like that.

Outputs markdown, easy to share.

Jupyter outputs JSON.

For NodeJS enthusiasts, Check out Docusaurus (by Meta).

For Markdown Docs, Markdown Blogs, and JSX-static pages

- https://docusaurus.io

- https://docusaurus.io/docs/next/api/themes/@docusaurus/theme...

For Node enthusiasts, learn Elixir or Erlang and don’t waste your time with Node anymore.

This is about live "documentation". Docusaurus doesn't have programmable cells linked to live processes or systems. Docusaurus is fantastic, but it's static. Livebooks are like next-generation REPLs with slick UI, security, reproducibility, etc.

Hey, Docusaurus maintainer here.

I'm not sure to understand what you mean by "Docusaurus is static". Docusaurus builds static pages and allows you to plug JS/React code anywhere in your docs, so it's quite interactive and can run anything that can run in a browser, including REPLs.

I read the Fly article but still don't really understand what Livebook is about. They say they use Livebook themselves, but the examples linked to only display a regular non-interactive doc to me.

Do you have any production url showing me an experience that is possible in Livebook, and impossible in Docusaurus?

Sure, here's a simple demonstration of what livebook can do that docusaurus cannot, at least without maintainer expertise ;)

Imagine the following.

<here's a code block containing project metadata>

we click a play button to start an elixir runtime based on that metadata code block

<here's a "component" that securely connects to an elixir system running in a kubernetes pod in production, exposing an elixir REPL to interact with it>

with this component, we then issue commands to inspect state, manage elixir "processes" in the runtime, etc

> we click a play button to start an elixir runtime based on that metadata code block

Can you show a concrete example? IE a real production url running this?

What is "metadata"?

In Docusaurus you can have a live playground evaluating on your browser, or you can embed any embeddable playground if it requires a server integration.

> with this component, we then issue commands to inspect state, manage elixir "processes" in the runtime, etc

Another example would be useful.

So this is just an embedded widget to interact with something remote? Why can't this be built as a React component that you can add to any Docusaurus page?

---

It looks to me that you don't need maintainer knowledge to build that, and React knowledge is enough.

Not to be rude. The live playground probably can’t run elixir/erlang code. And it most certainly cannot connect to a remote running node instance. It’s unfortunate, that the angle here is documentation since, while Livebooks are great for testing and documentation ( https://blog.appsignal.com/2022/05/24/livebook-for-elixir-ju... ), they also provide a great environment for experimenting with elixir code. I’m sure docausorus is great, I’m not sure why you would think Livebooks would be a competitor in the same space. As to you asking for in production links, I’d encourage you to install it and test it for yourself.

here's a recent video recording of the author of Elixir showcasing some latest releases for Livebook: https://www.youtube.com/watch?v=dSjryA1iFng

The code blocks need to be run in sequence, from top-down, as one builds on the next. The "metadata" at the very top often describes the project and its dependencies.

After reading this article, I'm still not sure to understand what Livebook is.

Can someone show me a real production url of what is possible to achieve in Livebook and impossible/difficult to achieve with other tools?

I'm the Docusaurus maintainer, and making your docs interactive, and giving the ability to run the documented project inside its doc does not feel like something new.

One simple thing I need thats stopping me from jumping into Livebook fully: Salesforce API integration. Right now, I can't manage to make the auth work there (since oAuth needs some http redirect dance).

Connecting to some proper SQL database is a breeze, but if I cannot access some silo'ed data that is absilute critical, its not that useful after all :-/

Please open up a discussion on Livebook repo? We would love to understand more what is missing here. :)

I slightly complain a single time for the last few years, and jose directly chimes in with kind words immediately. You rock, honestly!

But on the topic: I think its a simple problem ultimately, the SF Api (https://developer.salesforce.com/docs/atlas.en-us.api_rest.m...) needs OAuth, which needs a redirect at some point, where I have no idea how to capture it from a livebook instance (easy in a phx app).

This is a reference lib in JS ( https://jsforce.github.io/ ) and there seems to be no maintained alternative on hex.pm unfortunately. So, from scratch it seems, somehow.

Additional insight: many big companies do use salesforce, and its internal tooling to produce reports/insights is rather limited, but there is a SQL-like language to do complicated stuff, but there is no way to execute a handwritten query in SF itself, except via API calls from outside. This... would be a perfect fit fro livebook! query SF + other databases to generate a report ad-hoc thats not easily possible right now without adding some heavyweight data warehouse connection!

I wonder if we could proxy OAuth flows to notebooks. For example, a notebook could initiate an OAuth2 flow, Livebook handles it, and once it gets back, we notify the notebook which may approve or deny the request. The biggest concern though is that we need to have a static callback URL, so you would need to run Livebook internally at a fixed address?

If this is something you would like to work on, please reach out, I think it could open up interesting possibilities!

update: this specific issue is something I could work around differently: defining a user that has its own credentials within SF who can be used in the auth flow did the trick without any redirect needs!

See: https://gist.github.com/Anonyfox/3f36c26fb00a9bc4067d6dc9c9a...

I sound totally dumb now to bitch about it, but glad I could solve it in like 1 hour in a sufficient way

currently starting to run stuff on fly.io, so the notebooks are public (albeit password protected) - should work this way!

The OAuth2 authorization code workflow involves a redirect step.

Org-babel can do a lot of this, no?

Yeah but to be fair, it's a bit gnarly to get set up and it can't do interactive graphs.

Don't get me wrong, I love org babel but there are some advantages of the browser based approach.

You’re right. I love emacs, though I must admit it’s quite a barrier to entry

this is really interesting. I've never thought on using this or any jupyter-like tools to document working software. Maybe I'll try it later

Jupyter outputs to JSON. Livebook outputs to markdown, so it's more naturally renderable outside of the tool itself.

There are still some rough edges -- it would be nice if livebook could cache prerendered content. for example, kino graphs only chuck a big fat JSON into a html comment, it would be nice if it also popped out an svg or encoded png with data uri.

I believe you can hover the corner of a graph and hit the (...) disclosure for a "save svg" button.

Well it should be automatic and it should inline it.

xcode swift playground are supposed to be the same kind of ideas. However, it's really one of the worst maintained piece of software in the whole toolsuite. Really a shame..

hopefully this will get us closer to a orgmode-ish confluence that can run any language, with CI docs test checks, and something more human than jiraql or whatever

Reply on news.ycombinator.com