Livebook: A collaborative and interactive code notebook for Elixir
dashbit.co
dashbit.co
I imagine two interesting uses for this.
1. producing reports. Connect a livebook session to your system in prod, run a livebook (which pulls data from DB eg), then get back a report. User engagement, cloud resources consumed, p90s, or an incident report...
2. manual database intervention. Suppose you need to manually make a batch of changes to the DB. It would be very cool if you could run a livebook inside of a sandboxed checkout (a temporary transaction) and write your db changes, then write your code that verifies that the db is in a state that you want it to be in. If you mess up your code, then you just terminate the transaction without committing. Then when you're happy with your notebook, you run it once with a commit at the end, and then save your notebook immutably in your records so that you can keep track of what you've done.
I think for intervening in prod, it would be very cool if someone could figure out how to extend erlang distribution over websockets so you don't have to figure out complicated mechanisms (open up ssh port, forward encrypted epmd, e.g.) to get into your system in prod
The problem is that it's one password for the node network, and you either have it, or you don't. So there's no real administrative control over access; it's shared password.
Sure, but I mean...
But how would u get into livenotebook in first place if u shouldn't have access to the container it and the connected nodes run on? I mean u can easily destroy a production system with couple erlang commands
Nginx has an auth_subrequest module that yeets all the cookies in a requests over to a configured URL in a server-initiated subrequest (hence the name), then proceeds with the original request only if the subrequest returns 200 OK. The module works with any type of incoming request, including FastCGI and reverse proxy situations.
The idea is that some config magic turns the 401 into a 302 redirect that walks you through a single-signon page, then you get redirected back to wherever you left off. However since you can make failed requests do just about anything you could technically reroute the request to a WebSocket server that just serves error responses or something, but that feels very... wronky.
The module does feel somewhat unwieldy to use in general as well, although this may be due to general unfamiliarity on my part with Nginx's way of doing things (which I'd summarize as "AAAAAAaaaaaaaa (aaa!)"). (The whole `if`-statements-can-SEGV thing etc.)
The module can only return 200 (allow original request through) or 401 (deny original request); any other response causes nginx to cancel the original request and return 503. (In case of 401, anything else, or error, the FastCGI server or upstream never sees a connection at all.)
It's possible to pass interesting state information such as a username or non-opaque session token from the subrequest to the main request by capturing subrequest response headers then injecting them as request headers into the main request, but this is sufficiently into the weeds (and unlike any existing system Web developers would typically have experience with) that I have a strong tendency to just go \ N O \, but considering that it is admittedly an obscure yet established corner of nginx's configuration, I figured I'd say something for completeness... for the benefit of the 2 people still reading up to this point. :D
I've incidentally been trying to figure out how to set up a simple BeyondCorp-esque system with it for a while now (for remote access to home systems, so focusing mostly focusing on brain-dead simplicity), but the giant brick wall I keep running into is how to a) make things auth-aware, b) keep the cookie properly opaque, and b) prevent everything from exploding horribly if auth_subrequest gets turned off. The only method I can think of to enable (b) (so that individual services don't have to duplicate auth logic, kind of defeats the point) is to pass headers out from the subrequest, which then breaks (c) (making it possible to access things directly).
A small microservice that handles the cookie side of things and makes HTTP requests itself starts looking like a breath of fresh air...
To add, something similar can be done with Traefik and forward auth plugins.
The fact is can connect to existing nodes also makes it can also be great for operation and administration - instead of building an awkward admin panel, you can just having a collection of scripts.
The Elixir community is insanely productive. Also, the Elixir system is impressively pretty balanced - productivity-wise like Rails, concurrent-wise like Erlang, and continuously brings novel yet practical ideas to life in a robust manner!
- Minute 25: Saved notebook is readable (markdown)
- Minute 23: Live collaborate on a single notebook
- Minute 22: Visual indication that cell is stale after an upstream cell changeshttps://github.com/srush/streambook
This is a proof of concept that combines Jupytext for markdown readable notebooks with Streamlit for in-order execution+caching. More difficult to get some of the more advanced features of Pluto / Livebook due to Python state management.
Curious to hear any thoughts.
By in-order-execution you mean pluto's reactivity, right.
But with the idea of real-time collaboration I get it. And with the momentum of Jose and team behind it, I think there's real promise here, especially once they figure out widgets and charts, which they have an open issue for.
I program in Elixir at work, so I usually have an iex shell open, but lately I've been using a LiveBook and it's great.
And I love the new Mix.install in Elixir 1.12 demonstrated here. Using a hex package in an adhoc way was a real shortcoming in Elixir before. Shows the power of having this thing developed by the language team.
Since that time, things have moved quite a bit. Between the Erlang VM JIT, then Nx, and now this, it's starting to look hopeful!
I'm looking forward to digging into this.
This is tremendous work, they're running their business and come up with generic/generalisable solutions instead of narrowing the outcome to their own benefits only.
Not really sure how it sounded like I was diminishing the accomplishments of others. How would you have phrased the point?
Not disagreeing that they are productive and accomplishing a lot.
http://lon-screenshots.s3.amazonaws.com/2021-04-18_Screensho...
And I've heard a lot of concern about it being niche so I tend to mention that it is growing. Because in my experience the biche is quite large and .. growing. I find plenty of work.
The TIOBE Index represents the world of computing at large, including embedded development, industrial, financial, automotive, medical, hardware, etc. This is why C and Java are at the top of the TIOBE index and I'll argue will be there for the next 40 years, and doesn't mean that any language which is not C or Java are unpopular or niche. Ours might be a small percentage of total computing, but would you argue that web technologies/the Internet as a whole is a niche?
There is a reason the embedded world still uses C, and there is a reason the web world doesn't. Using the TIOBE index as popularity contest is misguided.
I don’t think anyone is arguing Elixir isn’t niche, we are just explaining why Elixir gets attention on HN even though it is top 50 on TIOBE. HN itself is focused on certain domains and tends to be early adopters.
How to deal with what? That Elixir isn't as popular, so I should tell my company to drop it? I should stop enjoying programming in it?
Sorry, sounds like you don't want to accept that Elixir is on the up, for whatever reason.
Yes, it's not super popular. Yes, people are being productive with it. I don't get the point of your comment.
Lisp - 142 Clojure - 700 Elixir - 828 Rust - 3160 Golang - 9205 Ruby - 24353 PHP - 68444 Java - 132606
So Elixir, at its peak, has about the amount of jobs Clojure has (never knew anyone using Clojure btw). That's quite a niche, ±800 jobs in the entire U.S is very very little. You might not care or have any action items from these facts, others may. As for companies - I would never choose something like Clojure/Elixir for my startup that's quite crazy considering the small eco system.
> I would never choose something like Clojure/Elixir
Then don't, nobody is making you. I still don't see why you want to stress so much Elixir is not a top language. From what I can tell, you have a bone to pick with the language and being confrontational with no discernible reason. Good day.
EDIT: checked your post history, you're definitely being a troll on purpose on most Elixir posts.
Someone mentioned Elixir has just passed top 50 on TIOBE - that’s growth. It also just crossed top 20 on GitHub by number of pull requests on Q1/2021.
I am not disputing it is niche. I am not disputing the job market is tiny. But for someone who mentions acknowledging the truth, you don’t seem to be very truthful on your claims the language has peaked, especially given the last time you claimed so, you used a milestone (Top 50 on TIOBE) that has been reached since then.
The best I've got is Stackoverflow survey (are you saying Elixir devs don't use Stackoverflow at all? I find it bizarre. Even if you use Elixir forum a modern dev has questions on many things he needs answered (css, javascript, sql etc).
So please compare https://insights.stackoverflow.com/survey/2019#most-popular-... With https://insights.stackoverflow.com/survey/2020#most-popular-... Elixir was dropped from the list. That's definitely a sign of decline. Now it could be that on absolute terms Elixir is keeping it's usage but only dropping on relative terms but I find it a bit hard to believe. I have a feeling Elixir devs who look for the next gig use other tech more often than not.
The language could be shrinking in comparison to everything an still be growing. I don't think that's the case but it wouldn't change anything for me. I don't work with a proportion of client according to trends. I work with particular companies and I'm at capacity doing Elixir as are many I know.
I believe it has a stronger trend then you believe it does. But neither of us have strong data.
And even if they think that’s true, do you think being antagonistic to them on forums is what is going to change their mind?
I remember a recent comment from you saying that Elixir has probably peaked without even crossing top 50 on TIOBE... and yet here we are.
Regarding bills, I made my biggest salary bump after I jumped to Elixir, but I appreciate the concern.
This makes me even more excited to start an upcoming new position writing Elixir full-time!
(For programs that I must have compiled to a distributable executable, I've pretty much settled on Crystal. I find it more productive and pleasent than Go or Rust.)
If you want to experiment with ML and scientific programming where you do a lot of matrix processing, go with julia. thats literally what its made for.
If you're doing computational/numerical stuff, stick with julia, for now. You really can't do that elixir, not today, and maybe not for at least another year or two. It will likely never be as good as julia since computation is not first class in the vm it runs on.
If you're doing or planning or doing anything with web, or orchestration, go with elixir.
LSP is in a similar situation right now, where it's great for languages that lacked tooling to begin with (like Python), but it's not yet possible to migrate existing powerful language tooling (Lisp SLIME, OCaml Merlin) to use LSP.
Not sure what parts are in Jupyter but I've heard collab editing isn't.
Pretty hype about trying this for some living docs and teaching. Will see if I can do anything neat with it. Already tried it some.
I just tried to use the enum_type (version 1.1.3) by just installing it inside the cloned livebook repo, but I'm getting the error that EnumType isn't loaded.
Why is this?