Electric Clojure – A signals DSL for full-stack web UI
github.com
github.com
IMO this is a paradigm shift similar to GC. No one wants to manage their garbage. No one wants to manage the network. It's grunt work. Let the compiler take care of it.
I've naively rewritten a re-frame (react/redux in js) app that started getting prohibitively slow due to a bunch of deserialization on the client in electric and the performance is outstanding! TBH I'm taking a break from additional UI work on the app to do research because I realized that with electric I can build something much more amazing than I thought possible before.
Caveat: the future will not be served on a silver platter. it's early days and there are some quirks
Demo videos: https://hyperfiddle.notion.site/Electric-Clojure-progress-De...
Prior HN discussions addressing many of the FAQs: https://news.ycombinator.com/item?id=28630209 https://news.ycombinator.com/item?id=31217448
Some notes on the network layer: https://www.reddit.com/r/Clojure/comments/vizdcc/hyperfiddle...
I'll try to address some FAQs in another post
If I understand the idea correctly effectively here you have a platform where alternative choices are simple rearrangements of code.
One thing I'd be particularly concerned about is debugging. One of the things that's on the roadmap is "developer experience", and I don't see any particular debug docs or helpers, so it tells me that if there is a bug somewhere debugging it is going to be a pain, even more so because I won't easily be able to see whether the problem is my client or server code as they live together and are compiled into separate things. As a critique the fact that developer experience is on the roadmap rather than thought about from the beginning is a little off-putting, but maybe that's not what was meant here.
Another thing that gives me hesitation is that I can't see how I would extend the setup to support offline capable applications. This is increasingly becoming a big thing for web+desktop apps, does this easily support that?
Lastly, the docs seem to point to use a homegrown DOM library, rather than one of the major players. How easily would this work with a library like Reagent, Helix, or UIx?
And yet. It's also true that HLL (high level languages) like C, Java, Python all faced the same issue. The benefit is a huge gain in terms of ability to reason about ever larger programs. Java gave us garbage collection. Not everyone liked that; C++ developers want to manage their own memory. But at what cost?
Our thinking is: first get the semantics right. Then polish the experience (today we are here). Then add observability tools. Then make it fast. And if we've screwed up the semantics, then the project will fail for the reasons you describe.
The saving grace is we're seeing 10x LOC reduction (18k to 2k) in rebuilding Electric's sister project, Hyperfiddle (a spreadsheet like tool for robust UI development), as well as massive gains in performance.
Re. offline-first - we think the model can be extended to that but this is a blog post for another day
imagine a database like DataScript (already OSS and built in Clojure), move key components like the query engine into Electric Clojure, now the "DAG slice" can cut right through the query engine and it's internal data structures, such that the indexes are partially on the client, and the final stages of query filtering are on the client. A network-transparent database! The bulk of the work, and sensitive data, stays on the server; but the last mile user data querying happens on the client. Electric can transparently move portions of the underlying database indexes to the client for local offline query.
0 LOC cost to userland! No loss of relational query when offline! No performance cost of putting the entire query engine in the browser (a low compute environment)!
Implementing it is on the roadmap but that's no evidence that it hasn't been thought about extensively and for something like this you really -have- to think things out up front (it's more like a math proof than a CRUD app in a lot of ways) or you're going to find yourself with holes in both feet and a codebase you have to throw away and restart from scratch.
From what I can see, they've been laser focused on getting the core to be correct by design, which will have required repeatedly gutting parts of it as conceptual mistakes are shaken out, so if you -wrote- the DX code as you were going rather than simply keeping in mind that the codebase needs to be amenable to it -being- written (and periodically thinking through how you'd do so as a sort of mental dry run) I'd expect the DX parts to have to be thrown away entirely, repeatedly, which is a motivation sink and in the best case scenario would've substantially slowed down getting the core right.
But yeah, the key thing is "not yet implemented" and "hasn't been thought about at all" are very different things and I'd strongly suspect that my read of what was meant is more likely to be the right one given the nature of the project.
(please take the above as "an explanation of why I suspect their actions are fine and their chosen trade-offs make sense" rather than "let me tell you why you're wrong" because you do have a point that if you don't keep DX in mind at all in the early stages the end result tend to be unhappy-making, and, well, let's just say I speak from repeated experience of doing that that to myself ;)
Here's a demo of React.js interop via Reagent, the wrapper is 25 LOC: https://gist.github.com/dustingetz/9854d23037b55bfab3845539f...
Our DOM module is only 300 LOC - it's bare metal DOM point writes + Electric (reactive language) + macros for JSX-y syntax. When the programming language itself is reactive, DOM rendering falls out for free.
Mechanically, Electric is comparable to Solid.js except the reactive engine is general purpose, not coupled to DOM rendering, which is a special case of incremental view maintenance.
Electric's reactive engine is https://github.com/leonoel/missionary ; Electric can be seen as a Clojure to Missionary compiler (if you ignore the distribution stuff - which btw is optional). Missionary is cross-platform functional effect and streaming system that today is tested on JVM, Node and Browser platforms. Missionary was created by Léo Noel, who is also the architect of Electric.
A bit unsure where the observation of more offline capabilities comes from. I have the impression that things are moving in the opposite direction: Everything needs to be online.
Ignoring the point that the frameworks we already use are major abstractions on top of abstractions, if this new "abstraction" truly eliminates the complexities of client-server-client it will greatly benefit developers and users alike.
A good many (most?) of the bugs and performance problems (and security lapses) in today's complex webapps is due to the complexities of managing this client-server divide. Even when the developer does things right according to docs and examples, it's possible to end up with a system which does not perform well without solid understanding of what's happening in at the endpoints and in the middle and clever optimizations to solve the special needs.
Regarding the causes of many bugs, your guess is as good as mine. You'll continue to have bugs and you'll have to continue looking under the covers to understand them.
Electric is an attempt to find exactly the right level of abstraction. The goal is to remove and flatten layers, not add them, thus decreasing abstraction weight in the end if we succeed. Maybe we fail, but first let me share some details about how we think about this:
1. I've personally failed to build this project several times, Electric Clojure is something like the 7th attempt.
2. strong composition model as a starting point, based on category theory generalization of "function" -> "async function" -> "reactive function" -> "stream function" -> "distributed function". ORM does not have this. (This rigor is in response to the past failures.)
3. Functional effect system (monad stuff) at the bottom, which provides strong semantics guarantees about glitch-free reactive propagation, process supervision (like Erlang) (transparent propagation of cancellation and failure), strong resource cleanup guarantees (DOM nodes can never be left hanging, event handlers can never fail to be detached and disposed). Already this results in tighter operational semantics than we have ever achieved with manual resource management (and, again, we tried, see past failures). Our functional effects library: https://github.com/leonoel/missionary
4. Electric affords the programmer trapdoors to the underlying FRP/concurrency primitives. Electric is essentially a Clojure-to-FRP compiler, so if you code raw concurrency and effect management, that actually typechecks with what Electric generates, allowing seamless transition in and out of the abstraction.
5. 3k LOC + 3k test LOC is the size of Electric today (includes a rewrite of the Clojure analyzer). Spring Framework is, let me go check, 59k just for `spring-core/src/main/java`, and there are like 20 other modules I excluded. Indeed it is not a fair comparison but certainly we have complexity budget to spare.
Here's what I did - killed the network - added a bunch of items to the todo list, which disappeared from the input field but didn't appear in the list - switched network back on, items appeared in the list
I'm curious to see how adding optimistic updates to the mix would work
Not sure if it is too late, because Elixir's LiveView seem to have gotten important because it works around the same issue.
But it's time to push the wooden stake through the heart of the SPA (and the phone app too, but that's for another day).
Are there any mitigations in place to prevent this, except the obvious "be careful and don't write bad code"? Or have I misunderstood things completely and it's a non-issue?
It seems like you're doing SSR but not exactly.
You return the wrong prop in a getServerSideProps and boom, your stripe keys are dumped in a nice JSON for all to see.
Is anyone aware of automated testing/scanning for secret leakage via js framework magic? If not, how would you implement one?
If you accidentally move filtering of a list of users from server-scope to client-scope during refactoring, everything will still work just as you expect it to and you'll be none the wiser – but suddenly every user has access to all the user data in the list. There's not many other frameworks I'm aware of where moving an operation to an adjacent row or forgetting to change a word suddenly (and silently) exposes the data to the client – except perhaps PHP, which does not have a good reputation when it comes to security to say the least. Altough to be fair adlpz mentioned something similar with Next.js in another thread.
Tagging values as server-only mostly seems like another thing that could be easily forgotten. Personally I would feel more comfortable if values intended to be sent from the server to client had to be explicitly tagged as mutual, with an optional compilation flag that enforces this, such that the intent to share it with the client must always be stated in plain text during creation.
I'd actually take a punt that it's less likely because of Clojure's data-orientation. Given that the above mentioned ORM-preferring frameworks typically have to define a whole new class and map the data from a server-side representation to a client-side representation, you're more likely to see developers not bother, forget or bungle the implementation aspect.
What I'm interested in is how Electric Clojure handles mass assignment (unconstrained deserialization)[1], sort of the flip side of excessive data exposure. If an (e/server) s-exp uses the symbol `is-admin?` will the server respect or discard values for `is-admin?` sent from the client?
[0] https://github.com/OWASP/API-Security/blob/master/2019/en/sr... [1] https://github.com/OWASP/API-Security/blob/master/2019/en/sr...
(e/server (let [is-admin? (e/client ...)]))
I would imagine you can inspect the network traffic. I would hope it would be readable so that you could have a good idea of what's going on and errors would be more obvious.
Your projects are very “futuristic” and the kind of thing I imagine may be hard to get financial backing for, because of the novelty and other considerations… so makes me curious how you balance innovation with monetization in your work.
Thank you!
This make it sound like client and server segments are determined on the fly based on prevailing conditions, but then the sample code has forms wrapped with e/client and e/server. What do you mean when you say you determine “the optimal network cut” and how do you reconcile this with the expressly marked client and server code?
By the way I’ve been following this project for a while and am very excited about it, kudos on getting to this point.
It gets interesting when you introduce lambdas, revealing that this optimization is non-local – you don't know if the transfer is superfluous until you know where the function is called from, and consider the caller and callee together.
The good news is that the DAG (Circuit really) contains everything there is to know about the data flow, so insert compilers research here and poof, insanely fast app.
Congratulations on the release! A great showcase for the flexibility of Lisp.
Others invent whole programming languages around a paradigm, while Lisp just marches on with s-expressions. Well done!
But on the first example, I see a lot of e/client and e/server calls. Am I misunderstanding this? If so, how is this different from meteor.js?
- Would be better to show an example where magic would happen
- Selectable text instead of a screenshot would have been nicer
The idea is that we write what should be done and it is scheduled onto different systems and communication is inserted.
https://github.com/samsquire/structured-logging-2-system
I really like the idea of being capable of switching where something runs on the server or the client by changing a boundary.