UIs are streaming DAGs [video] | Hacker News Reader

UIs are streaming DAGs [video]

hytradboi.com

235 pointssimongray72 comments

On the surface it looks really cool, but I'm deeply skeptical because they've intentionally put off handling what I think is the hardest part making the idea production ready: handling network unreliability.

from the talk:

> Photon is at least as reliable as today-era web applications, there are lots of well-understood ways to approach reconnecting local-first etc. We have not done this work yet

I think this really understates the truly immense amount of engineering work that goes into making modern web system software appear reliable to users. Maybe Photon has a good solution but if my years in engineering have taught me anything it's that every line of code written and design decision made before tackling the hardest part of a problem need to be redone, so you should start with the hardest part first.

I am hopeful that I'm proved wrong though :)

(Speaker here) We appreciate your concern, the risk is real, and you're right – the hard parts cannot be put off. And we have not put them off. We've already thought through how to implement recovery, resync, long running sessions. We have several wiki pages worth of notes answering questions like how we will deal with network partitions, reconciliation, durable session state, dealing with OS sleep timer state, OTA code updates, etc.

What we have not yet done is the implementation work, because our pilot use case driving our eng priorities – an internal support app for a Series B SAAS startup – does not actually have this theoretical problem you describe. Photon's protocol today is tolerant to arbitrary delays, IFF you can guarantee message delivery and ordering, which is the problem TCP solves. We do already have pending/loading states, which are trapped locally through reactive try/catch.

BTW, the Photon codebase is only 2k LOC (analyzer, compiler & interpreter for JVM and JS targets, standard library) and a substantial part of that is redoing parts of the Clojure/Script analyzer infrastructure.

"truly immense amount of engineering work that goes into making modern web system software appear reliable to users"

That's a bit much for me. The current industry state of commercial SAAS/crud apps is a dumpster fire, Photon's speed and responsiveness is already miles ahead of literally every laggy SAAS tool we all suffer where programmers have manually hand-coded the network in the form of REST calls, manually split HTTP routes, backends-for-frontends, client side ORM, GraphQL resolvers, etc. How often do you Cmd-R your gmail or notion because it stopped working? React.js-era software is rotting to death, "reliable" is not a word that remotely describes my daily experience with web software today.

> our pilot use case driving our eng priorities – an internal support app for a Series B SAAS startup

While your passion for the project and excitement at its prospects are clear (and I've very much enjoyed following along), this comment makes it sound you've opened a yak barbershop inside this poor support department.

Wanted a vanilla zendesk implementation; got a new UI development paradigm.

> While your passion for the project and excitement at its prospects are clear (and I've very much enjoyed following along), this comment makes it sound you've opened a yak barbershop inside this poor support department.

Perhaps something useful comes out of it. If I recall correctly, CMake (the de-facto standard build system for C++ applications) originated as a yak shaving subproject of a medical visualization application.

they are excited by our “lowcode for CRUD apps” RAD tool whose next-gen I/O requirements are the reason we made Photon- “minimum level of ambition that is useful” - see http://www.hyperfiddle.net

I'm happy to hear you're in a battlefield environment that's supportive of the additional opportunity, and I'm excited to see what pops out next.

I hear the Facebook people once tried to make the unread notification counter stop getting stuck, and React came out :) In all seriousness, though, if people are constantly trying to shave the same place on the yak (like web frontend state) ... Maybe there’s just a concentration of unsolved suckage there?

Worse, I am pretty sure. That counter is what spawned the Flux architecture. Because React had no solution, mind you.

What is your Series B startup? How does it use what you are building here? And how did you get it funded? :)

> Photon's protocol today is tolerant to arbitrary delays, IFF you can guarantee message delivery and ordering, which is the problem TCP solves.

TCP solves that only within the context of a single TCP connection (which I’m sure you know).

That doesn’t seem to handle even common moving laptop use cases, let alone actual mobile users.

> That's a bit much for me. The current industry state of commercial SAAS/crud apps is a dumpster fire, Photon's speed and responsiveness is already miles ahead of literally every laggy SAAS tool we all suffer where programmers have manually hand-coded the network in the form of REST calls

This is a fair criticism! I am biased because my work history has placed me in companies that really care about maintaining this illusion.

> React.js-era software is rotting to death, "reliable" is not a word that remotely describes my daily experience with web software today.

I will say, that some of this is the result of how hard the problem we're talking about is. While React could definitely improve the affordances for this logic, having worked with a few systems that try to maintain the illusion of a reliable network I've found inherent tensions balancing speed, performance, cost, and time the UI spends in an inconsistent state. It's a Hard Problem™.

Regardless, I'm wishing you the best of luck. I'm heartened to hear what you have is 2k lines, it will make a re-write much less painful if it's needed :)

"I'm heartened to hear what you have is 2k lines, it will make a re-write much less painful if it's needed :)"

My takeaway was that they successfully identified the small essence of the problem ("It's DAG, stupid!") and wrote a relatively few but intelligent lines of code to solve it. And they have gotten pretty far with the implementation.

God does not require rewrites of such.

For Dustin, since you're following the thread:

The area you're covering is closely related to an active area of development within the JS frontend framework community. They consider it hydration perf work and the working terminology is partial hydration or resumption/resuming rendering (there's a couple approaches/tradeoffs). In particular, the Marko team has a system under development that works this way without explicitly defined client/server pieces. They're mostly solving it on the server and streaming the HTML down since they have a priority on page load perf but the models look very close to me. I don't have a definitive piece to link as an intro and the concepts haven't made their way into the broader JS community but I thought you might be interested.

I'm happy that hyperfiddle is still going, I recall the Clojure NYC presentation and this is looking a lot more polished.

There's a lot of gotchas with SSR and hydration. I got bit just today because Next.js hid all the complexity from me so I was surprised when localStorage didn't work. Just popped up an error about it being undefined -- in my browser. How could that be? Took me awhile to realize it was trying to render it server-side.

There's a lot of gotchas/drawbacks with the current approaches, which is why it's an active area of development. Remix, Astro, Qwik, and Marko are all taking different approaches at mixing client and server side rendering.

> There's a lot of gotchas with SSR and hydration

I don't mean to be dismissive, but most of us still do classic SSR (like we did in 90s/00s except maybe with Rust/Go instead of PHP) and it's still awesome and resource-friendly (both for server and client).

I assume you meant there's problems with "hydrating" client-side scripts (Next.js in this case) and templates for use server-side. Which is fine if you need it! I'd just like to point out that 99% of web pages don't need any form of client-side rendering logic (apart from the browser's HTML/CSS rendering engine) and are much more user-friendly without any scripts running at all, even if you leave aside memory/privacy concerns. And if you really need some client-side interaction for some reason, i've found HTMX to be a very refreshing approach to this question.

EDIT: To be clear, i'm not saying this for you specifically, but for someone who would read through this topic and currently believes they need client-side scripting to build "modern" websites.

(speaker here) I’m with you, HTMX is great for a very wide class of websites.

our Photon technology is for next-gen stateful applications like an IDE or low-code tool, not websites.

How can we scale cloud software applications to parity with desktop quality UIs and beyond?

(this was not covered in the talk, we will clarify it, thank you for the feedback)

This talk blew my mind when I saw it "live". The lexical separation between client and server, within the same program description, reminded me a lot of Distributed Algol (see the chapter in Practical Foundations for Programming Languages). DA uses a "spatial" type system to track which site a fragment of program executes at, and it's ridiculously cool to see this kind of thing leveraged for realsies. The site-based program slicing in Photon, and the API synthesis across that boundary, is really exciting!

I didn't watch this video, but https://github.com/purpleidea/mgmt/ has a core concept of streams of DAG's since the beginning. I came up with the idea, but if there's a greater commons discussing these kind of algorithms, I'd be interested for more people to get involved!

That looks like a really good project!

great project, for others here’s the best problem/solution overview i could find - https://purpleidea.com/blog/2018/02/05/mgmt-configuration-la...

This is cool, but how is state managed here? Does the server need to maintain any per-client state between renders? Does that have scalability or concurrency implications? I'm having trouble discerning the ramifications.

Modern stateless rest microservices are popular for many reasons, including ease of scaling and ease of reasoning about performance.

This entire system seems very stateful.

Also, in the real world, requests to one web service end up cascading out to multiple web services, with differing permissions models. For example user comes in with a token, makes a request to a web service, which then has its own secret store that it fetches an API key from to make a request to another web service.

Very rarely in life has it been as simple as "fetch data from this API". It is "fetch data from this API given this auth token and that API takes that auth token and uses it to get another auth token, attaches some tracing info for diagnostics in case things go wrong, writes some metrics to a DB so the team can track API usage, then goes out to a couple more services, gets data from them, collates it, and eventually gives some data back to the user."

I'm hopeful about this approach, honestly. We do build stateful flows in web applications, today, but we have to manually manage the serialization of that state to a shared store so that whichever worker the next request hits can pick up where things were left off. It seems to me that a system like Hyperfiddle / Photon could automatically manage the collection and persistence of this kind of state, so that the "server"-located code can be scaled horizontally without change to the written program. I can even imagine an Erlang-like solution to updating application code, so that existing flows can gracefully upgrade or time out.

"Could", of course, is the operative word! But given that we have to manage state now anyway, this seems like an approach with a lot of potential.

Sure, i think you're basically describing distributed actors with durable state, and yes the Photon model can be viewed as distributed actors (with amazing syntax). Other great teams are working on the infinity scale streaming infrastructure, we'd like to be the last-mile UI for next-gen end-user applications.

(speaker here) regarding the limits of stateless apps see https://driftingin.space/posts/session-lived-application-bac..., the trend is that the more complex your app (think an IDE), the more the "stateless" approach is too slow and breaks down. (Note that what people call "stateless" is not stateless at all; there's database query effects, network effects, session state – and then on the client we have dom effects, reactive rendering and component local state. Modern web apps are actually entirely imperative except in isolated little components where we pretend)

The system we describe in the talk is indeed stateful, but it's worth pointing out that the Photon runtime is referentially transparent in the pure functional programming sense. People think functional programming is about avoiding state and effects; it's not. It's about gaining control over our effects so that we can orchestrate scaled up fabrics of millions of fine grained effects without loss of composition, reasoning or control. This is the promise of modern functional effect systems, and is the underpinning abstraction that powers Photon.

+1 for understanding that "stateless" doesn't necessarily mean better OR more performant. There's too much reductionist thinking based on not enough evidence. I'm not building a twitter clone, i'm building something that needs to frequently re-evaluate business rules at each step of a lengthy workflow. Reactivity is a natural fit. And there's state that exists there. Where do you want it to stay? (please don't say 'in a jwt')

There's room for so many different solutions to so many different problems. Shooting something down because its "stateful" well, okay if you know your problem domain well and that's significant to you then so be it, but that smells to me like "I have nothing valuable to contribute so I'm going to throw out a meaningless non-sequitur". I'll happily prioritize developer productivity first when user experience is not meaningfully impacted. I suspect that ~95% of web developers are in that same position.

Does this approach end up causing you to keep around more state (or other resources, like connections or whatnot) than would be typical in the traditional approaches? That's kind of where my mind was wandering with my comment earlier—not expecting statelessness for every application, but wondering if it becomes any more difficult to manage state than it would be normally. (Part of why I thought this might be the case is that people naturally try to minimize resources when they have to keep it around explicitly, but when it's done implicitly, I'm not sure if that becomes more of a concern or not. I could see it going either way.)

Hopefully the author is still here. I skimmed the Twitter thread but couldn’t find the video referenced at the end. Can someone provide a link?

The hypermedia demo video didn’t make it out on time (due to me not having time to narrate and produce the actual video). Soon!

Please, if you have to use an abbreviation in your title, define it in the synopsis. If you use an abbreviation in audio/video, use the unabbreviated term first and the abbreviation immediately after.

I’ve never heard of DAG, yet it’s used as if it’s commonplace.

I thought they meant Directed Acyclic Graph [0].

[0]: https://en.wikipedia.org/wiki/Directed_acyclic_graph

You have to make some assumptions about what is common knowledge in your target audience.

Probably, the author didn't design the title for a general public on HN, but for Getz' fellow software UI researchers.

Someone that's not in the target audience will have to do some extra work to get up to speed. Just like an undergraduate will have a hard time understanding a graduate text without a primer.

Should they have defined UI? Should they have defined DB? It is subjective where to draw the line for commonplace, and it depends on the audience.

Is DAG a commonplace acronym for people attending HYTRADBOI? Maybe (I see 2 other talks mentioning graphs/trees in their title).

It costs nothing to use the full term once or twice. It can prevent confusion, and some people in the audience may not know the concept and googling a few letters is often not the best way to get useful results. Saving a couple syllables in a talk isn't worth it.

> I’ve never heard of DAG, yet it’s used as if it’s commonplace.

DAG, as in direct acyclic graph, is indeed a basic concept of computer science.

But indeed the author could do a better job with the text. One of the very basic principles of technical writing is to always introduce the definition when an acronym is first presented, something like "blablabla a direct acyclic graph (DAG)".

Wouldn’t really call it a “basic principle”: as with anything else, it really depends on what you assume your audience knows (and you will need to make assumptions about that). If you are describing advantages of your new CRDT over past ones, most likely expanding that into “conflict-free replicated data type” will not help any of your readers; if you’ve come to a statistical physics lecture, “BBGKY hierarchy” might be even more evocative than Bogoljubov-Born-Green-Kirkwood-Yvon (seriously, I know the concept but just had to look up one of the surnames).

Of course, it’s possible to speak about either of those things without assuming this knowledge, but this is then a very different talk, most of which will be useless to most of the people who do already have it (not all—it’s useful to occasionally read introductory stuff on topics you think you know!). This is essentially the same thing as power-user usability: it can and does evolve into gatekeeping if left unchecked, and a fresh perspective is a useful check, but assuming everyone is a novice is not the right approach either.

(Some mathematics books choose the worst of both worlds: you really do need to know X to read this book, but “to make this book self-contained” we just included a short two-chapter summary of X that instills horror in anyone who doesn’t know it and forces everyone who does to dig through every word searching for conventions or non-standard assumptions. I’ve been in both categories, sometimes simultaneously. On the other hand, other introductory sections of this kind are remarkably crisp summaries that I go around recommending to everyone.)

> Wouldn’t really call it a “basic principle”: as with anything else, it really depends on what you assume your audience knows (and you will need to make assumptions about that).

Directed acyclic graphs are a basic computer science topic. There is no way around it. DAGs are introduced in intro to algorithms and data structures courses, right on the first semester of any first year course.

Arrays, linked lists, trees, graphs. Far from obscure topics. Well, DAGs lie right between trees and graphs, and are typically the very first type of graph presented to freshmen.

Also, DAGs pop up all the time in practical applications.

I think you misunderstood what I meant (and I failed to check the surrounding language to make sure it was unambiguous where the quote was from). I was referring to you saying (emphasis mine)

> But indeed the author could do a better job with the text. One of the very basic principles of technical writing is to always introduce the definition when an acronym is first presented

and saying that no, not expanding the acronym is not always a mistake or sign of inferior writing.

It's commonplace in data engineering circles, so if that's where the author comes from it's understandable that it wouldn't occur to him that it's not the case in every discipline of programming.

(speaker here) you’re right, sorry, i actually messed up a slide transition that presents the full form, i will also verbalize the full form in future versions, thank you for the feedback!

Are there any other spots that need better explanation?

Not that I’ve noticed, thanks for taking it on board!

The idea and explanation is great and sounds like it can work.

To use the general startup cliche of “execution over ideas”, show me the code/demo ;)

Would love to see this work (having toiled away for 20years writing FE/BE/DNS glue) and you can make a pretty penny too.

I get the idea, but I'm not convinced about its usefulness. I fear that it would make it really easy to write horribly inefficient apps, interspersing client and server code willy-nilly, ending up with an entangled async mess that noone can really reason about. I saw this unfolding first-hand in functional reactive codebases...

Sometimes explicit boundaries are a good thing. In fact, I would argue that they are more often than not.

I really can't see how unifying the value domain and language semantics of the back and front end necessarily involves more complication than maintaining the mapping between them.

ORM frameworks also unify the value domain and the language semantics of the database and the host app, and make it very easy to shoot yourself in the foot with the N+1 query issue and such.

This will suffer from the same issues once you move beyond toy examples.

just for your one example, it seems pretty clear from the discussion that the datalog planner here making the cut across the network connection specifically to minimize data motion.

certainly people can build bad things. there is however a very real likelihood that a programming model which includes distribution will turn out to be effective

(speaker here) Photon is not an object mapper, we do not have an N+1 problem and in fact that is the exact topic of this talk.

Photon is designed to scale in complexity without loss of performance or ability to reason. The computational structure that achieves this is single directional dataflow with pure functions; the abstraction is “referentially transparent” which means it composes mathematically the way pure functions compose.

Strong composition as a basis for abstraction scalability (in the domain of UI) is the core innovation here.

Isn't this what React Server Components [1] do? Where parts of the tree are executed on client or server or both. And the data is "streamed" to client. Their "DAG" is essentially the react tree. So sounds very similar.

1. https://www.youtube.com/watch?v=TQQPAU21ZUw

So for their "ideal example" in the beginning, isn't it basically PHP?

You can freely mix the HTML with PHP code and even query the database directly using the user's input. (We know how well that turned out...)

OK so they actually mention this towards the end of the video. I guess this is more about how crazy Javascript frameworks and frontend in general has gotten, so it's time to step back and trim down some of the fat.

Looks like an interesting project. One thing I worry about is debugging. I once had to help out with some weird bugs in a relatively simple RShiny application written using the reactive model, and it was a total nightmare.

Looks really interesting. Is there a way to play around with it? Programming in that way feels a bit mind bending but would be interesting to try out!

Looks like this might be it https://github.com/hyperfiddle/hyperfiddle

They should port it to the de facto standard benchmark. https://krausest.github.io/js-framework-benchmark/2022/table... Until then I can't trust performance expectations.

Maybe also the 7 GUIs https://eugenkiss.github.io/7guis/tasks

THANK U! I have always looked for a set of GUI rendering tasks benchmarked across different GUI libraries (chromium, QT, GTK, flutter, etc) and see the effect on performance and energy consumption metrics. edit: wait they seem to only measure code quality and not publish rendering/FPS performance ? such a missed opportunity...

(Speaker here) Our approach is to issue point writes to the DOM like Solid.js (no virtual DOM), so we hope our numbers are in the ballpark of that. You're right, we need to start measuring it, it's been on the backlog. hopefully soon. We do have 7GUIs btw

Ryansolid, the author of solidJS has made a small library dedicated to enabling optimally efficient dom writes: https://github.com/ryansolid/dom-expressions It's the core behind Solid.js and he has brillantly ported it to Vue.js making this vue fork on par with solid https://github.com/ryansolid/vuerx-jsx so you would probably benefit from using and contributing to it. Also, where is the github repo of ur project so that I can follow it?

Thank you, had not seen that! Github is still private, we're entering technical alpha right now. Best place to follow is https://twitter.com/dustingetz or the email capture form at https://www.hyperfiddle.net/ for major announcements

I'm perfectly fine with side projects but it seems you are making it a lucrative company? What's ur business model then ?

This reminds me of:

Bonsai, a UI state machine composition library - https://opensource.janestreet.com/bonsai/

DatalogUI, Build UI declaratively with Datalog. - https://github.com/datalogui/datalog

HYTRADBOI (the conference the OP was presented at) also had a talk on DatalogUI! https://www.hytradboi.com/2022/datalogui-rubbing-datalog-on-...

There's also this in the Clojure ecosystem: https://github.com/oakes/odoyle-rules

I misread this as “UIs are screaming DAGs”. So disappointed!

So… React, but in Clojure?

Did you watch the video?

I’ll be honest, I just flipped through the video without sound. It’s clear that I misunderstood the content of the talk.

No idea how long I'd have to watch the video for to have this question answered, so I'll ask here:

What is DAG in this context? Directed Acyclic Graph? Please explain your acronyms.

Directed acyclic graph, yes.

For the record, you'd have to wait 1m38s if watching from the start.

Which is after him first using the abbreviation.

Reply on news.ycombinator.com