PyFlow – Visual scripting framework for Python – NodeRED alternative? | Hacker News Reader

PyFlow – Visual scripting framework for Python – NodeRED alternative?

github.com

193 pointspunnerud53 comments

It looks nice. If UI performance is a concern, one could move the GUI part to C++ and do some quick pybind11 Python bindings for better performance.

I've used this C++ ImGui node editor with similar features for some hacky synthesizer gui using this library: https://github.com/thedmd/imgui-node-editor

Ive used the same ui lib for the same purpose! :D

https://connoise.lon.dev/

DearPyGui is built with C++ ImGui and includes a node editor [1]. It’s actively maintained and I’ve found it a joy to work with.

[1] https://github.com/hoffstadt/DearPyGui#node-editor

Sounds really cool! I'd love to check out what you made, if you are interested in sharing?

Nothing special, it is a synth interface for the Korg Wavestate. I reverse engineered the synth protocol using Wireshark, it uses Protobuf data over some zmq. The Korg Wavestate has a Raspberry Pi compute module, so I blanked the root password as well with a compute module SDK. See the visual gui here: https://youtu.be/LH4qDhW8TAQ?t=55

I love visual languages. I previously made something similar that imports JavaScript functions and supports importing TypeScript declaration files: https://devev.com/

I'm currently working at Rec Room on an in-game visual-scripting language already used by millions. Here is an example from a creator on YouTube: https://www.youtube.com/watch?v=L4yvvoWdpWA

If anyone here is interested in working on tech like this as their day job we are hiring: https://recroom.com/careers#openings

If you are interested, but you don't see a role that fits your desires, send me an email at tyler@recroom.com.

I tried it in 2020 and it was basically in an unusable state for even the most basic work. It seems nothing has been committed since then and the issues remain open.

For something like this to be successful, I think it has to have some kind of domain focus. I've been working in bioinformatics-adjacent jobs for the last couple decades and I've seen things like this get used when there were a lot of integrated, domain-specific tools. Even then, the users were a curious slice of the population that could think like a programmer, but had not bothered to learn a language.

It also has to have a lot of buy-in and support. As everyone here is fully aware, a lot of problems get solved by Googling for a Stack Overflow answer. That's tough to do with a niche-y tool.

Not saying this isn't a good thing; it's just that it'll be difficult to be successful as a general purpose tool.

I work in the software side of the same field and visual flow programming tools like this help bioinformatics and scientists visualise analytics code bases, but they're not sold well to customers. Somehow the teams that sell these products still think in terms of low code or data science, when they probably should be speaking in terms of translational medicine

Talend is sold sort of well

(ex. https://help.talend.com/r/mjoDghHoMPI0yuyZ83a13Q/YYVSsDiyJ3v...)

Talend is, in my experience, flaky and otherwise quite poor at what it aims to achieve.

Have a look at FME by SAFE software if you want to see what a better, more well looked after version of what Talend could look like is.

Very interesting.

I recently learned about OpenAPI generator [1], which uses a spec-first approach to generate boilerplate for several frameworks, including Flask and FastAPI.

I would like to build or find a no code / visual tool that lets you build a JSON Schema file, and then generates boilerplate for you. I want to move on from rigid cookiecutter tools that make you go through 15 or so steps and force you to start over if you make the tiniest mistake.

[1] https://github.com/OpenAPITools/openapi-generator

I tried a pyflow a couple years ago. It was very confusing and not intuitive at all, which is what I think one of the biggest advantages of these types of GUIs is supposed to be. Doing simple things like adding inputs and operations to the graph were very difficult.

I ended up developing my own tool based on the flowchart editor in pyqtgraph. The GUI for that was quite intuitive and the code to modify it was also far simpler. That said there are a few features from pyflow that I wish the pyqtgraph editor had, like subgraphs.

Is it just me or are these things harder to write, read, understand and modify than plain old code?

Like comparing Word to Latex. Latex is faster when you know it.

Scientific literature seems to disagree with this statement: "An Efficiency Comparison of Document Preparation Systems Used in Academic Research and Development" [1]

For example, amount of written text was higher among Word novices than among LaTeX experts, with overall less mistakes. The only category where LaTeX users produced more content than Word users was equation text, and even in that case, authors suggest that the difference in productivity between Word experts and LaTeX experts does not differ significantly.

Disclaimer: I like LaTeX a lot.

[1] https://journals.plos.org/plosone/article?id=10.1371/journal...

Granted it was 3 decades ago, but when I was in grad school, choosing to use LaTeX seemed to add a few months of effort to the "writing" stage of a thesis. However, it was hard to assign root cause because LaTeX and Word were producing different things. The best technology allowed you to create more, but at a price. Typeset dissertations were beautiful, mine was not.

What LaTeX allowed you to do, in principle, was to "build" your document, i.e., type a single command and have camera ready output roll out of the laser printer. Using Word (or even more primitive tools -- I was on a MS-DOS machine), always required some manual intervention. My workflow involved scissors, glue, scotch tape, and a copy machine. But I got done very quickly once I was ready to start writing.

LaTeX has the massive (!!1) advantage of being trivial to put into version control.

I wonder how they incorporated that into their productivity metrics

As an aside, I have used a workflow where I give someone a a paper to review as an overleaf project, they make all their changes, and then I merge it back into my work picking and choosing the bits I want. And this all works fairly nicely since overleaf has support for git.

There are a lot of problems with that article. See https://news.ycombinator.com/item?id=8797002 for a discussion.

I don't think that is the primary benefit of latex.

Latex is exceedingly expressive and lets you be very precise about exactly what you want on the page and how it is typeset - much more so than Word is capable of. This comes at the cost of learning the language, which isnt user friendly - e.g. I've never had a word document that didnt compile.

Turing complete code, by contrast, demands a high level of language expressivity to do everything beyond the most basic tasks.

Using a GUI to write code becomes like using Word to professionally typeset a an academic textbook very, very quickly.

So quickly I'd argue that there's almost no point even starting.

Interesting, came across https://ryven.org/ recently which looks in the same domain.

Any practical use cases of either being used?

Huge fan of NodeRED, always thought the paradigm could be leveraged for other heavy workflow applications

A node based frontend can provide non-programmers with powerful configuration/scripting capabilities. For example customizing the routing of a synthesizer. Or teach kids coding (starting with concepts, instead of coding language syntax), such as MIT Scratch.

A lot of users is using NodeRED for home automation within HomeAssistant.

HomeAssistant is build in Python, so this could be more included part of HomeAssistant.

This is awesome, but one of the reasons I have started to rely on Node-RED is the web interface and the ability to run it as a service.

This would only replace Node-RED for some use cases if you could export the result as a standalone CLI app…

Scripts from another dimension, now in Python.

https://scriptsofanotherdimension.tumblr.com/

These kinds of visual programming systems always seem to degrade into spaghetti hell, don't they?

It depends. My nodeRed flow for grabbing various sensor data and sending them to influxdb: https://pictshare.net/mvarpf.png

And my home alarm system flow: https://pictshare.net/82qfpa.png

To be fair though you can create sub-flows to have it more compact but when I designed it, I didn't know about theseyet

I use node-red for the same kind of things, with subflows. At some point it makes a lot more sense to write a function node (javascript) than to use node-red as intended. Avoids a lot of the spaghetti.

I suppose so, but perhaps trying to prevent the spaghettification has some positive benefits in terms of DRY + code structure.

No.

Shameless Plug: We are working on a similar tool: https://github.com/hetida/hetida-designer

We have no GUI programming in the broader sense, in our case nodes are simply run one after another (DAG).

Our tool is a web application and workflows can be triggered / executed via API, which allows for automatisation. Workflows can be nested and are tagged with a version tag making production runs reproducible. It is very much taylored to (simple) Data Science use cases with the goal to make the Python data science stack accessible for Business Experts (or power users), maybe collaborating with Data Scientists.

It is actively developed, and our industrial customers happily use it, however it is still in early development state.

I'm doing something[0] similar with Deno, currently it configures workflows based on yaml files, but my goal is to make a gui tool to help generate yaml config files. My work is still very early stage, currently it can use any remote Deno module or typescript/javascript to execute steps and fetch and filter source data.

The Deno runtime is great for workflows because Deno is lightweight and can import modules using arbitrary URLs.

[0]: https://github.com/denoflow/denoflow

If you write it in Python and add it as HomeAssistant plugin (http://hacs.xyz) I think you will get a lot of users, and help from the community building it further

I thought I read that the NodeRED team had set up some sort of spin-off project/company, but I can't remember the name of it... any pointers?

Node-RED project creator here. The company is FlowForge and we're creating an OSS management platform for Node-RED. We just made our first release public.

https://flowforge.com.

Very cool.

We're designing an open source automations engine for Kubernetes ("Zapier/IFTTT for devops") and have thought about adding a UI for automating stuff visually. So far, simple YAML configurations have won out and we're still debating the benefits of a UI.

Does anyone have experience using tools like this at scale or for technical domains? (E.g. devops.) Would love to hear a convincing case why a UI is better than YAML. You can see the configuration we currently offer at http://robusta.dev/

I think the key here, as a responsible developer, is ensuring isomorphism between the GUI representation and the generated code. To the extent that the development method could be swapped without repercussion.

Most GUI tools, presumably in speed to market, skip this. But as soon as you've created something that can only be viewed & modified in a visual editor... you've become part of the problem.

So I'd approach this from a "If my tool is successful, and users use all the editors we give them, what will their support story for a 10 year old codebase look like?" (If you care about the greater good)

I like this. SaaS platforms typically don't want to do this precisely because it limits vendor lock-in, but I think it's the right approach.

> But as soon as you've created something that can only be viewed & modified in a visual editor... you've become part of the problem.

Why is that? You don't view the bytes your text editor generates do you?

ASCII and Unicode are standards. Presumably OP isn't building a tool with enough market share it's going to have an international standard encoding.

What you end up with, if you don't have a standard, and you don't maintain isomorphism... is user "code" that requires a vendor tool to view & modify.

Which sets up some really screwy incentives & business models.

A thing you built. That you paid for the tools to build. That you have to continue paying for, otherwise you'll be unable to guarantee maintainability.

Charge for execution if you want! But IMHO user developed code, in whatever format, deserves to be owned by users. Even if they decide to no longer be customers.

> ASCII and Unicode are standards.

Right, and those didn't just materialize out of thin air. They took time and effort to develop. My point is that we shouldn't keep shooting down visual based tools and languages on the notion that there aren't standards, because there isn't anything (besides preconceived biases) that prevent them. There are many things that text is just not good at describing, such as dataflow, and by claiming everything must be text and isomorphic to text, we're holding things back.

I'd be more sympathetic to this idea if I hadn't seen the tactic executed repeatedly over the last 10 years to ensure vendor lock-in, which ultimately burns the end user. And/or requires them to insecurely freeze a portion of their IT infrastructure in time.

And honestly, if your GUI/designer is standardized, then it's trivial to define a stable serialization to and from text anyway.

You only need to sympathetic because you have let yourself become jaded by your anecdotal experience.

It didn't feel anecdotal when I was the one trying to modernize systems like these in the face of vendor apathy (or active hostility) and business resistance to change, because they had 10+ years of legacy "code" built in these things.

So every other software project for every business ever? Got it, still anecdotal data

Ah yes? and diff, track, and version the bytes with tools like git. Or perhaps I’m misunderstanding the question here?

YAML parsers don’t read the bytes or the GUI. They read the text.

Look to HomeAssistant, where there is a lot of both YAML and NodeRED. I am hoping there could be a Python alternative to NodeRED build into HomeAssistant, to make it easier for beginners.

And Home Assistant itself is python so it may be a better fit than the often used NodeRed which is JS based.

Cool, thanks

We ended up using Puppet Relay quite a bit, which is sort of a halfway house between the two ( visual editor, but yaml backed with low code instead of no code).

The main advantage from my side was being able to get adoption from a wider group of people. This included both very junior technical folk who were able to take on more complex tasks than they would have been able to otherwise, and non technical folk who found starting with the UI a lot less intimidating.

Eventually, almost everyone has gravitated to using YAML over time but I don't think I'd of seen the adoption internally that we had without the UI component.

Mileage will vary though. If we had been at a more technically mature place when adopting relay, then I reckon the UI would have seen a lot less use.

Heyo! I used to do a bunch of work for a major SOAR player. I think the biggest benefit to tools like this is that it greatly reduces complexity and barrier to entry into automating various tasks allowing the SME’s that may not know how to program, to automate away their pain points with less risk of things being lost in transition while communicating with developers who may not be.

I know yaml isn’t difficult per se, but being able to visualize it can be beneficial for those that don’t normally think programmatically.

Also bonus points - the diagrams from the flows make great process diagrams for your documentation.

Having worked on a flow based UI for a few years, I would say it very much depends on your core use case and the “standard library”. Since you’ll probably be making a DSL of sorts, there is a balance of simplicity and flexibility.

It really shines in the ETL area. Configuration and generation would be alright.

You can get an idea of where I got with js at [fluxion.app](https://fluxion.app).

Reply on news.ycombinator.com