Workflow.py: A Minimalist Workflow Engine | Hacker News Reader

Workflow.py: A Minimalist Workflow Engine

github.com

70 pointsdungwiz12 comments

This is not a workflow engine but rather a rules engine. A workflow engine is like a navigating a directed graph or a flowchart. Executing one node would take you to the same or different node based on some rules. So a rules engine is typically a part of a workflow engine or invoked by it.

There are two types of workflow engines: sequential and finite state machines. The latter is more general and includes the former. In a finite state machine external event let you move through the nodes of a graph. The present node is the state. In workflow.py the creation/update of a file is the external event. The state is also characterized by the files present in the file system. The change of state is associated to the execution of a program. I agree it is a little different than what people are used to and I am not arguing semantics. It was useful to me. Hope it is useful to others. That is all. I will be happy to take patches to improve functionality.

Sorry, if I sounded argumentative. I just shared what somebody's perception could be when they read the title 'Workflow Engine'. It took me a while to understand that probably 'Rules Engine' would have been a better nomenclature.

So someone remade cron+incrond, in python, using more memory, with a more complicated syntax. Okay. I'll pass.

Yeah, but it seems like one of those tools you write because you want it to work your way, not necessarily because it doesn't already exist. I have a few of those: tools that already exist but I wrote my own for fun or because it fit my workflow better.

Also, the Windows version of cron and the like is not as developer-friendly, IMHO. I do not see why this could not work well on both systems.

Isn't this what inotify and related programs (incron, etc) are for? I expected it to at least be based on python-pyinotify or python-inotifyx.

Eventually it should be rewritten to use pyinotify. For now it was good enough for my needs.

Speaking as a dev who uses Python on Windows primarily, I like this a lot. Much more lightweight than Windows Workflow Foundation and doesn't feel like sysadmin work compared to doing stuff with Powershell + Scheduled Services + other parts of Windows automation. In other words, this feels like "developer work" rather than "Windows IT Pro" work :p

Kudos!

How is this a workflow engine rather than an interpretation of a *nix daemon?

I think this is a good implementation of a Python program, but there's - from what I've personally seen - a tremendous opportunity for an affordable, flexible workflow engine (that's not based on Microsoft SharePoint).

Does Python have a BPEL/BPM-style workflow engine like Ruby's ruote?

This doesn't seem to be it, but I would love to have a workflow engine which is designed for long running tasks, with periodic human interruption, conditional flows, and so on.

I'm not sure how they compare to Ruby's "ruote", but there are quite a few Python workflow managers and libraries out there.

GC3Pie (http://gc3pie.googlecode.com/) is a Python library for running many-task workflows featuring interruptible execution, interfaces to SGE, PBS and LSF clusters, and composition operators to build dynamic task dependencies (so, not just DAGs). Disclaimer: I'm one of the developers.

Weaver (http://bitbucket.org/pbui/weaver) is a Python front-end for building workflows that can run on top of the Makeflow engine, supporting SGE, Condor and WorkQueue as execution back-ends. (See the comment by its author "pbui" on another HN thread: http://news.ycombinator.com/item?id=4047100)

NiPyPe (http://nipy.sourceforge.net/nipype/) is a Python workflow engine especially targeted at Neuro-Imaging processing (but the core framework is generic, as far as I understand).

I'm pretty sure this list is not exhaustive: many people seem to be re-writing the same core functionality, coming from different fields and/or with slightly different requirements.

Indeed the title made the content look shitty.

Workflow engine means an engine to drive workflows - at least for me.

This thing is meh, no reason to use it over existing C tools, I wonder why someone rererererereinvented the (not actually round) wheel.

Reply on news.ycombinator.com