Hacker News Reader

Hey, wait a second. Are these just a modern incarnation of the enterprise service bus? Is there a significant difference?

Yes. The only difference from what I can tell is that each service/node on the bus is a VM/Container rather than a bespoke built machine.

dragonwriter below you says the opposite - based on a plane-level analysis. would be interested if that changes your mind.

The real answer is it depends on what you think what responsibilities exactly ESB or Workflow Orchestration should have. dragonwriter is of the reasonable opinion that ESBs focus should be a message plane. If your of my opinion, ESBs generally include some of the features found in workflow. devonkim is also right in that ESBs in general let the business data/process bleed into the rest of the stack, where as with workflow, its completely agnostic to the business data/process.

The most accurate answer would be from comparing various ESB products to Workflow Orchestration products, as every vendor/product will have slightly different opinions on where their responsibilities are.

fair enough. appreciate the thoughts!

By modern you mean using "docker", have an "io domain", and use node-js/go/rust? That seems to be the pattern these days. Take existing items, rewrite/repackage and io-ify it, maybe make it a SAAS product while at it.

> Are these just a modern incarnation of the enterprise service bus?

No, ESB is the messaging plane, workflow orchestration is a higher level service, on top of the messaging plane.

ESBs historically required a domain-aware object model (when I saw them they were routed based upon rich types and terms like CORBA and RMI were still in vogue). This is more obvious when you look down the page for features in Apache Camel and they mention different data type support. Well, a workflow or orchestration engine doesn't need _any_ idea of your business layer data model to make decisions on where they need to be broadcast, but it's a convenience similar to listening to specific topics and segments in Kafka after querying a data broker service. Decoupling routing and representation of objects means that the bus is now dumb but the scheduler / router is now interchangeable. Which is how we can see so many possible competitors to Conductor and EBPM and all that now as shown in these comments.

Not sure if this relates but I was wondering if you heard of ASCI ActiveBatch? If so what are your thoughts on it?

I'd like to have some easy to set up orchestration/job scheduling engine in my team so we can help clean up the tangled mess that we are in but something like Ansible seems like too much work to setup and add more jobs over time. I tried Activebatch and I wish there was some free or cheap alternative of it.

I hadn't heard of it before. Thanks for sharing!

What are the types of problems you're hoping to untangle, more specifically? I assume you want something like... A workflow where each step maps to some provisioning process?

If that guess is reasonably true, then I immediately think of Terraform. You can specify dependencies and write hooks that trigger callbacks when certain steps are called. We use Terraform a bunch under-the-hood for Refinery, and it's great. I haven't used Ansible (only read about it) so I can't contrast them very well.

If you want to chat some more, I'd be curious to hear more about what you're trying to build and see if there is a way we could collaborate. Or if I can offer any recommendations for software that I know of. My email is free at refinery.io :)

Thanks for the resources! It seems like a lot of these implementations expect to run as independently managed services in some microservice architecture. Are you aware of any workflow engines implemented as a library inside your application, maybe with storage backed by an external database? I think that you could still have a highly available model, provided the database supported that.

Funny, even though Conductor isn’t designed to be- this is exactly how we use it. In a Kotlin and Spring Boot codebase nonetheless!

Like celery?

Prefect. It's a workflow engine in a python library.

https://github.com/ing-bank/baker is one such library for JVM languages. The state is kept in Cassandra or in-memory. We've been running production workloads with it for the last 2.5 years. A feature comparison with alternatives can be found here https://ing-bank.github.io/baker/sections/feature-comparison....

The idea of a pseudo flow based programming worfklow has always appealed to me because I think having logic as components that can be used as legos makes sense. It seemed like the other stuff out there was either too overkill/obtuse or even too low level for what I had in mind. So I started writing something for personal use in JS that fits my mental model of what that would be.

In a nutshell it's just a runner and uses a project called moleculer which is a microservice framework that I'm more or less just using as an rpc client to execute the tasks of the workflow.

One thing I've been debating with myself is how I should work with the dataflow. Would you say it's better to have the results of every node merged into a singular context across an entire flow that is passed to every other node/block (ie. always one input, the context, one output the context), or would it be better to explicitly declare and pass in inputs/outputs.

Here's one project that does it that way: https://github.com/danielduarte/flowed

One benefit of this seems to be that nodes can run in parallel the moment their dependencies are met without having to care about each other.

I poked around on your site and I like what you have to offer. You appear to do it the singular context way which I suppose makes sense because it seems like every block is an individual lambda. This also seems like overkill for my purposes because I'm interested in being able to do granular things if I wish such as a simpe rule action. I'm not sure having an individual lambda to run "if email exists return true" would be practical. That and the warm up time/latency.

The other thing is managing something like npm dependencies might be annoying across blocks.

Being able to create arbitrary API endpoints is nice though.

I wish there were something a little more in between than all of these enterprisey offerings.

My company is slowly switching from Tekton to Argo (much to my chagrin). They seem very similar in that they’re incredibly alpha and fighting over similar parts of the stack.

Argo seems more CD focused right now whereas Tekton really is a “toolbox”/make what you want.

That's funny because the Tekton website[0] has this as their first text:

"Tekton is a powerful and flexible open-source framework for creating CI/CD systems, allowing developers to build, test, and deploy across cloud providers and on-premise systems."

Do you know why they are switching, out of curiosity?

The market is definitely very young and I don't think anybody has really nailed down every use case.

There are the parts of the market targeting "business model" use cases like Camunda/Zeebe.

Then there are ETL-style systems like Airflow for dealing with massive data processing.

And still you've got the CI/CD side of things like Argo/Tekton for automating complex build systems/running tests.

Then for systems like Netflix Conductor, Uber Cadence, and AWS Step Functions (among others), you have systems trying to abstract on top of existing complex systems (microservices, etc).

That's not even including low-code spaces like Zapier or IFTTT that try to target making integrations trivial.

It's a crazy world!

0: https://tekton.dev/

I had Tekton in my mental filing cabinet under "Cloud Native CI/CD", how's the UI / DAG-visualization for that toolchain? Like the sibling comment, I'd also be interested in hearing your A/B thoughts on Argo vs. Tekton for the generic workflow management usecase.

Tekton's UI is very bare bones. Really all you need to see job status and logs from each job/step running. It's not _good_, it's fine to get the job done. It's not good at surfacing errors during execution though. We spend a lot of time diagnosing why pipelines failed.

At this point in my experience for both (I have much, much more time with Tekton though), Argo is good for CD. It's visualization and syncing of environment charts (we use Helm) is useful.

Tekton is amazing for defining highly complex build/deploy workflows (which we have). Along with Tekton's sibling project Triggers, you can have full end-to-end PR -> Build -> Deploy -> Feedback into Slack/Monitoring/Alerting/etc.

If I had to make a comparison (with my current knowledge level), I'd say Argo is like Angular, batteries loaded with addons you can include if you wish. Tekton is like React, you get pieces and build them into whatever you want.

interesting project you started. I had used snaplogic before and its similar, although not built specifically for AWS serverless. Do you have any videos of any walkthroughs, demos, or tutorials?

EDIT: nevermind, i see your getting started docs have lots of video clips.

I hadn't heard of Snaplogic before. Had to dig in. Thanks for sharing -- this is helpful context to have. There are so many companies across different spaces that it's hard to find them all!

Glad you found the docs here[0] (for anybody else who is curious).

We're still iterating on Refinery a lot and trying to find product market fit. If it doesn't make sense or you're confused, that feedback is super helpful for me. The goal is to build something people actually want to use and it's an iterative process to get there!

0: https://docs.refinery.io/getting-started/

Is there a serverless workflow solution for google cloud yet?

What kind of support is present for testing workflows?

It's fairly limited right now to testing single blocks in the editor.

We recently added support for Git-based development to address testing. The idea there is that each block is just a chunk of code living in a Git repo, so you can use whatever testing tools you'd like to.

It's actually pretty slick -- it's bidirectional. You can use Git to make a commit, like normal, and then refresh the editor to see it. And you can make also changes in the editor and it will commit + push to Git. If you want to play with that, I can share details on how to enable it (it's fairly hidden in the UI atm).

In terms of testing workflows together as a chain, that's still TBD. It is something that we'd like to ship eventually though. Along with staging environments and canary deployments.

On our immediate roadmap is to get the core of the tool open-sourced so that people can start playing with it outside of our hosted platform. We have it gated behind a credit-card form right now to fight fraud and because of some technical limitations.

Temporal (Cadence) comes with Unit testing framework. The most interesting feature of it is ability to skip time automatically when workflow is blocked waiting for something. It allows testing long running workflows in milliseconds without any need to change timeouts.