Comment by aleqs | Hacker News Reader

Comment by aleqs | Hacker News Reader

I'm working an an agentic graph-based workflow execution engine/framework. The concept of the harness is completely abstracted away/generified - a 'node/agent's is a harness (cc, codex, open code, pi, etc) + model (I test different model and harness combinations). I have a set of tasks from trivial to complex - a set workflows (a workflow is a set of initial nodes and their behaviour) is defined and each one is asked to perform each task (multiplied by each harness/model combination roughly). The workflow can include agents/nodes which are able to modify the workflow graph and create nodes. Other nodes can break down tasks and send subtasks to other nodes. Mostly experimental stage at this point. I'm exploring/tracking metrics such as total wall clock time to complete a task, total cost in tokens and $, among others. This gives me a decent amount of data/insight into the abilities/performance of different harness/agents/models for different tasks, and gives me a great testing/dogfooding of my own harness (which is one of the harnesses being tested, and as of now the most efficient one).

The main bottleneck at this point is the cost of all of the tokens in the fairly large test matrix of tasks, harnesses, models.

I hope to release/open source all of this stuff eventually.

My company tried to build something like this pre-TUI as a tool-AI-IO dag dispatcher. The biggest mistake I made was thinking that people would have no problem figuring out how they could translate their work or define multi-step automations, and focusing on the orchestration and sandboxing thinking that was the core, when it was really figuring out how to get the onboarding UX/complexity to not feel daunting or more trouble than it was worth.

Eventually for my own work, I discovered that the context management and runtime was more like a stream or active service mesh than a dispatching / one-off processing problem, most others' were too. Then all my prompts would degrade across model versions or providers, and I realized that actually setting the context for the tasks and keeping track of it all was a ton of work and something I had to do everytime as an actual user, but never when I was testing or demoing it on existing data.

Curious how you're testing your work and if you've managed to avoid the problems I ran into. I need to permute across the same set of workloads/configs you mention (and maybe more) for my next set of work so I'd be very interested in sharing or collaborating on the test infrastructure! At Google I did a lot of permutation testing using https://github.com/cloudprober/cloudprober and was going to start using it sometime in the next couple weeks. It exists basically one layer above the workload content/targets so it's probably compatible with everything except the test client/driver you're using.

I'm my case a workflow is basically an active/living graph of nodes/sub-tasks. One node can process a task (with all relevant context) and create multiple fan-out tasks, or it can add additional context/requirements and pass it along to another node. The message/task passing is all implemented as queue - nodes subscribe to messages/tasks addressed to them and execute them, producing more tasks (or zero new tasks). For each task there is a context and a parent task/context, as well as a key/value store of all tasks and their context. Each agent/node gets instructions injected into their prompts that tell them how to look up parents tasks/context as well as how to output new tasks.

There is also a feedback loop - a node can fail to process a task, and pass the reasoning/context for that back to the parent or another node - this might result in a new adjusted task replacing the failed task, or it might require human intervention.

How do you test it across different workloads and are you running it in a datacenter or cloud provider?

I forgot to mention it but the other major problem I underestimated was giving the permission to potentially spend lots of money to AI calling each other in ways I didn't have a good way to monitor, and didn't want to actively watch. So I wanted to set budgets and have them get passed to children, and realized that meant I had to build a pretty complicated billing/scheduling system with a way to keep the part of it with all the permissions and money safe from the AI doing AI stuff on its own, and set up NAT and firewalls and all this other stuff.

If every child can loop back up to its parent, and everything can run stuff from the Internet, and make expensive resource decisions, and get restarted if it fails, then it might not ever converge on being done, or get infected or just mess up and spend a lot of money. I ask about the testing matrix/driver you're using because that's where I realized there was a lot of work and cost involved in getting that part working well enough to run real workloads.

I have a 'node/container' abstraction at the infra/engine layer which is essentially either a cloud VM or a local podman container. The engine/infra layer can spin up more of these as needed. I have a relatively beefy dedicated machine for working with AI, which is where I do most of the testing.

I aggressively try to keep costs down so the workflow DSL I have supports configurable limits which can be set at the $, token, or time dimension , at task, workflow and agent/node levels, with some same defaults. I have a pipeline which keeps LLM API pricing data up-to-date, and I use AI to estimate total costs before runs and manually approve those.

I rolled my own simple execution DAG program.

It’s shockingly effective due to rooting sub-DAGs into Planner nodes which are the only mutators of the DAG. The deepest topological leaf nodes become the blockers to the next Planner node.

The only other special node is a Human node; structurally impossible for agents to close (I rolled my own harness) and block on my attention.

Nice, yeah also I have planner nodes, review nodes and organizer nodes (organizers can mutate the graph/workflow, create new node types, etc.) Trying to automate the node type/role definitions and overall workflow definitions as much as possible.

I split my project into 3 layers - the engine/infra layer (handles task dispatch/queuing, spinning up node/agent containers, etc.), the 'brain' - basically a collection of different workflow models and related stuff (multiple different models for testing/exploration purposes), and the harness.

I’d be curious what performance / behavior changes you’d observe with two changes:

- planner nodes and topo deepest for rescheduling, no inflight modifications. (repair protocol = redispatch root planner for rare cases where required; ~9% of the time for me)

- no review nodes; strongly enforced on orchestrator to always do adversarial reviews post-codegen and fix out of band

I found that putting myself in the graph is critical to ongoing fidelity, even if subpar to if I had written it all myself.

> The main bottleneck at this point is the cost of all of the token

Are you using Chinese models? Quite a bit cheaper, but maybe still too expensive?

Yeah mainly deepseek, it performs near top pretty consistently in terms of price/output (with a basic quality measure). I would love to test with more models but that's not cost-realistic for me at the moment.

Reply on news.ycombinator.com