Hacker News Reader

Comment by empath75 | Hacker News Reader

I think a lot of these libraries are made by people who don't really design systems.

At its core, what chatgpt and the like really are, are ways to transform natural language into structured data and vice versa. All you really need is an interface that allows you to plug in these APIs to _normal_ workflow engines that lots of companies are using every day to do work, both fully automated workflow engines like airflow or whatever, and also just plugging them into the _exact same systems_ that human engineers use -- it's perfectly possible to get chatgpt to use Jira and submit PRs or whatever.

View on HN

> are ways to transform natural language into structured data

I've seen quite the opposite, because at their core they are next word predictors. Telling it to structure an output biases the context into data that it saw in a structured format. From what I've seen, this always leads to lower quality reasoning/output.

I get much better results if I leave the data unstructured, to maximize reasoning, then add a later step to structure that output.

You mean like an intermediate "thoughts" or "observations" output step?

My use case has always been:

1. take some input

2. run it through the llm to get some output

3. make that output useful to my non-english-speaking software by structuring it.

Forcing step three to happen in step 2 means you're in a very unusual/unrealistic "head space"/context for next word completion.

Leaving structuring as a later step means the human like output/reasoning can be maintained for step 2, with the inhuman like structuring being more of a reformatting than reasoning+reformatting.

Leaving the "context" as small and precise as possible (which might me large for a given task) gets the best output, from what I've seen. Being a next word predictor, this makes sense to me.

I suppose this is just "chain of thought".

This is pretty easy to benchmark. Have it put the answer into json format, after producing the answer. Then, have it come up with an answer in json format, as one step. It's a night and day difference, at least for my tasks.

Agreed that it feels like a cowboy show. We get 2 main uses out:

1. Inverting the control structure. Traditional orchestration frameworks blackbox simple calls, like asking an LLM to analyze a patch. Chaining is about an LLM-based planning agent + ensuring all tools etc are abstracted so the planner can orchestrate them. Airflow era tools leave too many integration gaps here.

2. Plugin interfaces. Related to 1, it puts in patterns for keeping vendors easy to swap out. (In theory, practice takes more bc tuning, but at least the shape is right.) Orchestration frameworks and new LLM ones have a lot plugin ground to cover here.

We had to do a lot of work around langchain to make prod ready for louie.ai, and just as we have been figuring out patterns we need, was clear so was langchain. Don't get me started on threading and eval..

How?

Is OData still a thing?

This is going to sound like a smart-ass answer, but "programming". Jira has an api, openai has an api.

You need some script that takes structured json and uses it to modify a jira ticket, you need some other script that asks openai to generate that json based on some prompt and context, and that's all there is to it. I wrote a script right after I got access to the API that could open and refine jira tickets based on conversations in a chat window.

Reply on news.ycombinator.com