AIConfig – source control format for gen AI prompts, models and settings | Hacker News Reader

AIConfig – source control format for gen AI prompts, models and settings

github.com

97 pointssaqadri16 comments

Hi HN! OP here - this is our first open source project and we are really looking for your feedback about this direction, because we haven't seen something like aiconfig in generative AI developer tools yet.

Our basic premise is that AI application development should be config-based, so you can track the prompts, models and model parameters being used more rigorously. Having this AI artifact then lets you iterate on it separately from your application code, and also set up evals that provide "test coverage" for the gen AI parts of your application.

We were also inspired by the ipynb format for Jupyter notebooks, and you'll see parallels to that in the aiconfig format.

Please ask any questions, and share your thoughts on config vs. code.

I wish there was some more documentation on dynamic parameters - how they interact with the Workbook editor?

A related question - I want to have chains of function calls - so that the prompts should contain all previous function calls and function call results. The chains can have variable length and end when the LLM calls a 'finish' function. How can I do that with AIConfig?

I published https://www.prefab.cloud/blog/dynamic-config-for-openai-pyth... a few weeks ago as a way to do config of AI params like temperature, etc. But I've had a few people start using it for prompts as well and I really curious to see where it goes.

In particular for 1. teams that have complex slow deploys, but want to change prompt now 2. when there are data analyst types doing the prompts and people don't want them to be able to "break things". 3. being able to alpha test / rollout / target new prompts easily.

Definitely an interesting question whether prompts is code or configuration.

Well done. Now that I've seen your getting started video (https://aiconfig.lastmileai.dev/docs/getting-started) this seems obvious. I feel really silly that I copy & paste from the browser back into a file just so I can save different prompts and results to see what I liked best. Thanks for sharing and for open sourcing!

Thanks! Would love your feedback both on the UX editor as well as the APIs for interacting with aiconfig. We are also working on a local editor so you can have a locally hosted "AI workbook" playground

I love this! Been working on something similar that uses containers and Jinja2 templates: https://github.com/kordless/SlothAI. A demo is running at https://mitta.ai. No docs yet and the auth system is getting an upgrade.

Nice! Let me know if you try out aiconfig. We would love contributions

Why not store them in code? Easier to read and source controlled.

We have a bit of context about this in the readme: https://github.com/lastmile-ai/aiconfig#what-problem-it-solv.... The main issue with keeping it in code is that it tangles application code with prompts and model-specific logic.

That makes it hard to evaluate the genAI parts of the application, and also iterating on the prompts is not as straightforward as opening up a playground.

Having the config be the source of truth let's you connect it to your application code (and still source controlled), lets you evaluate the config as the AI artifact, and also lets you open the config in a playground to edit and iterate.

For example, compare how much simpler openai function calling becomes with storing the stuff as a config: https://github.com/lastmile-ai/aiconfig/blob/main/cookbooks/... vs using vanilla openai directly (https://github.com/openai/openai-node/blob/v4/examples/funct...)

Yeah, looking at this just briefly it might be the wheel I've (pre)-invented like 1/3 of for one of my couch projects. Definitely can see the appeal of a conventional format here, I'll check it out in more detail when time permits!

Please definitely let me know when you get a chance to try it out! The readme has a link to our discord too if you want to get in touch (and you can email me directly too)

I mean don't get your hopes up too far, it's gonna be a minute :D But I'll pass along what I find when I can!

Thank you. I think most things we store in the config are non-business logic. Like the connection string of database or some feature flags.

However, the prompt is your business logic in most cases and put your business logic into a separate file make it harder to read and harder to maintain.

Relatedly similar project:

https://promptfile.org/

How does this compare to Ollama

I assume you're talking about the Modelfile in ollama. AFAIK it doesn't work with other models. It may also be a bit more low-level, since it has information about the LoRA adapter. That said, there are some things that are overlapping, and we'll reach out to the ollama team to see if an aiconfig integration makes sense.

Reply on news.ycombinator.com