Hacker News Reader

I am playing around with building my own similar and am faced with the question you pose.

How can you tell if your prompt process works? I feel like the outputs from SDLC process are so much more high level than could be done with evals, but I am no eval expert.

How would you benchmark this?

View on HN

For sure the proposed approach is more token-consuming than just ask high level the final outcome of the project and make an AI agent to decide everything and deliver the code. This can be acceptable for small personal projects, but if you want to deliver production ready code, you need to be able to control all the intermediate decisions, or at least you want to save and store them. They are needed because otherwise any high level change that you will require will not be able to make focused and coherent enough code changes, with previous forgotten decision that are modified and the code change that will produce lots of side-effects.

Reply on news.ycombinator.com