AI proxy middlewares are a hack

blog.lastmileai.dev

I'm glad to see more focus on middlewares. Getting the architecture right is crucial to building effective models, but this has largely been overlooked as shown by the inefficiencies in many models today

Agreed. The author makes the point that middleware is trying to solve valid problems. I think the issue is that there are better ways to address it by “avoiding the middleware all together”. Especially since it introduces other 3rd parties into the mix

Yea, half of this seems like problems we've already solved for APIs generally, the other half is LLM specific, like prompt management, log/mon, response quality in prod, real time feedback

It seems like tools from our current ops ought to work just fine for the non-LLM uniqueness. At the same time, Datadog is pretty popular for a managed experience, and the LLM services as proxies kind of fit that model

This project looks great and aligns with my thinking

Reply on news.ycombinator.com