Hacker News Reader

Comment by t-writescode | Hacker News Reader

> I am having trouble understanding what the complaint is here.

The appropriate level of due diligence for each LLM model transition is to run your various prompts into the new model and make sure they still produce the correct output; and, if they don't produce good output, to update the prompts so that they continue to produce good output.

Just yesterday, I was experimenting with 4o and assumed I could do a flat migration for some work. 4o actually provided worse results - results I explicitly asked to *not* have in my 4 output (and that I didn't have in my 4 output).

It's tedious to have to change models after you've already done a proper validation suite against one model.

That would be (at least my) complaint.

I've even version-stamped the models I use on purpose to avoid surprises.

View on HN

Not directly addressing your point but asking an LLM to not include something in its output often doesn't work well. Its a bit like saying to someone "whatever you do, don't think about elephants."

4o is pretty bad in my experience. They shouldn't have used the "4.0" naming for it or said its Lite or something. Instead they are trying to market it as roughly equivalent which it definitely is not.

At that point why not use your own hosted open source model that is more reproducible for you?

Cost of maintaining architecture, cost of complexity of internal infrastructure, knowledge level required for self-hosting, complexity of local one-boxing, and a slew of other reasons.

Everything's a tradeoff, but it seems that part of the tradeoffs include access to tools critical for your product to function correctly being taken away without a way to get it back. Maybe that can be an acceptable tradeoff, but I'd personally not like living with that.

At present, my startup isn't making money - in fact, it's not even released. As a result, I'm trying to prioritize getting it out of the door while still being affordable for myself enough to bootstrap it.

To do this, I've made, and continue to make tradeoffs. Among many of the tradeoffs I'm currently making that I intend to resolve ASAP is having OpenAI as a single source of failure. I intend to have some of the other hosted solutions as other options for LLM processing. One of the many, many options that will be considered at that time is self-hosting it, as well, as another option.

I've already spent more time than I should perfecting various smaller pieces, increasing reliability, etc. Each time I choose perfection, I lose more time, more runway, more potential market share; and, something I've recently had to learn:

Each time I lock myself in a previous step to get that step perfect, I miss the lessons I'm about to have to learn in the next stage of the process, including new issues I'll run into that increase the next step's complexity above my initial estimates.

Everything is a tradeoff. Choosing to use a commercially available solution with known and relatively set costs while accepting it may slowly change underfoot (while also knowing I have alternatives I can swap to if an emergency comes up that should only take a little bit to transfer to) is one I've made.

Because open source models aren't as good as GPT-4.

Reply on news.ycombinator.com