Backup If OpenAI Fails

3 points·roxyrox··3 comments

Hi ! We are implementing a bot using chatgpt for my company but we have some concerns about downtime failures. Do you have some backup solutions that could be implemented in prod in case of failures ? Knowing that the model is kind of finetuned because we are doing information retriavial using embeddings (created with gpt 3.5)

View on HN

We use VertexAI from Google Cloud instead of OpenAI. We switched because of downtime, both the hard and soft kinds. You'll have to change your prompt, like how it took you time how to use ChatGPT effectively, you'll have to do the same for any other LLM.

Mistral just early accessed their API too: https://news.ycombinator.com/item?id=38598568

argh so you had to retest the entire prompting / results ? did you do it exhaustively?

You don't have to redo the whole thing, but there are nuances, just like switching between any framework. I was actually able to use a much shorter prompt with VertexAI than OpenAI.

One of the big problems we will have with these large, hosted models is version changes. They don't support them for very long because they are expensive to run and they only have so much capacity. You typically have to redo prompts with each new version. This is why I think self-hosted will eventually win. The upstream vendor will break your pipelines and there's not much you can do about it. With models you control, you decide when they update and can test / update thoroughly before you do.

Reply on news.ycombinator.com