I tested a bunch of models while building https://double.bot but ended up back on gpt4. Other models are fun to play with but it gets frustrating even if they miss 1/100 questions that gpt4 gets. I find that right now I get more value implementing features around the model that fixes all the GitHub copilot papercuts (autocomplete that closes brackets properly, auto import upon accepting suggestions, disable suggestions when writing comments to be less distracting, midline completions, etc etc)

Hopefully os models can catch-up to gpt4 in the next six months when we fixed all the low hanging fruit outside of the model itself