It’s definitely not. Our prompts that were generating JSON output went from around 95% valid JSON to about 10% overnight. The model just started inserting random commentary. We’ve reverted to the 0314 model and it’s working fine again.
I'm certain it's now doing significantly worse on the same tests, but alas I have lost the historical data to prove it.
I ask, because I think it’s going to be a big challenge, so I built a service to record feedback / acceptance data: https://modelgymai.com/
If you think it can help, I’d love if you’d try it out and let me know if it helps.
a. Worse at general code-like tasks without using functions
b. Equivalent or better at code-like tasks if you use the function API
c. Much faster than the older model either way.
I'd guess it's cheaper to run, too, and that they use the presence of a function in the API signature to weight their mixture of experts differently (and cull some experts?). The degradation in general purpose coding tasks is pretty obvious and repeatable (try the same prompts in the Playground with the -0314 model vs the -0613!), but it does seem like you can regain that lost capability with the new function call API, and it's faster. The tradeoff is that you only regain the capability when it calls functions; you can't really have a mix of prose-and-code in the same response as easily, or at least not with the same quality.
1: https://twitter.com/reissbaker/status/1671361372092010497