This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great.
So, how do you avoid having to spend hours on figuring out if the model is just dumb, or don't know the task? Your own private benchmarks! Figure out a way, ideally without using another LLM, to score how good a model is at doing your specific task. Come up with 3-5 examples for this benchmark yourself, ask a SOTA LLM to fill out 45 more, review everything VERY closely, then use this whenever you want to figure out if $new_model actually is an improvement over what you use today, and once you have a bunch of different tasks, you'll see that all these HUGE improvements tend to be specifically for the benchmarks they mention in the press release, as many of your own benchmarks won't show that big of a difference in reality.
Yes there is a higher upfront cost, but if you're building longer-term projects that rely on LLM models, particularly local ones that seem very benchmaxxed a lot of the times, you need a quick and reproducible way of scoring them somehow, where you can just add more models to compare, and you need to keep these benchmarks to yourself.