Yep, that's definitely this challenge with commonly available models. In a real-life product development there's most often an important step of evaluating the model(s) and fine-tuning if necessary.