I'm making the stronger claim that fine-tuned BERT is almost always better than ICL GPT4. That is, you beat ICL GPT4 with a model of less than 1b parameters. These are the results in the literature.

Fine-tuning GPT3 is actually a strange middle-ground: you're constrained to solve the tasks via prompting, rather than attaching output layers and doing gradient descent over a loss function that directly matches the task. So fine-tuning LLMs for these classification tasks is indeed often worse than just using the larger model.

The other thing is, as you scale up the model the ratio of output weights to the whole model weights gets quite out of balance. This introduces some technical challenges, and so it's kind of tough to make it all work with medium size LLMs of around 7b parameters or so. In contrast the process of fine-tuning BERT-sized stuff is really well explored, and it's easy to find more or less push-button solutions.