This is a very interesting study, but the generalization implied in the abstract -- that specialized models may not have advantages in specialized domains -- is an overreach for two reasons.

They introduce medprompt, which combines chain of thought reasoning, supervision, and retrieval augmented generation to get better results on novel questions. This is a cool way to leverage supervision capacity outside of training/fine tuning! But they compare this strategy, applied to GPT-4, with old prompting strategies applied to MedPalm. This is apples and oranges -- you could easily have taken the same supervision strategy (possibly adapted for smaller attention window etc) to get a closer comparison.

Second, MedPalm is a fine tuned 500b parameter model. GPT-4 is estimated to be 3-4x size. So the comparison confounds emerging capabilities at scale, in addition to prompting impact, with anything you can say generally about the value of model specialization.