Well, I would say that it's a very good reason, and that learning about branch prediction and caches is not a "drastic" step by any means.
Is there any software that you write whose users would not be made happier if the software performed better? Any at all?
> Many of these platforms have undocumented CPU instructions
You don't need to know the secrets of a platform to understand branch prediction and caches, or to use that knowledge to produce software that performs far better than software written without that knowledge. You don't need to know the platform on a logic gate level, you need to understand how the platform executes your code, so that you can take advantage of the strengths of the platform. You don't need to know any hidden instructions or secrets to take advantage of the platform.
> we've had CPU-level issues like spectre and meltdown introduced that affected performance in some cases
those things didn't affect performance, the fixes for those things did. The fixes also required no code changes outside of the firmware and the operating system, and knowing the platform is still the best way to write performant software, no matter what is going on to avoid hardware vulnerabilities.