This isn't really the case. In general in any class of mathematical objects, the physically realizable ones are the most tame. Physical realizability is a huge constraint that typically dramatically simplifies problems.
> Quantum field theory is insanely complicated math wise with infinite number of infinite integrals, mathematicians has still not figured out how to formalize that.
My understanding was that the issue is that QFT as used by physicists is known to be mathematically inconsistent. For example, see Haag's theorem [0]. As Wikipedia puts it "there does not exist a well-defined interaction picture for QFT, which implies that perturbation theory of QFT, which underlies the entire Feynman diagram method, is fundamentally ill-defined."
So the problem is not that we don't know how to formalize it, it's that what physicists are doing is known to be wrong. So we need some better argument for why it sometimes seems to work under the conditions it does. And that's arguably more about getting the physics right than it is about math.
This is something that happens in general: physicists will often use tools or arguments that are known not to be correct despite the fact that they work well in practice.
I think in general, that's a good thing. Physicists need to be able to brainstorm ideas in a way that is unconstrained by needing to formally prove them. In this process they might find something that works 60% of the time. Then someone will refine it to work 80% of the time, etc. But at the end of the day, the tools do have to be mathematically consistent for them to actually describe the physics, unless we live in a mathematically inconsistent universe.
So the characterization I'd give is that physicists are generally better at modeling, but also less constrained by the need to be correct.
> Mathematicians are experts at verifying tools that exists. Those two skills has less overlap than you might think.
This part is reversed IMO. Mathematicians created nearly every major tool used by physics since the history of humanity. Occasionally physicists will run ahead and come up with an idea that is lacking in tools, and they'll turn to mathematicians as the tool experts for help. Like, for example, Einstein did when trying to formulate relativity. More rarely, they'll come up with a novel tool like Feynman diagrams without knowing why they work, and then the mathematicians will be called in as the experts to sort things out.
And of course it is true that physicists will come up with ideas like the Ads/CFT correspondence that arise from physical considerations. It's also true that the problems of physics can inspire new mathematics simply because they're new and interesting problems.