I think I agree with you, and I also think that, "physicist do not really understand what they are doing" is phrased a bit flippantly.
Physicists understand what they're doing perfectly well, and they're not, for the most part, producing rigorous math. They are identifying phenomena which can be described systematically, as well as the systematic representations of those systems. Rigor, and math as a whole, is meaningful for a physicist only insofar as it facilitates that outcome.
It's interesting that you say that, "almost certainly, a rigorous formulation will come about later..." You seem to agree that experiment is a useful test for the rigor of operations, at which point it comes down to simple division of labor. The physicist has an expectation that an operation is rigorous, and his job isn't to establish its rigor; it's to arrive at the expression. He does his job, and mentions to the mathematicians that this operation 'should' be rigorous, and it'll be interesting to find out exactly how.
I think it's exceedingly wrong to say, "Their derivation rules are due to habit (in the sense of Hume), not due to inference or logic." Derivation rules support only one goal: predict outcomes of systematic phenomena. Physicists are neither nor are they trying to be mathematicians. The property that guides derivation is physical intuition, which I would say mathematicians malign at their peril, since the universe has thus far proved to be a useful machine for checking rigor.
And that's the heart of it. The universe has, for whatever reason, proven to be a detector, or perhaps executor, of mathematical rigor, and in describing the universe with ever-increasing precision, we develop results that feed back in to be explored and made complete. We can say, "it must be mathematically rigorous that this infinity can be discarded, so long as the universe is still functioning as a system that exhibits mathematical rigor," and be satisfied because the universe has shown us it is so. This does amount mathematically to not understanding exactly what it is you're doing.
For interesting reading on this by someone smarter than myself:
The Unreasonable Effectiveness of Mathematics in the Natural Sciences (https://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.htm...)