"There is another remark which suggests itself here and which physicists may find paradoxical, though the paradox will probably seem a good deal less than it did eighteen years ago. I will express it in much the same words which I used in 1922 in an address to Section A of the British Association. [...] I began by saying that there is probably less difference between the positions of a mathematician and of a physicist than is generally supposed, and that the most important seems to me to be this, that the mathematician is in much more direct contact with reality. This may seem a paradox [...] but a very little reflection is enough to show that the physicist's reality, whatever it may be, has few or none of the attributes which common sense ascribes instinctively to reality. A chair may be a collection of whirling electrons, or an idea in the mind of God; each of these accounts of it may have its merits, but neither conforms at all closely to the suggestions of common sense. [...] A mathematician, on the other hand, is working with his own mathematical reality. Of this reality, as I explained in section 22, I take a 'realistic' and not an 'idealistic' view. At any rate (and this was my main point) this realistic view is much more plausible of mathematical than of physical reality, because mathematical objects are so much more what they seem. A chair or star is not in the least like what it seems to be; the more we think of it, the fuzzier its outlines become in the haze of sensation which surrounds it; but '2' or '317' has nothing to do with sensation, and its properties stand out the more clearly the more closely we scrutinize it. It may be that modern physics fits best into some framework of idealistic philosophy -- I do not believe it, but there are eminent physicists who say so. Pure mathematics, on the other hand, seems to me a rock on which all idealism founders: 317 is a prime, not because we think so, or because our minds are shaped in one way rather than another, but because it is so, because mathematical reality is built that way."

That's not quite the argument you describe -- his point is more that mathematical objects as understood by the mathematician are more like mathematical objects as we encounter them casually, than physical objects as understood by the physicist are like physical objects as we encounter them casually -- the physicist will insist that the chair you're sitting on is really a sort of configuration of fluctuations in quantum fields, but if you count up to 23 then the mathematician will agree that what you just did really does reflect the underlying nature of the number 23.

(If you build all mathematics on top of set theory, then you will most likely treat the number 23 as some much more complicated thing. But you'll see that as an "implementation detail" that could be done in lots of different ways, rather than saying that really, deep down 23 is this complicated thing with lots of weird internal structure.)