His style of thinking seems very categorical, where one does not focus on structure of the objects in question (say prime numbers), rather on the mappings between/from/to them. In fact, from the cat. perspective you need to put extra effort to point out particular objects in your collection (http://ncatlab.org/nlab/show/generalized+element).

Grothendieck certainly knew what it means for p to be prime. I don't know, but maybe his thoughts went along the line of "primes are some substructure of the natural numbers and such and such changes happen in the functions that we can build on them".

So he was obviously able to speak about some collection of primes, yet whether an actual number is prime he probably doesn't think much about.

You or I might try to understand some mathematics by mapping it onto some kind of real-world analog, for example a model or graph or specific example case. This guy was a genius who didn't need to do that.

He was a very abstract mathematician, he built and worked with concepts and neither used nor needed numbers much.