That's a tricky one. Should we be able to actually write down each and every digit? If so then you'll need to invoke some sort of infinite related concept to allow it to happen. I'm not a mathematician but:
Let the digit for one be a single pixel (the simplest mark I can make), two is two lots of one pixels etc. Now we need an infinite number of these.
I don't think we can change over to say symbols made up from pixels either, nor mess with multi dimensions to add extra "depth". In the end we still need an infinite number of pixels or pixel properties - we could mess with colours but that is simply adding dimensions.
We can conceive of base infinity but I don't think we can actually use it as such except symbolically. We can decree that a particular symbol or construction for a symbol represents a particular digit within base infinity that we can define by other means, and we can do that a lot but can we do it infinitely often? If we relax the physical representation requirement, then I'd say yes, otherwise no.
I've no doubt that this concept is well understood and dealt with already by the pros.