In general I think you're right. With the correct encoding, it's just a mechanical limit.
It just depends on the specific encoding you use. GMP, I believe, is only limited by the physical memory size. Python's implementation is also limited by the encoding (not sure how it works concretely, but it doesn't seem to be a memory overflow):
x = 1
while True:
x <<= x
> OverflowError: too many digits in integer