Forget arithmetic; I remember conversations with exceptionally capable academic mathematicians who in many cases couldn't count without making mistakes. The likely reason: every number triggers a symphony of mathematical results and complicated ideas in their minds, from both memory of association as well as logic itself, and by necessity their minds also conjure up new concepts every now and then. All this added mathematical machinery interferes with the actual counting process at hand.

In an extremely interdependent field like mathematics, maybe the more you know, the harder simple operations become, because our short term memory inevitably gets occupied and interrupted with all the more complicated stuff.