You're right that you can't have a theory without some language to talk about it (and it also requires people to come up with it, so is psychology more fundamental than physics?), but the theory of computation is about the laws that govern the power of mathematics. I.e. mathematics (and the universe) was constrained by computation long before anyone knew that. On the other hand, Turing was very careful not to rely on any mathematical and even logical results in his 1936 paper, not even on the principle of explosion (which is used in modern formulations of the halting theorem), precisely because he knew he wanted to get at the fundamental limitations of the very process of deduction itself (he does say that the theorem can be proven using the principle of explosion, but that would be "unsatisfying"). Turing showed that if a premise can be written as some string of symbols and so can the conclusion, then, due to constraints on the mind and body of the mathematician writing them, the process of deduction is subject to the laws of computation. This applies without any need to describe what the symbols represent, if they represent anything at all.
While the theory of computation does use concepts such as functions or sets, it knowingly treats them as higher level ideas than computation. I.e. a number or a set or a function is a name given by humans to something that can be computed or an abstraction of such a thing (so we can talk of non-computable things).
And if you want to include TOC as part of mathematics (some do, some don't), then it is its most fundamental part, more fundamental than formal logic (some people include that in mathematics, but most don't), which is subject to the laws of computation but not vice-versa.