(I think there is a discussion to be had about making the distinction "CS" vs "theoretical CS" as the GP comment does, or if it should be "CS" vs some other term ("computing"? feels a bit general))
Computer science was developed by mathematicians as a study of algorithms, procedures for computing, and methods of abstraction. In the words of Hal Abelson, computer science is not a science and it's not really about computers in the same sense that geometry is not about surveying instruments [2].
Numerical methods and algorithms are fields of math as old as geometry, especially if we focus on the Babylonian or Chinese styles.
Hermann Grassmann sought to formalize arithmetic, not wishing to assume them as granted. In doing this, he also connects recursion, induction and the natural numbers (he would have known of recursion from its early application in the theory of combinatorics). Peano, Dedekind, Frege, Zermelo and many others would also work on the foundations and axiomatization of mathematics and deduction. Computing began as a side-effect of attempts to formalize just how far such an approach could be taken. The Turing Machine arose to tackle Hilbert's Entscheidungsproblem. The lambda calculus as an approach to the foundation of mathematics. Functional programming languages were originally part of tools meant to study formal mathematical objects while Logic programming sought to apply ideas from formal logic and the axiomatization of mathematics to automatically search for programs.
Dedekind said: "In speaking of arithmetic (algebra, analysis) as a part of logic I mean to imply that I consider the number-concept entirely independent of notions or intuitions of space and time, that I consider it an immediate result from the laws of thought."
What we find is computing reaches right to the foundations of mathematics. Whenever we try to systemize thought, we end up with ideas which seeming inevitably also lead to the foundation of computation.
(Mostly joking. Mostly!)