With arbitrary-precision, you can essentially replace the entire neural network with a single trainable parameter. [1] Likewise, in model checking, one often uses bounded-length bitvectors, because even a PDA with two stacks is Turing-equivalent. The poor separability between theoretical and physically-realizable models of computation is one of the weaknesses of the Chomsky hierarchy and classical approximation theory, and why more descriptive theories to characterize function complexity, e.g., circuit complexity, algebraic complexity theory and/or logical expressivity, are often needed.