Godel's proof relies on the self-referential nature of the Godel sentence; without that, his theorem does not apply. Generally you need arithmetic, but also (something equivalently expressive to) universal quantification. Physical theories do not need to include that.

Note Godel's proof is mechanically exactly analogous to Turing's proof of the undecidability of the halting problem, because ultimately it's the same thing (Curry-Howard, Prolog, and all that). So you can bypass arithmetic, but you can't really bypass self-reference; just like programming languages need some looping or recursion (or equivalent expressiveness) to be Turing-complete, mathematical theories need universal quantification to be subject to Godel's Incompleteness Theorem.

Of course, you can have a physical theory that _is_ Turing-complete, say the Newtonian billiard ball model (and, y'know, we can build computers); but that doesn't mean the theory will necessarily tell you, as a static, measureable physical fact, whether a particular physical process (say, an n-body system) will ever halt or loop, or go on forever with ever-increasing complexity; so you could (in principle, in Newtonian mechanics) build some (mechanical!) physical system that simulates the Goldbach conjecture, or looks for solutions to an arbitrary Diophantine equation, but if there are no integer solutions you'll never actually be able to show it; the theory is incomplete in the mathematical sense, but just as complete a description of reality's rules.