Any physical system that can be described by a set of mathematical laws is bound to have some axioms that simply cannot be proven within its context. Godel's incompleteness theorems prove that.
Hence, we will never get to the bottom of the reality of our universe.
Gödel's incompleteness theorems are about mathematical theories, not about physical theories. It may well be that for physical understanding a very limited arithmetic is sufficient. Also, there may be statements that are unprovable, but these might turn out to be irrelevant to sufficiently understanding the universe.
Also, an axiom, by definition, can not be proven. The things that are proven are theorems, statements, etc. An axiom is an assumption.
Eventually, it all depends on what level of understanding you want. But Gödel's theorems have little to do with that.
In fact, the theorem showing how to do so (Bayes' Theorem) is so utterly elementary we teach it in the first probability course everyone takes.
This means that at some point, existence must have been started by something non-existent. Because we only comprehend cause-and-effect, we can't possibly find the origin in a provable way while researching from within existence.
Where Gödel comes into the picture is that he gives us no right to believe that only those axioms are all there is.
If the reality is mathematics, Gödel justifies the Multiverse hypothesis: just because we are objects within some theory doesn't mean there don't exist other theories (perhaps superset of this one).
"Unable to prove" in the Gödel sense can be then equated with "unable to peer into another universe".
Gödel's sentence becomes analogous to some phenomenon or artifact of another universe which cannot exist in ours because it cannot be derived from its axioms.