In your statement, I'm not sure by which definition physically grounded things are a sufficient condition for being computable, but I think Penrose depends on the Turing notion of computability and the hardness of the halting problem.
We can of course move goal posts, redefine computability however we like, to get whatever conclusion we care for, but I think that Penrose effort is intellectually honest.