Dereferencing an arbitrary integer cast to a pointer means you are dereferencing an arbitrary memory location, which includes memory that is outside the purview of the C abstract machine model, say the return address of the function. That means the effect of that operation can't be meaningfully constrained, and the only option for that is UB.

(Integer-to-pointer conversions beg the question of pointer provenance, which is a long, complicated topic that is still not fully solved, but does go to show that UB is actually a lot more complicated than most people expect.)