The other portions that are uncovered have more sensitive analog circuits. You need to be more careful of how you route power, to avoid noise, so you don't cover it all in a big network on the top metal layer, but instead it's routed more selectively.
You need to think about the return currents closely matching. In many cases on 4 layer boards, you are required to have two Vias per... Via. (One for the forward current and a second for the reverse current).
The PCB also serves as the fastest capacitor. The PCB layers itself form a capacitor measured in hundreds of Picofarads in most cases but more importantly, almost no parasitic inductance or resistance.
In many cases, boards are simulated at the Maxwell equation level to understand crosstalk issues. It seems like RP2350 is just fast enough where these effects could be noticed (at the highest clock rate)