But with the Vision Pro there is zero latency and the possibility of having many big screens, whilst when remote-desktoping into a Mac from an iPad you gain nothing and the high latency makes it worse.

This isn't correct. The AVP can run apps side by side so you can have a window for xcode open along side your app. With the iPad you only get one app at a time, which is a bummer. (Even if you had multiple windows open only one app can have the full screen experience at once.

Beyond that, even though an iPad can act as another monitor, it's not a big monitor. It doesn't really achieve the ideal of a workstation replacement.