The "absurd payload size" is the effect of having at least four different SDKs (macOS, watchOS, i[Pad]OS, tvOS) bundled, including a simulator for three of them.
Xcode 15 is a ~3.5 gigs download, and includes only macOS SDKs, with all of the other (including the visionOS SDK + sim) ones being downloaded on-demand.