It's not that those roles are much harder to get as such. It's that the most effective way to get those roles is to work at GS (or an equivalent) first.
However if you start at GS/another IB then you get plugged into a network where you see a lot more of these opportunities without having to go out to look for them as much. It's partially a numbers game so the more opportunities you see the better the chance of landing one.