Nah. What you say is partly true, but I and others build xnu locally and incrementally all the time. Fast enough.
Linking does take some time, but it’s only really noticeable if you do very incremental changes (e.g. change a single value, build, boot, repeat).
Overall, building a whole kernel from scratch even for multiple configurations is not unbearable during normal development.