It works so much better when it does but it’s not fully hashed it and a lot of stuff doesn’t quite work interface wise

Totally true, and I suspect that the same will hold true for Vision OS — controllers will always be optional (even when Apple has "the world's most advanced spatial controller" to sell us for $999/pair).