In VR, dropped frames are quite noticeable. Even with Asynchronous Time Warp, while you won't get hitches in your view of the scene, you get hitches in anything that is animated. So, for example, waiving your controller around, it'll start to skip around. And you're always waiving the controller around. Nobody is a stone-still statue.
It's not really "round about ways of doing manual memory management". You can heap-allocate as much as you want, as long as you aren't doing it in a critical hot path. Object pooling is good practice, regardless of language and runtime, for real time anything. I do my major allocations and deallocactions during scene transitions, where the view is faded out already because making visible scene transitions in VR is nausea-inducing on its own.
And from a project management perspective, I've found it best to avoid using as much of Unity's built-in stuff as possible. Most of what they create are just garbage reimplementations of stuff that already exists in the .NET ecosystem.
I'm long-term going to get off of Unity. I'll eventually be building my own system on top of Xamarin, as I excise more and more of Unity out of my projects. I've already had to code my way around so many of their systems just to make quality apps that don't hiccup when things load that I don't really have a lot left to do.
All of the VR vendors provide C APIs for interfacing with their systems, which are the actual, first-class systems. Providing a P/Invoke layer on top of that won't be too hard. And Xamarin already provides wrappers around ARKit and ARCore.
Shader-wise, there's not enough processing budget to do anything but very basic shaders that I already know how to write, anyway. Any of the more advanced shaders that work well in VR are open-source.
Light baking can be done in Blender directly.
Unity's Audio pipeline is really bad. I have 50% of a reimplementation already. And again, spatialization is provided by the VR vendors, so there is very little of Unity's own code running in my audio systems anymore.
Sure, I wouldn't have learned how to do all of this stuff if I hadn't started with Unity, but once you get into trying to build scalable apps with pleasant user experiences, you have to ditch so much of Unity's built in stuff that it starts to look less and less like a good value proposition to continue to put up with the rest of their stuff, considering how buggy it can be.