There's a library Valve made for spatial audio for games (inc. VR). I've played around with it a bit, it's incredible. I'm surprised more games haven't adopted it.
edit: ah, it's because it is a plugin for Unity's audio system which I've already determined adds more latency than I want.