I've never done this myself, but as I understand it [1] you can choose certain cores that the OS scheduler doesn't schedule anything on, then you can manually assign processes to them.
Assigning an entire CPU core to audio might be a bit much, unless you've got CPU cores to spare - the context in which I heard of this was high frequency trading, where the server really didn't need to do much except run this one application.