That is also true. But as said, I haven't yet personally run into a case where refcounting was contested to a point of making it an optimisation priority.
Your requirements are probably similar to this C# queues class: https://github.com/Const-me/Vrmac/blob/master/VrmacVideo/Aud... That library decodes and plays realtime video + audio, both low latency and consistent performance over prolonged use were rather important. BTW that code runs on Raspberry Pi4, CPU performance is a fraction of what you’d expect on modern desktops or servers.