It can be done with BT - the $15 headphones I buy from random Chinese sellers work fine every time. The implementations seem to be at fault, not the protocol.
I'd say the protocol is at fault if it's difficult to implement.
The physical world always has implementation difficulties. It is a messy, noisy, uncontrolled place. On top of that, many implementers will avoid going full-ass when half-ass will do. No protocol will ever fix these issues.