If it went away entirely, as in also from laptops, desktops and Raspberry Pis, without adapters being available, maybe.

But you're not going to be driving a speaker with a phone anyways. You'll need an amplifier, and if the aux port goes away, the amp will likely have whatever-alternative-input-there-is.

You can use something like hifiberry or justboom to embed a spotify / airplay compatible amplifier into your speakers: https://www.justboom.co/product/justboom-amp-zero-phat/

If Aux port goes away, it's still going to be Bluetooth or Wifi and of course there are plenty of plug and play chips out there on the market that should do the job pretty easily. So, to answer your question, no, the barrier wouldn't be that high at all :)