Myself and many friends who aren’t native have struggled with speaking fluently because of it. Most of us still mispronounce some words (my friend pronounced “draught beer” like the lack of rain, instead of like draft).
Doesn’t mean things should change, but it’s certainly not a “non-issue”
The bureaucratization of language is more problematic in my view, where things are seen as wrong and right and we try to cram the beauty of of natural language into a restricted box that can be cleanly and easily defined and worked with universally. I quite literally have nothing but detest for this conception of language, that it must bend to the whims of rigidity when it's very clearly a natural, highly chaotic dynamic system constantly undergoing evolution in unexpected ways.
It's also decentralized - there's no authority to tell the English-speaking community how to spell things or how to say things.
I think these are both advantages that outweigh the phonetic inconsistency.
Assuming we wanted to make English a phonetic language, then your question is kind of moot: phonetic means we need to pick the pronunciation rules for phonemes, which would make other ways to pronounce these phonemes incorrect. Some of currently-correct english would become incorrect english.
> For example, I would guess that 50% of English speakers are non-rhotic
Note that accent isn't really what people talk about when they complain about pronunciation. The problem is that there's no mapping from letters to phoneme in any english accent: laughter/slaughter, draught/draught, G(a)vin/D(a)vid...
Phonetic languages do borrow words from other languages too, they adapt them to their own language keeping the pronunciation (the only example coming to mind right now is the Czech for sandwich, sendvič). English could do that just fine being phonetic was a goal
The systems at work in English are not nonsensical like people like to parrot. To say it's not phonetic is just wrong on every level as well.
Frankly I'm fine with the historical oddities that have led to modern English. If non native speakers have issues, that's tough luck for them!
(Webster also went on to suggest dawter over daughter, to remove more of these vestigial augh spellings, but that one still hasn't caught on even in the US. Just as the cot/caught split is its own weird remaining reform discussion.)