I personally do not think I am all the special, but I from what I remember, I believe many of my issues with phonics were:
1. The inconsistency of the English language makes it so phonics is limited after a certain number of words, and then memorization and context must be used. For example, take words like cough, rough, through, though, etc. or words like read, lead, wound, etc. Not to mention all the silent letters we have too. If I am not mistaken, most languages do not have Spelling Bee contests because how clearly the language phonics map to spelling, e.g., German.
2. This is purely a hypothesis on my part, but I wonder if certain accents of English are better suited for phonics than other English accents? I grew up in the Southeast, USA. People slur words, drop off endings, contract words n >= 2 words, and even mispronounce words all. For example, the words "ten" and "tin" or "pen" and "pin" are not typically pronounced differently where I am from.
3. If you are like me and had speech problems, then phonics are substantially harder. It's hard to sound out the words when one's mouth cannot produce the proper sounds.
I do not doubt the other alternative methods are worse than phonics, and perhaps I am ignorant, but this debate also seems to be predominately an English only issue. Mandarin Chinese does not have phonics instruction to my knowledge, and they can read just fine. So, perhaps English is just a difficult language to read and pronounce correctly -- even for native speakers?