I'd slightly trollishly say nobody has aphantasia, it's just that some people make a false distinction between remembering what something looks like and seeing it. Same as what you're saying really, but makes people feel better about themselves, rather than worse.

It seems clear at any rate from the way these discussions always go that aphantasia vs. non-aphantasia, as real as the distinction may be, is hardly predictive of anything else about a person, and the subject only attracts so much interest because people mistakenly assume it's more predictive than it is.