I feel similarly, though less about the lack of objective testing and more about the inability to even be sure we're talking about the same thing. Like I can visualize stuff fine, but not literally see it like some Hololens overlay -- it's in a sort of abstract internal space separate from what you see with your eyes. Sometimes I wonder whether some self-diagnosed aphantasiacs only think they are because they overestimate how potent "normal" visualization should be. (Or maybe these visual hallucinations are normal and I'm underdiagnosing myself!)

One method to possibly tell the difference is visual analogizing. Like, "picture" the Sydney Opera House, and then (without thinking in words) name some other objects it resembles. Someone with visual imagination should be able to rattle off stuff like sailboats or seashells or folded napkins, while a true aphantasiac should be lost without being able to look at a picture or derive an answer from a mental list of attributes.