A tiny tiny fraction of people talk about their mind's eye being part of their visual field, to the point that they can sort of functionally see -- hallucinate -- an imaginary object on a table like AR.
Another small number of people talk about imagining complete objects out of nowhere that they then have to analyse to describe; the imaginary thing has complete shape before its shape is described.
Most children can sort of whimsically play the imaginary-drinking tea game, imagining the shape of the teacups, imagining spilling tea or milk, but the mental images might not have attributes like colour or weight until they are specifically assigned to the visualisation. Like: what colour are the teacups? They don't have colour until you pick one, and then they do. Is it a heavy cup? It's not, but I do have a saucer. The more you add to the visualisation, the less translucent/ghostly/formless/conceptual it is.
The fascinating thing about broad aphantasia is that it appears to be way beyond even that; it's like things only happen conceptually, yet they still happen. Like the sibling comment from andrewflnr perceptively said: as if it's a conceptual blind-sight.
And when you consider our evolutionary pathway, perhaps it actually is the same phenomenon: a part of the brain successfully doing part of the task, without the support of the visualiser.