The converse is also true. For example, the way native Swedish speakers pronounce seven (sju - example pronunciations at https://forvo.com/word/sju/ ) is 1) Hard for me to say and 2) No matter how I say it, the response from Swedes is, "You said <X>; it's actually <X1>" where both <X> and <X1> sound exactly the same to me, so I don't hear the distinction they are trying to point out. I assumed the same happens to Western folks when Indians/Hindi speakers try to explain the difference between the various T sounds.
> The bigger issue is English lacks a retroflex plosive (tongue curls), and aspiration is non-phonemic (does not carry a meaningful distinction)
But English words do seem to distinguish meaningfully between what you term 'voiceless' and 'aspirated voiceless' isn't it? For example, there is a difference between 'time' and 'thyme'. Ignoring the difference between 'y' and 'i' for a moment, wouldn't both words be the "same" to English speakers if what you are saying is true? Isn't 'th' just the aspirated version of 't'? (Not contesting what you are saying, just curious to understand.)