That's how I pronounce it in my head, too, but I still know what it means. All is well here.
Similar to i18n and l10n for internationalisation and localisation.
Interestingly unlike w3c, so it's not standing for aaaaaaaaaaay :)
A11y is probably the easiest one to remember because it looks like ally. Which is what you're being by worrying about accessibility when you yourself don't rely on the standards.
Agree with you re: screenreaders.