Because they're not letters of the German alphabet, by definition, and they're not German vowels, by definition.
They're characters that represent vowel sounds, sure.
It's kinda the same thing as how in the English alphabet, Y is by definition a consonant, despite being used to represent a vowel sound in many words. (beauty, many, cyst, tryst, for example)