Isn't this problem solved in the speech recognition domain by CTC? Nobody annotates phonemes there.