This is what I'd emphasize. To sight read, the mapping from sign to note has to be so automatic it's unconscious. In standard notation this mapping looks different at every one of the middle four octaves, which nearly quadruples the size of the "multiplication table" you're installing in memory. Since your exposure in practice to the further ends of that range is less frequent, you're still slowed down by some notes even once the middle ones are automatic to you. (And there's probably some "cross talk" for a long time -- at least, that's how it felt to me.)
It's strange to me when people are like "eh, what's the big deal" about a UX failure that seems this big.