There's been a lot of debate over the years about what exactly it was that Ambrose was doing differently, and whether silent reading was really so uncommon in antiquity.
It also doesn't seem that unlikely. I know a fair few adults who mutter to themselves under their breath while reading, never having managed to kick the habit of reading aloud.
The argument for the Ambrose case is that the surprise isn't him reading silently, it's him reading silently while he has guests. (and elsewhere in the text Augustine mentions himself reading silently by himself in a way that suggests it's entirely normal)