I repeat: I am pretty confident they only know who has permission to read it
right now.
It works like this: their server gets a request to read messages, and checks if the client currently has permission to read messages. If so, it reads the messages. If not, it returns a permission error. If you don't have permission, it doesn't attempt to analyze history to figure out whether you had permission in the past.
In fact, Discord is pretty explicit about not letting you have any interaction with things you don't have permission to view. It doesn't even tell you you don't have permission. It just silently removes them from your menu, and if you try to access them with a link, it tells you "unknown error".