Pixels are mapped to memory locations, but they don't have to be, if you can access the map directly. I don't exactly know what I'm talking about here, just a thought.
I mean, pixels are just like a memory except that they glow. They hold their value and can be writen a new value in sync with a clock (which is normally 60 Hz for most monitors which is MUCH slower then on chip memory). There could be no performance benefit.