Keep in mind that your visual cortex responds more to difference in colours and spatial frequencies more so than it does to flat expanses of colour. Things like drop shadows and borders add much-needed high-frequency acutance [1] at the edges of shapes, which helps with object recognition.
You see this all the time in the real world - take a look at the logo on these cans [2] and note the white border surrounded by the black border. Going from the base colour like this, all the way to white and then back to black effectively saturates the spatial frequencies at the edge of the shape and forces it to stand out from the background - I'm holding one of these cans in my hand right now and boy oh boy does that logo ever pop.
You see the similar techniques around all the time - look at the text labels on Google Maps, or the black borders around video subtitles. Similarly, see the gradient and drop shadows on the icons on an iPhone home screen - those are there to make them visually recognisable as objects, as opposed to rounded rectangles.
[1] http://en.wikipedia.org/wiki/Acutance
[2] http://i.imgur.com/WWRDsiE.jpg
Edit: further demonstrations of your visual cortex responding to differentials:
TEMPORAL: http://imgur.com/iqMFv5z.gif
If you stare at the center cross long enough, your retina fatigues and the pink dots start to disappear. Likewise, the rotating absence of a dot is replaced by green.
SPATIAL: http://upload.wikimedia.org/wikipedia/commons/0/0a/HermannGr...
The sharp spatial frequencies (unlike anything you would find in nature), aligned at right angles to one another interfere with one another so strongly that you start to see dots at the grid intersections (some may find this stronger than others, I sometimes get this effect from the Windows 8 homescreen, for chrissakes).