> If you can remember the duration of notes based on a few different symbols
I can do this but I find it a heavy cognitive load to translate symbols into duration while I am also reading/playing music: that was my point.
> Could you tell the difference between 31 and 32 1/32 rests?
Yes, if there was an adequately marked time scale/axis. There are already bars: just turn them back into the equal areas they are already abstractly representing. (I'm not suggesting that you notate a rest with only blank space - you can still explicitly notate a rest).
Actually I'll nitpick myself about this. They aren't the same, semiotically, because traditional musical notation is purely symbolic (the signs have literally nothing to do with the signified) whereas white spaces would be somewhere in between indices and icons, which do share some semblences of some kind with the signified. In this case, width of space maps to duration of time, which is a fairly basic embodied metaphor in the George Lakoff sense.