This is true, visual search has a optima when the contrast of the object in the search field is above certain threshold. Below that magic threshold, it is useless (for e.g searching for a Jigsaw puzzle piece). But, when visual search works, it is probably ridiculously fast for the evolutionary reasons you rightfully mentioned.

There are a couple of things when we search for something in a text box: We're pattern matching what we think their database contains. If the hysteresis between the search and the match is small, Album grid view might work. You have to know the cover art a-priori though. But if you are vaguely looking for something, list is far better.