The look of music

Extract from the abstract - "People consistently report that sound is the most important source of information in evaluating performance in music. However, the findings demonstrate that people actually depend primarily on visual information when making judgments about music performance."

Not too surprising a result. It would've been interesting to propose a model of cognition that might account for this effect. The "embodied music cognition" school might, for example, propose something like this -- "the brain needs to process the inputs and reconstruct the body movements required to produce the sound so that the music can be experienced in the listener's body. This is less work when given the visual signal alone than when given the audio signal alone."

Also, I was curious about what instruments or vocal music the competitions were about, but the paper doesn't talk about that (in my quick scan through). I venture a prediction that for vocal music, the audio alone is likely to be a better predictor than the visual alone, but for piano and violin the visual may show this effect.

Another point that struck me was that the experiments featured the performances of only the top-three finalists in competitions. These are already performers who the judges in those competitions selected as good, suggesting that perhaps these performers are not very distinguishable by their sound alone anyway. It would've been better to also include non-top performers to see how the audio-alone and visual-alone inputs fare when they are also included. Then again, stating the result as (simplified) "among top performers, the visuals are important" reduces the surprise factor :)

This reminds me of my occasional habit of watching movies with the sound off. When you watch a movie with no sound, you can get a great sense for how well the actor is "selling" the role based on their facial expressions and body language. Without the distraction of the soundtrack and dialogue (often dubbed in later) you are left with a raw performance and get a great sense for how well the actor is doing.

A copy of the actual paper is available at:

http://www.pnas.org/content/early/2013/08/16/1221454110.abst...

Unless the judges were judging blind, then this could just mean that the judges are as susceptible as laypeople to awarding the best-looking performers.

Kahneman's Thinking, Fast and Slow is all about this. He would say that the visual impressions are "System 1" and the fine musicological distinctions are "System 2".

Interesting. I don't go listen to live music very often, but when I do, I often close my eyes to focus on listening.