What Really Happened in that Double-Blind Violin Sound Test | Hacker News Reader

What Really Happened in that Double-Blind Violin Sound Test

violinist.com

177 pointsyonibot90 comments

From the study's abstract: Differences in taste among individual players, along with differences in playing qualities among individual instruments, appear more important than any general differences between new and old violins.

Another case of science reporting failing miserably at understanding and conveying the results of studies. The three articles all come up short at even reading the abstract:

Daily Mail "Concert violinists can't identify the sound of a multi-million-pound Stradivarius"

Phys.org "Study shows even professional musicians can't tell old master violins from new"

Npr.org - Doesn't even come close to the same universe as the study in the submitted article to the point I wonder if the npr article covered something else entirely?

It's the Daily Mail; we should be just happy they aren't claiming playing a Stradivarius causes cancer.

Have you seen http://kill-or-cure.herokuapp.com/ ?

I think you might get a kick out of it.

It is awesome. At least they don't claim that vaccines are causing cancer.

Some other people do!

They've said coffee causes cancer which is worrying.

To counter this I drink coffee in a 1:1 ration which they say cures cancer so I reckon I'm good.

perhaps it's on the "cures" side...

I didn't read the original study, but based on your quotes it seems like you could append (p>0.95) to any of those headlines and have a meaningful statement which is supported by the data.

Daily Mail, Phys.org, and NPR all mess up, DM get singled out.

I know. It's very easy to mock the Daily Mail...so that's what we should do.

I wouldn't call it "failing miserably." Although it doesn't appear to have been the main focus of the study, the antepenultimate paragraph of the results section does say most violinists couldn't identify whether their violin was new or old (3 got it right, 7 got it wrong, 7 said they didn't know) [0]. There aren't p-values provided for that result, though.

http://www.pnas.org/content/109/3/760.full

edit: Also, you criticize the publications for not reading the abstract, but I wonder if you read it closely. It says "most players seemed unable to tell whether their most-preferred instrument was new or old," which seems right in line with those headlines to me.

There is a fundamental difference between being unable to tell the difference between two things, and having a difference of opinion on which is better; the article titles seem to claim the former, while the paper's abstract only states the latter.

Because the violinists were only allowed to try the violins for one minute, this study has the same flaw that the Pepsi Challenge had in 1975:

"In his book, Blink, author Malcolm Gladwell presents evidence that suggests Pepsi's success over Coca-Cola in the "Pepsi Challenge" is a result of the flawed nature of the "sip test" method. His research shows that tasters will generally prefer the sweeter of two beverages based on a single sip, even if they prefer a less sweet beverage over the course of an entire can."

In other words, a saturated sound could initially be more attractive but not as good for an actual concert.

http://en.wikipedia.org/wiki/Pepsi_Challenge#Criticism

I've had this same criticism of audio A/B tests: 30 second clips are just not decent to get a sense of the feeling a particular setup or recording imbues; only its quantitative momentarily detectable constructs. And only the ones the puny human brain and ear can discretely identify and remember, at that.

That's a serious issue. I play a stringed instrument, and purchased a "professional" model a few years ago. Among the instruments that I was trying out, some of the physical differences were sufficient that there would have been little expectation of identical sound, so the game was to discover the differences and then make a choice.

There were instruments that I rejected within seconds, but others that took me upwards of 1/2 hour to figure out how to adapt my playing to the characteristics of the instrument. There were issues that might have been unnoticeable to an audience member, but that would have affected my ability to play with good technique over the duration of a performance. Some of the instruments, including the one I ended up buying, changed my opinion of how I wanted to develop my tone and playing style.

Granted, I was always faced with a slight doubt about whether any of it was real, but without any good way of settling that issue. Also, a virtuoso might have been able to make all of the instruments sound the same with much less effort.

Oh, and don't get me started about bows.

They also mention in the post that the old violins were shipped, and no adjustments made, which means bridges could be improperly positioned, sound posts in poor positions... having these properly adjusted can make the difference between a great sounding instrument and a screech.

You can find similar issues with loudspeaks (those overly enhancing basses sounds better, but perform worse) and TV displays (overly saturated colors).

Indeed, one of the challenges of comparing audio systems is that -- rule of thumb -- louder sounds better. To have any hope of exposing effects other than overall loudness, systems under test have to be matched for gain.

If it is to compare my thumb say that —at the same loud the better sounds lower.

I took the Pepsi Challenge at a trade fair in the 70's, and had no trouble distinguishing the Coke. I preferred the Coke, said so, and the Pepsi spokesman's response was:

  "Next!"

I figured I wasn't getting on camera.

Anecdata:

My friends and I will often dial in a guitar or bass tone that is very pleasing when we play individually, but sounds like junk when you have guitar + guitar + bass + drums. After noticing this we also noticed that a lot of the bands we thought sounded the best used tones that were individually good but not fantastic because the were shaped to best fit the sonic space they existed within than to sound amazing in their own right.

I'm surprised that the Pepsi Challenge thing isn't more obvious to people. I mean, hasn't everyone experienced this themselves when it comes to those "healthier" versions of snack foods?

First bag: "Hey, these taste much better than I expected! They're just as good as my normal snack. Eating these instead of my old snacks will be easy."

Second bag: "They're pretty good, but I do taste the difference."

Third bag: "Oh god, get those away from me."

It's not just sip test versus can, but it's how things taste when you come back again and again.

I'm not a violin player, but to me the allure of hearing a violinist play on a 300 year old violin isn't that it's the best sounding violin on the planet. It's an opportunity to witness a small part of the history of an incredible piece of craftsmanship.

I mean, how cool is it to hear music come out of an instrument that spanned the lives of Bach, Mozart, Haydn, and Paganini?

Wait a minute. She claims:

"Now a lot of you found it easy to pick the Strad... But it wasn't actually our task to pick the Italian in this study -- it was to pick our preference."

But NPR directly quotes the guy who ran the study:

"JOYCE: Curtin says of the 17 players who were asked to choose which were the old Italians...

CURTIN: Seven said they couldn't. Seven got it wrong, and only three got it right." [1]

Maybe she wasn't one of the 17 asked, but it definitely was studied.

[1] http://www.npr.org/templates/transcript/transcript.php?story...

The abstract [1] says:

"We asked 21 experienced violinists to compare violins by Stradivari and Guarneri del Gesu with high-quality new instruments. The resulting preferences were …"

It then goes on about preferences and says nothing about identifying them.

So it's the abstract of the study, written (I assume) by the researchers, and one of the participants vs a journalist "quoting" one of the researchers.

[1] http://www.pnas.org/content/early/2012/01/02/1114999109.abst...

The abstract you linked does say something about identifying them: "most players seemed unable to tell whether their most-preferred instrument was new or old."

And in the full text of the study: "Asked about the making-school of their take-home instruments, 17 subjects responded: 7 said they had no idea, 7 guessed wrongly (i.e., that a new violin was old or vice-versa), and just 3 guessed correctly"

edit: the link to the full study is http://www.pnas.org/content/109/3/760.full in case anyone was having trouble finding it from the abstract.

Yup. Maybe what happened is, since only 17 of 21 responded to that question, perhaps she was one of the 4 that didn't.

It is a pretty damning study: Few could identify the old, expensive violins AND most preferred newer, cheaper violins. It means that the tradition of loaning Strats to top violinists may actually make for (slightly) worse performances.

I love blind studies.

Most of the criticisms of the study are inaccurate, but I would like to see this redone with the old violins receiving the same tuning/treatment as the new ones. Depending on the condition the old violins were received in, not being allowed to tune them could have had a big impact on preference. I know I'd rather play a $50 guitar that's in tune than a $5000 one that's out of tune.

Maybe the instructions to the participants were so unclear that they didn't know whether they were supposed to be picking the strad, or their favorite.

This post was regarding a test conducted in 2012. I'm not saying it isn't relevant or that the reporting around the new one is good, but the study she was a part of is a different one from the study making the rounds today:

http://www.pnas.org/content/early/2014/04/03/1323367111.abst...

Yep, the one she was part of was basically a "pre-run", which allowed the run of a new, better and extended study.

Here's Ed Yong's reporting on the new one, gives most of the meat and introduces the old one (and criticisms to it): http://phenomena.nationalgeographic.com/2014/04/07/stradivar...

The objective of this tests seems to be more to prove a point than to be fair, I'm not surprised.

And the test looks like it's blind, but not double blind.

It seems all rigour goes out of the window in order for the researchers to prove their point

And of course taste is subjective, but a misadjusted violin will not sound great even if it's a good violin. (Same with a bad violin, modern or old)

> "professional violinists can't tell the difference between modern violinist and old Italians," then I think we need a

> different study in which violinists are actually asked to identify that

Of course.

I'm not convinced there is much interference from the conductors to warrant a double-blind study. Either way, I believe the original paper never claimed that "professional violinists can't tell the difference", that's the media twist. It proved beyond reasonable doubt that professional musicians do not prefer the antique models based on any objective qualities.

I wouldn't even say the study proved that point. The fact that the older fiddles couldn't be adjusted for sound optimizations while the new ones could makes a huge impact on the study - it invariably unlevels the playing field.

Normals don't understand the difference an adjustment can make. You can tap the sound post half a millimeter and feel like you're playing on a completely different instrument.

I believe rare instruments like those are always kept in pristine conditions, and nowhere it is mentioned that the new ones had adjustments done, you can assume they all were received in 'normal playing condition' from their owners.

It's a reasonable assumption to make that the new ones had adjustments done. Even larger was the unchanged strings. I could easily imagine that the older ones, since they're not allowed to be modified in any way, had older strings than the new ones. That alone could produce a huge difference.

That's a difference, but it's a difference with obvious large implications for using the things in performance as well as experiments. As such, it's hard to see how it's a problem in the experimental design; if you're not allowed to adjust the Stradivarius while it's in your possession, maybe a newer model really is the right choice for a concert?

That is fair; but, my assumption in this situation is that if the musician owned the instrument themselves, they would have been allowed to tune it. Not being allowed to change details may have been a way for the benefactors of the instruments to keep them from being harmed

With Stradivarius violins costing millions and millions of dollars, I always assumed that the people playing on them didn't, in the typical case, actually own them (much like how science is now mostly practiced by professionals funded by grants, instead of by the bored idle rich). But I'm not informed on that point; it could be very different.

I don't know the exact breakdown of numbers, but it's increasingly common for a Strad to be owned by a benefactor and loaned to the musician. I don't know of any other artifact that straddles such extremes of being a treasure but also a work horse. Even the owners of lesser instruments have to deal with the implications of taking a valuable and delicate instrument out of the house to play in a saloon.

For this reason, tweaking a Strad is unlikely to be done casually. Re-stringing? Sure, at the very least E strings, which break all the time. But a set of fiddle strings is damned expensive, and takes time to break in so they don't sound too harsh, so re-stringing has to be coordinated with one's performance schedule.

Online, one can find a list of every known Strad, its history of owners, etc. Interestingly enough, a couple of them are listed as missing or stolen.

You believe wrong. Also, even if they are, traveling with an instrument, especially and old finicky one (and they do get finicky when they get old) can throw it out of adjustment, a seam can pop open, a string can go false, etc.

The new ones, on the other hand, since they were actually new, were probably being maintained by their makers, who undoubtedly made sure they were in top condition when they went into competition with a strad.

I'd say they proved conclusively a modern violin you can adjust is preferred over an old violin you can't adjust, and pragmatically that's the only meaningful test since since there doesn't seem to be such a thing as an older violin you are allowed to adjust freely.

According to the abstract, the study was performed "under double-blind conditions"

http://www.pnas.org/content/109/3/760.full

tl;dr: Study seeks to determine if Old Master violins actually sound "better", media reports misleading headlines (shocker), musicians are angryface.

I remember when this study was first concluded and reported on. The CBC had a woman who participated in the study (maybe this very woman) on the program and it was so extremely obvious that her professional pride was wounded, and she really wanted to make herself feel better by telling the world that she really could pick out the Stradivarius.

I have no reason to doubt her; she's a professional violinist.

But it seems to me that the study was actually very effective at showing that masterwork violins do not necessarily sound better than modern violins, contrary to the regurgitated opinions of hundreds of thousands of novice violinists out there.

'I have no reason to doubt her; she's a professional violinist.'

You know that claiming to be a professional violinist is an appeal to authority rather than a logical argument, right?

That said the study does seem to be massively misreported.

It may not logically follow, but in the real world it is not a strange thing to trust an actual expert on these topics over, say, a writer at the Daily Mail. Why object?

It's even better to examine their argument sans credentials.

It's even better to do both, and embrace a complete and holistic argument.

>You know that claiming to be a professional violinist is an appeal to authority rather than a logical argument, right?

Am I on reddit or something? You know that starting a statement with "you know that..." sounds pretentious as fuck, right?

Yes, I did know that.

So you don't think that I, a lousy trumpet player who hasn't picked up an instrument in almost a decade, have reason to give a professional violinist the benefit of the doubt when she said she could pick out the Stradivarius?

"Appeal to authority" is a shorthand form. The thing people object to is more formally known as "appeal to false authority" (think "I'm not a doctor, but I play one on TV").

Improper appeals to authority are not limited to appeals to false authority; appeal to authority to dismiss evidence are also fallacious.

This is not so; we rely on authorities because we can't judge everything ourselves. If you show me some new evidence on the role of free protons in the aging process or whatever, it's not a mistake for me to say "I'll ignore that until such time as <someone I trust> addresses it". I might be wrong as to reality, but I can't avoid using that exact strategy in almost all of my life.

But that's not an appeal to authority, really, is it? That's your relying on an authority to inform yourself, rather than pointing to authority as why others should believe what you do. These strike me as subtly different, though there's clearly some overlap in considerations...

There's a difference between using authority status as a filter to find likely-to-be-valid arguments and using "so-and-so says X is not true" alone as a rebuttal to evidence that X is true. The former is a pragmatic research tool, the latter is the fallacy of appeal to authority.

> actually very effective at showing that masterwork violins do not necessarily sound better than modern violins

Really, it's only effective at showing that an unadjusted violin with strings of an unknown age doesn't necessarily sound better than a younger violin with younger strings, that has been recently adjusted...

Really, it's only effective at showing how violins compare when being played by blind musicians in hotel rooms, while being observed by researchers. But at some point you have to stop finding caveats and find some evidence for the contrary position.

What is the contrary position? That older violins have some kind of superior magical design that can't be replicated? Has anyone validated the claims for why these old violins are supposed to be better?

>What is the contrary position? That older violins have some kind of superior magical design that can't be replicated?

Pretty much.

Well, I don't subscribe to the idea that old violins are better than anything we can make... but you can't simply compare an adjusted violin to an unadjusted one.

I found this article [1] from another participant in the study a much better read. He also is a violin maker which could explain why he is happy with the results, but he isn't trying to justify any of the results. His closing words say exactly the same as the linked article, but in a very sober fashion.

[1]: http://www.artsjournal.com/slippeddisc/2012/01/exclusive-how...

I would love to see a study like this done with vinyl vs. digital.

1. Put some vinyl-loving DJs in a hotel room.

2. To factor out identifying pops & scratches, pre-record a vinyl song onto digital. Record it several times and use each digital recording only once.

3. Cover their eyes and randomly switch between vinyl & digital.

See if they can tell the difference.

I think your test actually factors in one of the reasons people like vinyl - it's not that the sound is necessarily better, but with analogue, there's a uniqueness factor because it's a physical needle hitting a physical groove.

And the fact that there aren't small samples of audio being played back at high frequency that rely on the quality of the digital to analogue converters, as well as the other systems in line such as the type of amplifier (class A, A/B, D), the circuitry it goes through (is it solid state or is there a valve somewhere in there?) as well as the speakers used (are they balanced and not too bass/treble heavy and do they reproduce the sound in a consistent way taking into consideration speaker box resonances, room set up and proximity effects next to walls and reflections, and can they reproduce proper bass note fundamentals or are they only producing harmonics of the fundamental?).

There is a lot to compare really. But most people think entirely vinyl vs MP3s (!) where MP3s have parts of the audio spectrum cut out in order to facilitate better compression; they typically cut out very high peaks and basses, but you need those high peaks and transients to reproduce snare drum cracks and cymbals (eg. listen to a 128kbps MP3 and see how RUBBISH the cymbals sound).

Even if you are feeding digital audio into a system, the speakers and room set up makes a COLOSSAL difference so it isn't fair for people to dismiss "analogue" vs "digital" arguments; it should be "good sound system" vs "bad sound system" arguments.

To factor out identifying pops & scratches, pre-record a vinyl song onto digital...

This would simply be a digital reproduction of a vinyl recording. (Something I quite like BTW). Vinyl is a "thing" because the process plays a part in creating that sound people love and not just reproducing it.

It's more a "thing" because of other social status reasons.

Which is fine, but that nobody admits to that. How many vinyl enthusiasts would be just as ok consciously listening to a digital creation of the pop and scratch filter they enjoy? PVC is as awful, environmentally bad material and heavy to lug around. Why wouldn't DJs rip their vinyl to digital or just apply a filter to digital sources?

Most enthusiasts believe vinyl is actually better (quite adamantly) and that's what this experiment would test.

I don't know about a pop and scratch filter but I like some of the recordings of my old records I've captured more than the clean digital versions available now. I don't want random pops and scratches, I think maybe I want my pops and scratches.

This reminds me very much of some studies where audiophiles did double blind tests of various solid state amps against tube amps. Some audiophiles say that the tube amps sound "warmer", maybe they like 60hz hum. Whatever. Tube amps look cool and that is what is really important to them I think much in the same way that it is a huge status symbol to have an old violin. Same thing I think. Not that there is anything wrong with that, you get your inspiration in whatever way works I think.

The difference between tubes and basic transistor amps is very noticeable if you know what to listen for, especially when they're used at volumes loud enough to cause distortion, and this is definitely measurable scientifically.

Some people do like the fuzzier, "warmer" sound (the hum is part of that) that tubes have.

The "warmer" effect is not the hum. I have read it's how the amps process harmonics (odd ones vs even ones) and that this effect is actually measurable through testing equipment.

It's the fifth harmonic isn't it? I would disagree with zw123456's "whatever" with regard to analogue systems.

(It isn't just the look of valves - they keep your room warm too!)

Well, uh, it seems much more complicated than I remembered: http://en.m.wikipedia.org/wiki/Tube_sound#Harmonic_content_a...

What would have been really interesting is if they had also conducted the test with another group of participants who were all informed about what each violin was. Then we could see how much influence lore and history have on what people like, by comparing the scores between groups.

Perhaps not with violins, but blind wine tasting is very hard to tell differences... http://www.liquidasset.com/WEILVDQS.PDF

They couldn't even tune the old ones or change the strings, looks like a useless study.

TLDR: "Im an audiophool. I know whats great by the price stick^^^I could totally pick Strad blindfolded, I just didnt feel like it that day."

read again.

He did a rather poor job of expressing himself in a sensible manner but I must admit I had the same takeaway from the article.

pick best does not mean pick oldest, news articles said study was identification, study subject says study was preference, study says study was preference. How you could possible reach the grand parents takeaway is seriously concerning.

explain.

Yes. Im sure this is why she herself uses 200 year old violin instead of the modern one.

> for the last five years, a mid-1800s Italian.

Whole article is a result of authors cognitive dissonance.

I dont see much difference between her and people arguing for $1K speaker cables and later failing to distinguish them from coat hangars.

Her mid-1800s Italian has completely different characteristics from a 17th century Italian. A string player spends months or years trying many instruments before deciding on one. I've never known a professional musician who cares whether they're playing on a modern or an older instrument, they care whether they have the right instrument for them. You have no idea what you're talking about. Please just stop.

The point of the article is: the study was "Pick the one you like", not "pick Strad". According to the article, they were never asked to identify.

What for?

Author owns old violin, author does the test, author picks modern violin over $10 million ones, author tries to defend herself post facto (bad test, not tuned, blabla), then some audiobabble (sweet sound, personality, other nonsense).

I think the whole problem lies in the underlying assumption that older, more expensive violins sound ABSOLUTELY better. The author does not support this claim, and the study also demonstrated otherwise. The author, and apparently the study as well, both are in agreement that it is all about personal preference.

In fact, the author admits that she preferred some of the newer violins to the older ones and with that "audiobabble" she says they SOUNDED better. So when how can you blame her to "choose violins on price stick" and then also call her ignorant when she prefers some other violin "over the $10 million ones" ??

The author concludes that the reason older violins are so valuable isn't just because they have some magical qualities that make them sound better, or they are very obviously more amazing than newer ones, she says that older violins have this history attached to them. This imagined beauty to them because of the legends attached to them.

Oh and that "not-tuned" "older-strings" stuff was there to imply that an old violin, is an OLD violin. A newer violin will have an advantage due to the new parts, the new strings. But apparently nothing is absolute. A new violin isn't always better than an older one (hence her preference for her 200-yr-old one at home). Heck, she even said how she has heard different complains about newer and older violins. So how in the world was your take "i am an auidophool, i choose based on price stick" ?!?!? She EXPLAINED how things are completely independent of age or price.

>This study certainly doesn't tell us what happens between a player and a fiddle over a long-term relationship, and this is an important factor.

No, it's not, because the same long-term relationship can happen between a player and a modern violin as well.

The article explained how the long-term relationship with an older violin, which is more 'broken in' but temperamental day-to-day, might be different than with a modern violin, which is more 'dependable' day-to-day but can experience significant shifts in tone as it ages.

Wood also seems to change depending upon how it is used. Conventional wisdom is that a guitar gets "used to" how it is being played. I do like the comments about finding the unique characteristics because every instrument has a personality.

Conventional wisdom in classical guitars ALSO says the guitars have a lifetime. The wood is tight, then it opens up, and then it breaks down gradually over time--roughly 15-20 years for cedar tops and 40 years for spruce tops.

If that is actually true, I suspect that feeds into this study. The Stradivaris, while phenomenal, are long past their peak and are now at the point where they are breaking down from playing and are likely only being preserved by the rarity with which they are being played.

I wonder if the "moderns" used in the study actually had some mileage on them. It would be interesting to see if a 20 years old "modern" with heavy use actually scores better than a 5 year old "modern".

Overall, it's actually good news. I suspect that this is like modern classical musicians, the geniuses are as rare as ever, but the average level of achievement across all classical musicians is WAY higher now than it was even 100 years ago, and certainly moreso than 200-300 years ago.

It means that those of us who can't afford Stradvaris can still get excellent instruments.

I forget the guitarists name, but as a joke his band mates went to a pawn shop and bought him a used guitar. That gig he played so well it became a tradition. Every new gig he demanded a used guitar. The name will come to me. I do know there are a few horrid guitars out there though--the one at costco--the Yamaha, I can't keep it from buzzing though.

Reply on news.ycombinator.com