How Well Can You Hear Audio Quality?
npr.org
npr.org
Double-blind means that the person administering the test doesn't know whether A is the compressed file and B the uncompressed or vice versa, and whether X is a copy of A or B.
Without a properly conducted test, users who want there to exist a difference between 320 kbps that they are able to hear will convince themselves that they can hear it.
But likewise, those who believe they shouldn't be able to hear any difference will convince themselves that they don't.
The A X B test eliminates this, because though you know that X is exactly the same article as A or B, you don't know which, and you don't know which of A and B is the higher fidelity one. If you believe that there is no audible difference, you can at best randomly guess at the identity of X. If a large number of subjects are tested and they all believe that A and B sound the same, the distribution of their identifications of X will be consistent with a random binary choice. If there are some subjects who can in fact tell the difference, that will show up in the data as bias toward the correct identification of X.
Those who think they have "golden ears" and convince themselves they can hear something they in fact cannot are called out by this test procedure, by their failure to actually identify X better than a random guess.
An old buddy of mine, Dan Dugan, hung two pink sheets at an Audio Engineering Society conference on opposite walls to illustrate this point, asking people, "Are they the same color?" It would be trivial to distinguish them side by side, but accuracy is greatly reduced when they are distant in time or space. Human perception is all about edges, not about absolute measurements.
Mine was "A X not-X B". I had a command line program that I could run and give two files, the reference file (typically a straight uncompressed rip from the CD) and a test file, typically an MP3 made from the reference file.
My program would then conduct a run of N trials. For each trial, it would assign the reference file as file A, the test file as file B, and it would randomly assign either the reference or test as file X. There were buttons I could press to listen to file A, file B, file X, or the file that was not file X. When I thought I had figured out whether X was A or B, I could signal my choice, and the next run would start. At the end, it would tell me how many I had gotten right.
To make sure I wasn't picking up clues from any slight delay from increased overhead in starting playing an MP3 file compared to starting a wave file, the program tossed in a random delay before executing a play command.
If I were doing this again, I think I would also toss in some minor randomization of the volume level on playback.
Anyway, the results were about what I expected. I do not claim to be particular astute at subtle audio listening, and found that the LAME encoder at 128 kbps was fine.
I was able to easily hear problems with BladeEnc at that bitrate. For instance, the opening piano chords of Cat Steven's "Morning Has Broken" just completely disintegrated with BladeEnc. I could also hear a problem with the trumpet on Dire Straits "Your Latest Trick". It's hard to describe, but if I were to compare it to being hit in the face with a baseball bat, the original and the LAME MP3 were like getting hit with a solid bat, and the BladeEnc MP3 was like getting hit with a hollowed out bat...the shell was there, but it couldn't do much damage.
When I say I could hear these problems, I mean that I could pick out the MP3 with BladeEnc every time in repeated runs of 20 trials when I was testing with those two tracks.
In fact, I was JUST reading about this sort of stuff last night when I decided I didn't think I have a need to keep my archived lossless flac almbums anymore. I carefully ripped mine and many borrowed CD's to flac using EAC like 6 years ago with intention that I could get rid of the cd's themselves to Goodwill. I then just converted them to 128kbit/s mp3 for portable/phone listening. Knowing at the time 128 mp3 had its limitations I would just re-convert in the future as the writing was on the wall we were getting close to near perfect compression in regards to the limits of human hearing.
I re-converted several of my favorite albums to Nero aac a couple years ago as I could not distinguish at the time with enough certainty Nero aac (@ 192kbit/s) vs flac using good headphones and my computer. Nero has also stopped development around that time but aac has continued to advance through Apple and Fraunhofer to where we are today.
So last night I ripped all my archived albums to 160kbs iTunes aac and just deleted the archives themselves from my hard drive saving huge chunk of GB's. One, today's codecs are basically transparent at that level for 99.9% of us and two, if I ever do want a full lossless copy of an album in the future for any reason, my high speed internet connection can get them in about 1-2 minutes each (since I purchased them once already I legally owned a copy? Technically though I don't have the hard version anymore).
Then went about enjoying some music for an hour that I had forgotten about.
Some links. More on aac: http://en.wikipedia.org/wiki/Advanced_Audio_Coding
more on Opus: http://en.wikipedia.org/wiki/Opus_%28audio_format%29
Here is more info on ABx testing and codecs as well as some results of scientific listening tests done by the audiophille community.
http://en.wikipedia.org/wiki/Codec_listening_test
Particularly the most recent test in 2014 where Opus is basically transparent to very discerning eats at a surprisingly low 96kbs bitrate. Apple aac also at 96kbs as well as Ogg and latest LAME mp3(needed higher around 136kbs) are not far behind. We are just about at end-game for lossy formats.
I got 5 out of 6 correct, and the one I missed was pretty near miss (I picked at random between 320 kbps and uncompressed sample). And these were quite clear choices, many times I just needed few seconds: 128 kbps sounded worse every single time, 320 kbps vs uncompressed was a bit harder, but still pretty noticeable if I paid attention.
It wouldn't probably make a big practical difference for a typical "background noise" listening, but it may have impact if you just want to sit back, relax and focus on music (lower bitrates for me sounded "muddled", losing details in high frequencies).
BTW I'm no audiophile, no special audio gear, just cheap (but decent) 9 EUR in-ear headphones plugged into a notebook.
The random average for a large number of people and 2 options would be getting 3 out of 6 (like a coin toss). But in the samples would be several 4, 5 and even 6 out of 6 too. In this case it's like 15/18, while is very good but still possible.
> there would not be much of a difference.
The fun poking is mostly about things like 192kHz sample rates and thousand dollar power cables. It's uncontroversial that people can hear mp3 artifacts.
48kHz and 24bits is what most audio is recorded at nowadays, then downsampled with aliasing for CD quality audio.
I don't really understand this, but I guess what it means is that the first sample of a digital signal gets from some place to some other place a ninety-six thousandth of a second more quickly. I'm not sure why that would matter. It's the time it takes sound to travel 4mm, which seems inconsequential, and the overall shape will be the same phase.
In many cases (like, if there's a computer involved anywhere), a high sample rate means you need higher latency to avoid the risk of underruns.
Also, if the sound processors are performing calculations in the idle time between samples (like calculating FIR filters or something like that) or in the idle processor time remaining after processing the sample, a higher sample rate will mean the calculation gets done faster (and is therefore audible faster). Else you'd change a setting and wait to hear it (and it would be noticeable perhaps), I guess?
Then a ninety-six thousandth of a second's worth of latency seems pretty irrelevant? Even if you were decreasing the latency by using a higher sample rate, which you aren't.
The signal coming out of a FIR filter will come out at the same time whatever the sampling rate. I guess it's conceivable, if you have no buffering whatsoever, that the very first sample will come out slightly quicker, but that is honestly irrelevant. The overall signal will have the same timing at either sample rate. Unless you've had to introduce more latency to cope with the demands of the higher sample rate.
What I see more and more online, is that people are falling into a pattern of poking fun of groups without really understanding the scientific, factual, or ideological basis for doing so. Furthermore, most people fall into the pattern after a semantics-free pattern match, quickly making a decision without substance. (One might suppose that the true priority in these situations is the opportunity to have fun at someone's expense, not the ideological or scientific issue at hand.)
When I was college aged, we called such jumping to conclusions "prejudice." One is coming prematurely to a conclusion, possibly contrary to a properly informed decision. Even in such a vaunted forum as HN, I see people proudly announcing how they have jumped to a conclusion based on signalling. How is this any different from a Mad Men character deciding another's credibility based on their alma mater and the cut of their jacket?
When it comes down to it, the "audiophile" set has myths and disinformation floating around within it mixed in with actual science. Note that this is true for any set of people derived from a shallow labeling, like "programmer."
Also, it's hard to know if I'm listening to a WAV of some 808 drum loop or an actual recording of a snare/cymbal encoded. The classical piano seemed easiest to spot the 128.
I listened through IEMs plugged into a Macbook Air in a quiet room and I wouldn't say it was easy.
The same is true of audio quality. I thought 128kbps MP3s were fine for years (I even had some MP2s somewhere) but now they sound like sludge.
I prefer to keep FLACs (lossless compression, so just like a WAV except smaller) myself, just so I can re-encode to my lossy format/bitrate of choice without incurring generational loss. 320kbps is probably clean enough but who cares, disk space is cheap.
But for listening, 320kbps MP3 is A-OK, I don't know whether or not I could ABX it but I don't hear enough artifacts to trigger me. For more situations where size matters, I use V0 or V1 (depending on what I'm playing it on and the type of music). At V1 I start hearing a few artifacts in tough passages, but it takes an enormous chunk out of the filesize relative to FLAC.
I used to have an iRiver H320 - with replacement firmware (RockBox) I could even play FLAC on it. I eventually replaced the 20GB HDD with a 16GB CompactFlash as a ghetto SSD, used to get like 20 hours of battery life on it. It could also do optical out, recorded in MP3 or WAV, and was super easy to change parts on. Great player for a hacker. I used it daily and hard for probably 7 years before the screws would no longer grab, I need to get it back out and see if I can get it going again.
1. I can reliably tell the difference between 128kbps mp3s and higher-quality files
2. I can not tell the difference between 320kbps CBR mp3s and uncompressed originals. (Or between ~256kbps VBR mp3s, 256kbps iTunes Plus mp4s, and uncompressed originals... though those weren't a part of this test)
It doesn't really apply to streaming, but I keep my music as lossless files despite that. It removes any questions of "Could this sound better?" and I can transcode to a device-appropiate format without compounding the quality loss (desktop has lossless files, laptop has a 256kbps copy, mobile devices have a 128kbps copy)
Modern songs have much compression and hard limiting which makes everything distorted (you can hear distortion on the Coldplay sample I think towards the beginning, unless I am mistaken?). Other offenders for brick wall limiting are most Chili Peppers albums, Paul McCartney's Memory Almost Full, and California Breed's (sadly) one and only album. Jason Bonham's snare gets lost for 75% of the album.
You'd hear nothing but BASS
Bass drums are also easy to pick out - if there isn't much bottom end then you know it has been compressed.
The a cappella version sounds like she's got a throaty cold on the MP3s. The SSSSSes sound different on MP3s too - it ssssoundsss like they've got a lissssppppp with MP3s.
You may find it easier to listen quietly because then your ears only pick up the highlights instead of getting fatigued with too much volume (where they will shut down after a while).
Rush Vapor Trails is the worst offender, perhaps?
In self-assured comments in forums? Very well.
In actual, properly conducted, A/B tests? Not so much.
Taking a gang of schlubs out of a shopping mall and playing audio, video, or still images to them and asking for an opinion could be OK in some contexts, like "Can you tolerate us putting this low-bit-rate codec in your mobile phone?" If the answer is "Huh, can't tell" then go ahead.
But is a JPG of an Ansel Adams print still a work of art if 90% of those same schulbs can't tell the difference?
But still, this is when comparing small sections of songs directly to one another, and being explicitly told there are quality differences. The 128 didn't sound bad, there's just some difference in transients and high frequency content (the hi-hats on the Jay Z track were an immediate giveaway for me).
Maybe a better test would be only having one audio track per song (instead of 3) and having to choose if it is 128/320/WAV. I wonder if anyone could distinguish the difference there reliably.
I've gotten a new phone 3 times in the past year (I'm clumsy...), and each time, I go a month or two with Spotify set to the default quality instead of "Extreme Quality", and I never notice until I listen to an album I know extremely well. WAV is great, but mp3 is still pretty good, even at 128kbps.
But more importantly, that test format preloads the tracks. In this one, some of the lossless tracks took several seconds to load for me, which completely killed its double-blindness.
Long story short, my takeaway is that perhaps it's not just about absolute levels of quality, but also what effect minor effects in bitrate can have on the underlying codec.
The quality on Sky used to be substantially better than DAB when I used it 10 years ago, but I don't know about it now. That said, I have Sky TV and the picture quality is variable. Most of the channels I watch are fine and all HD channels are very clear.
A chap at work listens to DAB over some big Questeds and the glitching and artefacting from Radio 2 (ugh I hate hearing that all day) is irritating (down to poor reception). Those Questeds sound great if proper audio is put through them though! It seems a waste to shove DAB Radio 2 and Jeremy Vine and his argument "show" through them.
(Well, technically, DAB+ would make things a ton better as it uses AAC instead, but progress is moving forward very slowly in rolling it out..)
I suspect this very much depends on how the analog recording was digitzed in the first place, if there was an analog recording to begin with. A sample from a CD is not the same as one made from a vinyl or a master tape.
Bottom line - there definitely is a difference, but in some cases it's hard to tell.
There is absolutely nothing wrong with preferring that! Nearly all consumer-oriented speakers/headphones do this, because there are other models specifically intended for monitoring use. However, it does make Bose equipment poorly suited to engineering-type applications such as recording work, comparing mp3 compression algorithms, etc.