Concluding The Great MP3 Bitrate Experiment

And now for the dramatic conclusion to The Great MP3 Bitrate Experiment you've all been waiting for! The actual bitrates of each audio sample are revealed below, along with how many times each was clicked per the goo.gl URL shortener stats between Thursday, June 21, and Tuesday, June 26.

Limburger / ~160kbps VBR / 10,265 Cheddar / 320kbps CBR / 7,183 Gouda / raw CD / 6,159 Brie / ~192kbps VBR / 5,508 Feta / 128kbps CBR / 5,567

During that six-day period, my overall Amazon CloudFront and S3 bill for these downloaded audio samples was $US103.72 for 800GB of data, across 200,000 requests.

Based on the raw click stats, it looks like a bunch of folks clicked on the first and second files, then lost interest. Probably because of, y'know, Starship. Still, it's encouraging to note that the last two files were both clicked about 5,500 times for those that toughed their way out to the very end. Of those listeners, 3512 went on to contribute results. Not bad at all! I mean, considering I made everyone listen to what some people consider to be one of the worst "rock" songs of all time. You guys are troopers, taking one in the ear for the team in the name of science. That's what I admire about you.

I belatedly realised after creating this experiment that there was an easy way to cheat. Simply compress all the samples with FLAC, then sort by filesize.

The higher the bitrate, apparently, the less compressible the audio files are with lossless FLAC compression. It's a small difference in absolute file size, but it's enough to sort exactly with quality. At least you can independently verify that I wasn't tricking anyone in this experiment; each sample was indeed different, and the bitrates are what I said they were.

But you guys and gals wouldn't do that, because you aren't dirty, filthy cheaters, right? Of course not. Let's go over the actual results. Remember each sample was ranked in a simple web form from 1 to 5, where 1 is worst quality, and 5 is highest quality.

The summary statistics for the 3512 data points:

(If you'd like to perform more detailed statistical analysis, download the Excel 2010 spreadsheet with all the data and have at it.)

Even without busting out hard-core statistics, I think it's clear from the basic summary statistics graph that only one audio sample here was discernably different than the rest -- the 128kbps CBR. And by different I mean "audibly worse". I've maintained for a long, long time that typical 128kbps MP3s are not acceptable quality. Even for the worst song ever. So I guess we can consider this yet another blind listening test proving that point. Give us VBR at an average bitrate higher than 128kbps, or give us death!

But what about the claim that people with dog ears can hear the difference between the higher bitrate MP3 samples? Well, first off, it's incredibly strange that the first sample — encoded at a mere 160kbps — does better on average than everything else. I think it's got to be bias from appearing first in the list of audio samples. It's kind of an outlier here for no good reason, so we have to almost throw it out. More fuel for the argument that people can't hear a difference at bitrates above 128kbps, and even if they do, they're probably imagining it. If we didn't throw out this result, we'd have to conclude that the 160kbps sample was somehow superior to the raw CD audio, which is … clearly insane.

Running T-Test and Analysis of Variance (it's in the spreadsheet) on the non-insane results, I can confirm that the 128kbps CBR sample is lower quality with an extremely high degree of statistical confidence. Beyond that, as you'd expect, nobody can hear the difference between a 320kbps CBR audio file and the CD. And the 192kbps VBR results have a barely statistically significant difference versus the raw CD audio at the 95 per cent confidence level. I'm talking absolutely wafer thin here.

Anyway, between the anomalous 160kbps result and the blink-and-you'll-miss-it statistical difference between the 192kbps result and the raw CD audio, I'm comfortable calling this one as I originally saw it. The data from this experiment confirms what I thought all along: for pure listening, the LAME defaults of 192kbps variable bit rate encoding do indeed provide a safe, optimal aural bang for the byte -- even dogs won't be able to hear the difference between 192kbps VBR MP3 tracks and the original CD.

Concluding the Great MP3 Bitrate Experiment [Coding Horror]

Jeff Atwood is a blogger at Coding Horror and cofounder of Stack Overflow. Follow him on Twitter @codinghorror.


Comments

    Replies saying:
    You guys are idiots, of course you can hear the difference, I have
    * better ears
    * $1000 cans
    * gold plated cables

    in 3, 2, 1..

    You left out my main one.... my cables have arrows that tell the audio which way to move! Without those arrows, how would they know if they were coming or going?!

    I can hear the difference. I have:
    * big puppy-dog ears that could hear a pin dropping on the moon,
    * $1000000 cans, and
    * Plutonium plated cables.

    (see what I did there???)

    "people can’t hear a difference at bitrates above 128kbp"
    That is absolutely ludicrous. Any one with a normal ear drum will notice a hi-hat at 128kbp sounds like a 'F' sound instead of an 'S' sound. which is unbearable IMO.

    I cant stand anything below 320kbps

      You are a special, precious flower with superhuman qualities in all areas of life. Please continue to keep us up to date on how awesome you are.

    All of my files are 192kbps, can't even tell the difference between those and 320kbps on my expensive car stereo, get more distortion from bad recording than from the file encoding. Below 192kbps, there is a difference on that stereo system so I replaced all 128kbps files I had. 192 really is the sweet-spot.

    Would have been good to be involved.
    I totally agree with the above experiement. I only encode at 192 for this reason, everyone back in the day did 128, cause it was good for size, especially when we were running 300,000MB hdd's. But as soon as space was not a problem, I went straight for 192. I feel I can tell the difference between 160cbr and 192, so I always go 192. But much above 192 was negotiable and feel was a waste of space. I definately feel the difference is in the high end of things, so maybe people that cant tell the difference are using an EQ with a low treblie end? Maybe those that feel they need 320 dont use any bass?

    Yo, not sure that ditching out of hand the first result because it doesn't conform to your expectations is good statistical analysis.

    If you didn't ignore that result it would seem to point to people generally being unable to discern a difference above 160kbps rather than 128 as was written in the article, which is at least something worth not automatically dismissing.

    Then again I'm not a statistician so it's not like I know what I'm talking about.

    One factor not considered was the choice of song for the experiment. I would have selected a song containing a lot of information, rich in detail where differences in bit rate may have been more evident. Secondly, presenting the selection in a different random order each time the experiment was accessed. Thirdly, cut the selection to 3 options - reducing song fatigue.

    Why preserve data at a lesser quality than the original?

    Ever tried playing an MP3 on a big enough sound system (e.g. what DJ's will play on in medium-large night clubs) - often it is not what you can hear which makes the big difference between 192-320MP3 & Lossless, it's what you feel as well.

    I can certainly hear a difference between 192 and lossless on a decent pair of studio monitors.. Admittedly, 320 is often less of a difference; however depending on the song sometimes the Mp3 does not feel as deep and earthy.

      Simple answer: most stuff won't play lossless files (iDevices and ALAC excepted). If I want to stream music around my house; carry it with me; share it with someone else; MP3 is still the only universally-accepted standard. I hope that one day everything will be able to play FLAC, but until then I'll just have to manage two libraries.

    I have all my music stored as FLAC, but transcoded to [email protected] MP3 for my portable devices. I did my own testing on a reference track of my choosing and while 192kbps would have done the trick, an extra 64kbps wasn't going to break the storage bank and ensures that any subtlety I could possibly hear ought to be represented in the resulting files.

    It would have been good for people to select the listening equipment and differntiate that in results (I can hear jack all on my computer using high quality headphones because of the sound card output quality. On my high-end stereo difference become a lot more apparent).

    I don't think this experiment answers everything. I found the quality of the samples to be *extremely* similar when I listened to the point I wouldn't care about 128kb vs 320kb. I had to listen very carefully to tell the difference. I was just using some cheapy Sony MDR-G56 headphones on my work PC.

    I rated them as:
    3 out of 5 = ~160kbps VBR
    4 out of 5 = 320kbps CBR
    5 out of 5 = raw CD
    4 out of 5 = ~192kbps VBR
    3 out of 5 = 128kbps CBR

    So I couldn't tell the difference between 128 and 160 nor 192 vs 320 but I guess you could say that if I tried really hard - I could tell generally which was better.

    The statistical analysis is weird I think - the survey needed to ask "Do you believe you can tell the difference between different MP3 / CD audio samples? [Yes/No/Unsure]"

    If a person answers Yes then that marks them as a 'dog ears' sample. If a person answers the other two then they are the control group.

    Right now his stats may contain a tiny group of dog eared people who got everything right but because they mixed in with the tone deaf and normal listeners in the statistical sample - the dog eared people are probably blending into the background when the T-test and so other statistical studies are done.

    We need to see how much better the Dog Eared group is compared to the normal population.

    All digital music is uninvolving. My ears refuse to identify between bitrates because music recorded 'in little sections' or 'discretely' is alw ay s go nna be mis sing stu ff. Grab yourself a turntable and a cartridge with a freq response up to 45kHz. Your ears will be awoken from their iTurnal slumber. acontinuousrecordingprocessfromtape.

      Don't the majority of recording studios these days use digital recording equipment?

      Or are just talking about listening to olden days tunes where the studio originals were recorded in analog.

    I did some research on this for a sound assignment at uni.

    There's essentially no audible difference between CBR and VBR - variable bitrate will just use less bits where less are required - for instance silence. Complicated sections use the bitrate stated and use more space (obviously). so there's nothing wrong with VBR.

    There was a large survey that found 190kbps to be the area where most AVERAGE listeners could tell the difference between lossless and mp3 (@190kbps). Audiophiles will probably be able to tell the difference between 320 and lossless. But for the most of us, 224 is safe, 320 is more than safe.

    In my opinion lossless is not worth the size for your audio library. Considering if you have the mp3 and no WAV to compare it to, you don't know what you're missing out on anyway.

    All is relevant to the listener and whether they can even tell the difference anyway. I try to keep my library in 320kbps. FLAC/WAV/AIF etc aren't very friendly formats among the car (USB) stereo, ipods etc.

Join the discussion!