codec listening test#Results
{{Short description|Scientific study designed to compare two or more lossy audio codecs}}
A codec listening test is a scientific study designed to compare two or more lossy audio codecs, usually with respect to perceived fidelity or compression efficiency.
Most tests take the form of a double-blind comparison. Commonly used methods are known as "ABX" or "ABC/HR" or "MUSHRA". There are various software packages available for individuals to perform this type of testing themselves with minimal assistance.
Testing methods
= ABX test =
{{main|ABX test}}
In an ABX test, the listener has to identify an unknown sample X as being A or B, with A (usually the original) and B (usually the encoded version) available for reference. The outcome of a test must be statistically significant. This setup ensures that the listener is not biased by their expectations, and that the outcome is not likely to be the result of chance. If sample X cannot be determined reliably with a low p-value in a predetermined number of trials, then the null hypothesis cannot be rejected and it cannot be proved that there is a perceptible difference between samples A and B. This usually indicates that the encoded version will actually be transparent to the listener.
= ABC/HR test =
In an ABC/HR test, C is the original which is always available for reference. A and B are the original and the encoded version in randomized order. The listener must first distinguish the encoded version from the original (which is the Hidden Reference that the "HR" in ABC/HR stands for), prior to assigning a score as a subjective judgment of the quality. Different encoded versions can be compared against each other using these scores.
= MUSHRA =
{{main|MUSHRA}}
In MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor), the listener is presented with the reference (labeled as such), a certain number of test samples, a hidden version of the reference and one or more anchors. The purpose of the anchor(s) is to make the scale be closer to an "absolute scale", making sure that minor artifacts are not rated as having very bad quality.
Results
Many double-blind music listening tests have been carried out. The following table lists the results of several listening tests that have been published online. To obtain meaningful results, listening tests must compare codecs' performance at similar or identical bitrates, since the audio quality produced by any lossy encoder will be trivially improved by increasing the bitrate. If listeners cannot consistently distinguish a lossy encoder's output from the uncompressed original audio, then it may be concluded that the codec has achieved transparency.
Popular formats compared in these tests include MP3, AAC (and extensions), Vorbis, Musepack, and WMA. The RealAudio Gecko, ATRAC3, QDesign, and mp3PRO formats appear in some tests, despite much lower adoption {{as of|2007|lc=on}}. Many encoder and decoder implementations (both proprietary and open source) exist for some formats, such as MP3, which is the oldest and best-known format still in widespread use today.
| class="wikitable sortable" |
| style="text-align: center;"
! style = "font-size: 84%" | Source ! style = "font-size: 84%" | Dates ! style = "font-size: 84%" | Formats ! style = "font-size: 84%" | Bitrate (kbit/s) ! style="width:24em;" | Codecs ! style = "font-size: 84%" class=unsortable | Musical genres ! style = "font-size: 84%" | Samples ! style = "font-size: 84%" | Listeners ! style = "font-size: 92%" |Best Result ! style="width:19em;" class=unsortable | Comments |
| style="text-align: center;"
| [https://web.archive.org/web/20110517111522/http://ff123.net/128tests.html ff123] | style = "font-size: 84%" | 2001 | multiple | ~128 | style = "font-size: 92%" |
| | 1 | 16 | style = "font-size: 92%" | Musepack and AAC | |
| style="text-align: center;"
| [https://web.archive.org/web/20110517111516/http://ff123.net/128test/instruct.html ff123] | style = "font-size: 84%" | 2001 October - 2002 January | multiple | ~128 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 3 | 25-28 | style = "font-size: 92%" | Musepack | |
| style="text-align: center;"
| [https://web.archive.org/web/20070918101411/http://ff123.net/64test/results.html ff123] | style = "font-size: 84%" | 2002 July | multiple | ~64 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 12 | 24-41 | style = "font-size: 92%" | mp3PRO | style = "font-size: 84%" | Both Vorbis variants were a close second. |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/AAC_at_128kbps_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2003 June | AAC | 128 CBR | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 10 | 11-18 | style = "font-size: 92%" | QuickTime | |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/128kbps_Extension_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2003 July | multiple | ~128 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 12 | 14-24 | style = "font-size: 92%" | Musepack | style = "font-size: 84%" | AAC, WMA, and Vorbis tied for close second |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/64kbps_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2003 September | multiple | ~64 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 12 | 30-43 | style = "font-size: 92%" | Nero | style = "font-size: 84%" | This test showed that listeners preferred 128 kbit/s MP3 audio encoded by LAME to all the tested codecs at 64 kbit/s, with greater than 99% confidence: "No codec delivers the marketing plot [sic] of same quality as MP3 at half the bitrates." |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/MP3_128kbps_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2004 January | MP3 | ~128 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 12 | 11-22 | style = "font-size: 92%" | LAME | style = "font-size: 84%" | The author noted that the results may have been affected by the use of an outdated version of the Xing encoder and non-optimal settings for ITunes. |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/AAC_at_128kbps_v2_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2004 February | AAC | ~128 | style = "font-size: 92%" |
| Various | 12 | 19-29 | style = "font-size: 92%" | iTunes | style = "font-size: 84%" | Open-source FAAC codec improved greatly since previous test |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/Multiformat_128kbps_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2004 May | multiple | ~128 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 18 | 12-27 | style = "font-size: 92%" | aoTuV (Vorbis) and Musepack | |
| style="text-align: center;"
| [http://listening-tests.freetzi.com/html/32kbps_public_listening_test_results.htm Roberto Amorim] | style = "font-size: 84%" | 2004 June | multiple | 32 CBR | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 18 | 47-77 | style = "font-size: 92%" | Nero | |
| style="text-align: center;"
| style = "font-size: 84%" | [https://hydrogenaud.io/index.php/topic,23355.0.html HydrogenAudio user "guruboolez"] | style = "font-size: 84%" | 2004 July | multiple | ~175 | style = "font-size: 92%" |
| 18 | 1 | style = "font-size: 92%" | Musepack | |
| style="text-align: center;"
| style = "font-size: 84%" | [https://hydrogenaud.io/index.php/topic,36465.0.html HydrogenAudio user "guruboolez"] | style = "font-size: 84%" | 2005 August | multiple | ~180 | style = "font-size: 92%" |
| 18 | 1 | style = "font-size: 92%" | aoTuV (Vorbis) | style = "font-size: 84%" | The author reflects on substantial improvements in Vorbis encoding since his previous test (above): "Vorbis is now –thanks to Aoyumi [creator of aoTuV]– an excellent audio format for 180 kbit/s encodings (and classical music)." |
| style="text-align: center;"
| style = "font-size: 84%" | [http://forum.hardware.fr/hfr/VideoSon/Traitement-Audio/mp3-aac-ogg-sujet_84950_1.htm gURuBoOleZZ] {{in lang|fr}} | style = "font-size: 84%" | 2005 August | multiple | ~96 | style = "font-size: 92%" |
| Classic, various | style = "font-size: 84%" | 150 classical, 35 various | 1 | style = "font-size: 92%" | aoTuV and AAC tied (classical), aoTuV (various) | style = "font-size: 84%" | The author selected each participating encoder by pitting multiple encoders against one another in an initial "Darwinian phase." For example, LAME was chosen as the representative MP3 encoder because it clearly outperformed four other MP3 encoders on a subset of the full sample corpus. |
| style="text-align: center;"
| style = "font-size: 84%" | [http://listening-tests.hydrogenaud.io/sebastian/mf-128-1/results.htm Sebastian Mares] | style = "font-size: 84%" | 2005 December | multiple | ~140 (nominal 128) | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 18 | 18-30 | style = "font-size: 92%" | 4-way tie (all except Shine) | style = "font-size: 84%" | "I think this test shows that with the current encoders, the quality at 128 kbit/s is very good... It's time to move to bitrates like 96 kbit/s or even lower (64 kbit/s)." |
| style="text-align: center;"
| style = "font-size: 84%" | [http://www.mp3-tech.org/tests/aac_48/results.html Mp3-tech.org] | style = "font-size: 84%" | 2006 March | AAC | 48 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 18 | 10-20 | style = "font-size: 92%" | 5-way tie | style = "font-size: 84%" | "... it seems that overall, plain HE-AAC might be better than HE-AAC v2 at this bitrate, but a lot more samples would be needed to be able to draw definitive conclusions regarding this. |
| style="text-align: center;"
| style = "font-size: 84%" | [http://listening-tests.hydrogenaud.io/sebastian/mf-48-1/results.htm Sebastian Mares] | style = "font-size: 84%" | 2006 November | multiple | ~48 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 20 | 22-34 | style = "font-size: 92%" | Nero | style = "font-size: 84%" | WMA Professional and aoTuV tied for second |
| style="text-align: center;"
| style = "font-size: 84%" | [http://listening-tests.hydrogenaud.io/sebastian/mf-64-1/results.htm Sebastian Mares] | style = "font-size: 84%" | 2007 July | multiple | ~64 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 18 | 21-33 | style = "font-size: 92%" | Nero Digital and WMA Professional | |
| style="text-align: center;"
| style = "font-size: 84%" | [http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm Sebastian Mares] | style = "font-size: 84%" | 2008 October | MP3 | ~128 | style="font-size: 92%" |
| style = "font-size: 84%" | Various | 14 | 26-39 | style = "font-size: 92%" | 5-way tie | style = "font-size: 84%" | "The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test." Also notes that Fraunhofer and Helix codecs are several times faster at encoding than LAME, although virtually identical in terms of perceived audio quality. |
| style="text-align: center;"
| style = "font-size: 84%" | [http://listening-tests.hydrogenaud.io/igorc/Public%20Multiformat%20Listening%20Test%20@%2064kbps.htm HydrogenAudio user IgorC (March/April 2011)] | style = "font-size: 84%" | 2011 March | multiple | ~64 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 30 | 25-13 | style="font-size:92%" | CELT / Opus | style = "font-size: 84%" | In [http://listening-tests.hydrogenaud.io/igorc/results.html results], CELT is referred to as Opus, its name when later standardized. |
| style="text-align: center;"
| style = "font-size: 84%" | [http://listening-tests.hydrogenaud.io/igorc/aac-96-a/index.htm HydrogenAudio user IgorC (July - August 2011)] | style = "font-size: 84%" | 2011 July/August | LC-AAC | ~96 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 20 | 25 | style = "font-size: 92%" | Apple QuickTime | |
| style="text-align: center;"
| style = "font-size: 84%" | [https://hydrogenaud.io/index.php/topic,100896.0.html HydrogenAudio user "Kamedo2"] | style = "font-size: 84%" | 2013 May | MP3 | ~224 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 25 | 1 | style = "font-size: 92%" | 4-way tie | style = "font-size: 84%" | Most impairment grades rated between 4 (perceptible but not annoying) and 5 (imperceptible). Both speech samples transparent (p<0.02) except for the low anchor. |
| style="text-align: center;"
| style = "font-size: 84%"| [http://listening-test.coresv.net/ HydrogenAudio user Kamedo2 (July/September 2014)] | style = "font-size: 84%" | 2014 July - September | multiple | ~96 | style = "font-size: 92%" |
| style = "font-size: 84%" | Various | 40 | 33 | style = "font-size: 92%" | Opus | style = "font-size: 84%" | In [http://listening-test.coresv.net/results.htm results] Opus is clear winner, Apple AAC is second, Ogg Vorbis and higher-bitrate LAME MP3 are statistically tied in joint third place. FAAC, known to be inferior in advance, was used to discard bad results and as quality scale anchor. |
| style="text-align: center;"
| style = "font-size: 84%"|Cunningham and McGregor | style = "font-size: 84%" | 2019 February | multiple | 192 - 1411 | style = "font-size: 92%" |
| style = "font-size: 84%" | Pop | 10 | 100 | style = "font-size: 92%" | 5-way tie (WAV, MP3, AAC, ACER HQ, ACER MQ) | style = "font-size: 84%" | Participants reported no perceived differences between the uncompressed, MP3, AAC, ACER high quality, and ACER medium quality compressed audio in terms of noise and distortions but that the ACER low quality format was perceived as being of lower quality. However, in terms of participants’ perceptions of the stereo field, all formats under test performed as well as each other, with no statistically significant differences.{{cite journal |last1=Cunningham |first1=Stuart |last2=McGregor |first2=Iain |title=Subjective Evaluation of Music Compressed with the ACER Codec Compared to AAC, MP3, and Uncompressed PCM |journal=International Journal of Digital Multimedia Broadcasting |volume=2019 |pages=1–16 |date=2019 |language=en|doi=10.1155/2019/8265301 |doi-access=free }} 50px Material was copied from this source, which is available under a [https://creativecommons.org/licenses/by/4.0/ Creative Commons Attribution 4.0 International License]. |
| style="text-align: center;"
! style = "font-size: 84%" | Source ! style = "font-size: 84%" | Dates ! style = "font-size: 84%" | Formats ! style = "font-size: 84%" | Bitrate (kbit/s) ! Codecs ! style = "font-size: 84%" | Musical genres ! style = "font-size: 84%" | Samples ! style = "font-size: 84%" | Listeners ! style = "font-size: 92%" |Best Result ! Comments |
See also
References
External links
- [https://hydrogenaud.io/ Hydrogenaudio] - Community audiophile site, host of most non-commercial ABX testing