Become a Site Supporter and Never see Ads again!

Author Topic: AES Paper: A Meta-Analysis of High Resolution Audio Perceptual Evaluation  (Read 2033 times)

0 Members and 1 Guest are viewing this topic.

Offline voltronic

  • Trade Count: (27)
  • Needs to get out more...
  • *****
  • Posts: 2403
  • Gender: Male
DPA 4061 | Line Audio CM3 | Naiant X-Q
Naiant PFAs | Shure FP24
Tascam DR-70D JWMod | Sony PCM-M10

Tascam DR-70D FAQ
Team Line Audio
Quote
I am hitting my head against the walls, but the walls are giving way.    ///    If a composer could say what he had to say in words he would not bother trying to say it in music.
- Gustav Mahler

Offline ~Jon Stoppable

  • Trade Count: (47)
  • Taperssection All-Star
  • ****
  • Posts: 1243
  • Gender: Male
    • Naiant Studio
This paragraph pretty much destroys any conclusion about speed:

Quote
In [56], test subjects generally did not perceive
a difference between DSD (64×44.1 kHz, 1 bit) and
DVD-A (176.4 kHz, 16 bit) in an ABX test, whereas [57]
showed a statistically significant discrimination between
PCM (192 kHz/24 bits) and DSD. However, in both cases,
high resolution audio formats are compared against each
other. Certainly in the first case, the null result does not
suggest that there would be a null result when discriminating
between CD quality and a higher resolution format. The
second case is intriguing, but closer inspection of the experimental
set-up revealed that the two formats were subject
to different processing, most notably, different filtering of
the low frequency content.

So the study where DSD and quad PCM are distinguished is discounted.  Let's stipulate that.  Thus, the author is arguing that any format of high datarate should perhaps be indistinguishable.  But if ultrasound is audible, then DSD would obviously sound different than quad PCM, because with DSD ultrasonic frequencies are filled with high-level noise (although not near-ultrasound, say below 24kHz--not a very good argument for DSD though).

Also, I've said it before, but I want to see a 44.1kHz vs. 48kHz study, because that will show perceptual difference of near-ultrasound.  If there is no difference between perception of 20kHz and 22kHz bandwidth (I think there could be in youngsters at least), then higher sample rates won't matter at all.

On another tack, in the highly trained group, so they can increase their perception to maybe 60% accuracy rather than the random 50% of the untrained.  What exactly does their hearing tell them?  We don't know if it's ultrasound or audible band artifacts of 20kHz filtering.  The author doesn't really need to know that though; if the fix is double-speed, then it doesn't matter if it's an audible band artifact or ultrasound.  Still, that means that many trained listeners can't identify high rate audio.  Not exactly the most powerful conclusion.  Compare that with something like a 1dB octave-band EQ shift; I'd expect nearly 100% of trained listeners to nail that.

Offline voltronic

  • Trade Count: (27)
  • Needs to get out more...
  • *****
  • Posts: 2403
  • Gender: Male
Not surprising that people couldn't distinguish between 192 and 176.4, as the ultrasound frequencies would be super-high.  48, 88.2, and 96 would be more relevant I think.  I would like to see DSD compared to those PCM rates.

If hearing ultra or near-ultrasound frequencies has anything to do with this, I think that the test subjects need to have a full range hearing exam first.
DPA 4061 | Line Audio CM3 | Naiant X-Q
Naiant PFAs | Shure FP24
Tascam DR-70D JWMod | Sony PCM-M10

Tascam DR-70D FAQ
Team Line Audio
Quote
I am hitting my head against the walls, but the walls are giving way.    ///    If a composer could say what he had to say in words he would not bother trying to say it in music.
- Gustav Mahler

Offline aaronji

  • Site Supporter
  • Trade Count: (6)
  • Needs to get out more...
  • *
  • Posts: 2249
Interesting read.  Viewed solely from a methodological perspective, this paper is appallingly bad (I am sure Dr. Reiss is a fine engineer, but he would have benefited greatly from taking on a statistician as a co-author).  The fundamental assumption underlying meta-analysis, that both the independent and dependent variables are fairly homogeneous, seems not to have been met at all and it just gets worse from there (I am happy to elaborate if anyone is interested, although it will probably put you rapidly to sleep)...

It is also interesting to note Reiss' inherent bias, which comes through in his writing in many places.  It is surprising that the reviewers (apparently) did not comment on that or the sometimes dubious decisions that arose from it.  Actually, I am curious what the peer-review process at AES entails; they mention a "review board", but I wonder who participates in that, as they don't explain it any further.  I can state with great certainty that they did not seek out an expert in the use of meta-analytical tools. 

As an historical aside, the appendix is hilarious, describing in excruciating detail methods that have been available since the late fifties (Mantel-Haenszel) or the mid-eighties (DerSimonian and Laird, which is what's in the Cochrane software he used) and are very well-known.

Offline ~Jon Stoppable

  • Trade Count: (47)
  • Taperssection All-Star
  • ****
  • Posts: 1243
  • Gender: Male
    • Naiant Studio
Was it a Dr.?  I thought it was a student paper!  No, I didn't read it that carefully :D

AES has always been a joke to me.  There are no real qualifications to join as far as I can see.

Offline voltronic

  • Trade Count: (27)
  • Needs to get out more...
  • *****
  • Posts: 2403
  • Gender: Male
Was it a Dr.?  I thought it was a student paper!  No, I didn't read it that carefully :D

AES has always been a joke to me.  There are no real qualifications to join as far as I can see.

Full disclosure: I posted this study after only reading the abstract and skimming through, hoping the more knowledgeable here would be able to cut through the meat of it.  So I really didn't read it carefully, nor was I endorsing the findings.

I looked into AES membership once.  The amateur choir I sing with hires a so-called professional recording company to cover our concerts.  One of their engineers said he'd invite me or whatever after I took an interest in the mics he was using.  But after seeing how completely incompetent he and the rest of his company are at recording a choir and orchestra, I lost interest.

It reminds me of this:
Quote
Please accept my resignation.  I don't want to belong to any club that will accept people like me as a member.
- Groucho Marx
DPA 4061 | Line Audio CM3 | Naiant X-Q
Naiant PFAs | Shure FP24
Tascam DR-70D JWMod | Sony PCM-M10

Tascam DR-70D FAQ
Team Line Audio
Quote
I am hitting my head against the walls, but the walls are giving way.    ///    If a composer could say what he had to say in words he would not bother trying to say it in music.
- Gustav Mahler

Offline ~Jon Stoppable

  • Trade Count: (47)
  • Taperssection All-Star
  • ****
  • Posts: 1243
  • Gender: Male
    • Naiant Studio
I mostly read the charts and the comments based on those.  I am already familiar with several of the more well-known studies.  The problem is any AES get-together is a mess; most of the attendees aren't engineers or scientists in any strict sense of those terms (me neither), and they have already made up their minds on just about every topic based on their subjective or anecdotal experience.

AES is useful for promulgating standards (although I think several of those suck, that's just my opinion), but that makes it more of a trade association rather than a scientific body.

Offline aaronji

  • Site Supporter
  • Trade Count: (6)
  • Needs to get out more...
  • *
  • Posts: 2249
I thought it was a student paper! 

Most students would probably do better.  Although many do like to demonstrate how much they learned along the way, even if the details aren't relevant to the manuscript at hand (see: detailed appendix to this paper)...

I did read this thing pretty carefully, as it covers a topic I am interested in and because it uses methods that I am very familiar with (scores of publications).  It is a mess, to the extent that I don't think the conclusions can be taken seriously.  Maybe they are correct, maybe not; the same study, properly implemented, would probably shine some light on the subject.  In fairness, though, science has become so specialized that it is basically impossible to do these things on your own (which is why I said he should have asked a statistician to help him). 

As for AES, I will take you guys at your word (I have no idea).  I wish more of the papers were open access, though...

Offline Gutbucket

  • record > listen > revise technique
  • Trade Count: (13)
  • Needs to get out more...
  • *****
  • Posts: 11537
  • Gender: Male
I wish more of the papers were open access, though...

^This.

The only reason I once considered joining was access to the papers, but then at some point they raised the fee rates even for members to access them and I lost interest.

I was not aware of any open access AES papers (except access to copies hosted by the writer themselves, or by some other organization for them outside of the AES) until seeing this, so I look forward to seeing if there is anything interesting in the other open access papers available through their site.
« Last Edit: July 19, 2016, 03:08:52 PM by Gutbucket »
volition > vibrations > voltages > numeric values | numeric values > voltages > vibrations > virtual teleportation time-machine experience

Offline Gutbucket

  • record > listen > revise technique
  • Trade Count: (13)
  • Needs to get out more...
  • *****
  • Posts: 11537
  • Gender: Male
AES search results for all "open access" papers- http://www.aes.org/e-lib/online/search.cfm?type=elib&title=&oa=yes
(organized by date of publication)
volition > vibrations > voltages > numeric values | numeric values > voltages > vibrations > virtual teleportation time-machine experience

Offline aaronji

  • Site Supporter
  • Trade Count: (6)
  • Needs to get out more...
  • *
  • Posts: 2249
^ Interesting pattern discernible in those open access publications:  once you get to the more modern ones (first couple of pages are from the early fifties), they are mostly from outside of the US.  A lot of the overseas public funding sources stipulate open access publication, which is funded by fees paid by the authors.  The US does not generally require this (to my knowledge), so this is not uncommon in a lot of journals.

Thanks for the list, by the way.  There are some interesting papers available.

Offline voltronic

  • Trade Count: (27)
  • Needs to get out more...
  • *****
  • Posts: 2403
  • Gender: Male
^ Interesting pattern discernible in those open access publications:  once you get to the more modern ones (first couple of pages are from the early fifties), they are mostly from outside of the US.  A lot of the overseas public funding sources stipulate open access publication, which is funded by fees paid by the authors.  The US does not generally require this (to my knowledge), so this is not uncommon in a lot of journals.

Thanks for the list, by the way.  There are some interesting papers available.

I may regret asking  ::)  but I'm curious as to why the methodology used by this researcher was so poor.  It sounds like you're someone who works in a field where you do this type of thing a lot, so I'd like to know what I as a layman am missing.  My research experience is limited to some papers and studies done as part of my Master's in music, and nothing nearly this involved.
DPA 4061 | Line Audio CM3 | Naiant X-Q
Naiant PFAs | Shure FP24
Tascam DR-70D JWMod | Sony PCM-M10

Tascam DR-70D FAQ
Team Line Audio
Quote
I am hitting my head against the walls, but the walls are giving way.    ///    If a composer could say what he had to say in words he would not bother trying to say it in music.
- Gustav Mahler

Offline aaronji

  • Site Supporter
  • Trade Count: (6)
  • Needs to get out more...
  • *
  • Posts: 2249
Well, there are quite a number of little things, but here are a few of the big ticket items (so to speak).  The first is that the independent variable (exposure, stimulus) and the dependent variable (outcome, response) should be relatively homogeneous.  For example, you could do a meta-analysis looking at the effects of statin use on LDL cholesterol levels.  Ideally, you would want to have all included studies using the same statin (say, simvastatin) and dosage in all participants and LDL measured in exactly the same way.  Outside of clinical trials, it is generally impossible to obtain that kind of data, though, so I think most reviewers would accept statin use versus LDL, even if there were several different medications and a couple of ways of measuring LDL (although they might very well ask for sub-group analysis stratifying on those).  Still quite homogeneous, since statins all work via the same pathway and LDL measurement variability is pretty consistent across measurement methods.  Reiss, in contrast, takes disparate outcomes and forces them into the same box and does the same with the exposures.  In some cases, he allows his own bias to influence how he does this ("for each trial, it was treated as a correct discrimination if the highest sample rate, 192 kHz, was ranked closer to “live” than the lowest sample rate, 44.1 kHz, and an incorrect discrimination if 44.1 kHz was ranked closer to “live” than 192 kHz").

Another major problem is that he considers proportions as means and analyzes them as such (this maybe gets into the statistical weeds a bit, but I will keep it brief).  In doing so, the methods he implements make a lot of distributional assumptions, particularly assumptions of normality.  The actual trial data, though, is not at all normal, it is binomially distributed (like coin flip data).  Any analysis needs to explicitly account for that and there are a number of methods for doing that (such as Stuart-Ord).  The R package "meta", which is freely available (as is the base R package), implements several approaches.  Incidentally, this criticism applies not only to the meta-analysis, but also to the binomial test panel in Table 2; the test is appropriate at the individual study level, but not for the aggregate.  Really, this issue is even more complicated, because the trials themselves are not independent.  There is correlation between a given subject's choices, so that 1 trial in 1000 people or 10 trials in 100 people or 1000 trials in 1 person are not the same thing, statistically.

Then there is the obvious publication bias problem seen in the funnel plot.  He explains this away, but omits mentioning a very plausible explanation, which is that supporters of higher sampling rates conducting these studies may shelf a study that doesn't fall in line with their expectations (i.e. fails to reject the null hypothesis).  This might not even be completely conscious on the researcher's part ("this is ongoing work, for which my sample size is currently insufficient").  Sensitivity analysis could help here (assuming he used an appropriate set of studies and statistical approach in the first place).

There are many other things, too, such as his curious, and curiously inconsistent, approach to multiple testing corrections or his unfortunate tendency to cite P-values as percentages.  Honestly, if I had to do a formal review of this, it would take hours.  The paper is poorly structured and Reiss' bias is all too obvious in many places.  Anyway, that should be enough (or more than!), but let me know if you are interested in more detail...

Offline voltronic

  • Trade Count: (27)
  • Needs to get out more...
  • *****
  • Posts: 2403
  • Gender: Male
Thanks, I understood a good part of that. ;)
DPA 4061 | Line Audio CM3 | Naiant X-Q
Naiant PFAs | Shure FP24
Tascam DR-70D JWMod | Sony PCM-M10

Tascam DR-70D FAQ
Team Line Audio
Quote
I am hitting my head against the walls, but the walls are giving way.    ///    If a composer could say what he had to say in words he would not bother trying to say it in music.
- Gustav Mahler

Offline joshr

  • Trade Count: (0)
  • Taperssection Newbie
  • *
  • Posts: 3
I'm the author of this paper.
I’ve avoided engaging in forum discussions, but Aaronji’s comments caught my eye, and I couldn’t help but respond.

- Full data, analysis methods, source code etc are available at https://code.soundsoftware.ac.uk/projects/hi-res-meta-analysis . I encourage anyone who is interested to perform their own analysis, and I will happily answer any questions. I expect that others may be more rigorous, or may uncover other interesting information that I overlooked. I’ll also try to answer any comments posted on the paper’s forum at https://secure.aes.org/forum/pubs/journal/?ID=591
- I consulted with statisticians and meta-analysis experts at various stages throughout the preparation of the paper. I would have liked a co-author with expertise in those areas, but the people I asked were unavailable.
- The Appendix was not included in the original submission, but was requested by one of the reviewers. I believe this request was correct, since the readers of the AES journal, including those who frequently apply statistical techniques to their data, are generally not familiar with meta-analysis and the techniques applied in that field.
- I’m aware of the importance of homogeneity, and the heterogeneity issues here are more serious than those that would typically be found in medical research, and a world apart from formal clinical trials. However, meta-analysis has been successfully applied to social and behavioural science research with far more heterogeneity problems than those seen here. Anyway, this is a judgement call. So the approach I took was to use all possible studies (for which I could do inverse variance analysis), and then do sensitivity or subgroup analysis on more homogeneous subsets of the data.
- bias. This made me laugh at first since in relation to this paper I’ve been accused of bias from all sides. Before beginning the study, I did not have a strong opinion either way as to whether differences could be perceived. But I could easily be fooling myself. So I committed to publishing all results, regardless of outcome. And again, I included all possible studies, even if I thought they were problematic, then did further analysis looking at alternative choices. I also decided that any choices regarding analysis or transformation of data would be made a priori, regardless of the result of that choice. However, I wrote the paper once all the analysis had been done, and so my writing style may reflect my knowledge of the conclusions.
- I agree that the work would have been improved by using an approach specific to binomial distributions. However, for much of the analysis, the normal approximation is justified. As for independence in the binomial test, under the null hypothesis every randomised trial would be uncorrelated, regardless of whether they involved the same participant or same study (think guessing a truly random coin toss). I also agree that the aggregate binomial test is not appropriate for meta-analysis. It was included only for completeness along with the binomial values for the individual studies in Section 2, and not used as part of the meta-analysis in Section 3.
- For King 2012 (the ‘closer to live’ study), it could have been either excluded it completely, treated higher preference rating as discrimination (which is fraught with issues) or treated closer to live as successful discrimination. Since the live feed was provided as a reference stimulus, similar to many other multistimulus evaluation studies, and the intention of the 192 kHz feed was to be ‘closer to live’ even if not perceived, this seemed a logical approach. Again, this decision was made a priori, in an attempt to minimize any of my own biases influencing the outcome.
- The studies were mainly from the audio engineering discipline and had a strong tendency to expressing and considering results (effect sizes) as means rather than proportions, and expressing probabilities as percentages. This is reflected in the paper, though better editing on my part would have resulted in more consistency with the notation of p values. I could have also performed sensitivity analysis where results were considered as odds ratios. But at some point, one has to stop looking at every variation and just submit the paper.
- The structure of the paper is in-line with the structure of most engineering papers (including IEEE). As such, it looks very different from the structure of papers in medical journals and other places where a lot of meta-analysis is published.
- The standard explanation for the publication bias problem was mentioned several times. The beginning of Section 3.6 first presents it. Figure 3 shows that the apparent evidence of publication bias from the funnel plot mostly goes away when subgrouping is applied. However, it then goes on to state "publication bias may still be a factor" and in Conclusion, "still a potential for reporting bias. That is, smaller studies that did not show an ability to discriminate high resolution content may not have been published."

 

RSS | Mobile
Page created in 0.195 seconds with 42 queries.
© 2002-2017 Taperssection.com
Powered by SMF
Website Design by Foxtrot Media, Inc., a Baltimore Website Company