^ First of all, welcome to taperssection and thanks for coming in and discussing this with us. To be honest, I was initially very surprised to see you post in this little backwater of the web, catering to the practitioners of a pretty uncommon hobby, but I am fairly certain I know how you arrived here on further reflection. At any rate, I would like to respond to a couple of your comments on my more major criticisms (the presence or absence of the appendix, for example, is immaterial in the end).
- I’m aware of the importance of homogeneity, and the heterogeneity issues here are more serious than those that would typically be found in medical research, and a world apart from formal clinical trials. However, meta-analysis has been successfully applied to social and behavioural science research with far more heterogeneity problems than those seen here. Anyway, this is a judgement call. So the approach I took was to use all possible studies (for which I could do inverse variance analysis), and then do sensitivity or subgroup analysis on more homogeneous subsets of the data.
With respect to the bolded part, what does "successfully" mean? Obtained a P
-value? Published a paper? Generated a useful result that led to downstream hypotheses that were also tested successfully? Settled an open debate? Whatever that defintion, though, do you think your work should fall into the category of "squishy" science (like a lot of social and behavioural science)? I always thought of engineering as "hard" science, with experiments conducted rigorously and in the most methodologically proper way possible. I am sorry, but "others did it worse!" is not a valid rebuttal of this criticism, which, in my mind, completely undermines the entire paper. You are right that, in the end, it is a series of judgement calls, but others can freely interpret the merit of the work based on their assessment of the quality of those judgements.
- I agree that the work would have been improved by using an approach specific to binomial distributions. However, for much of the analysis, the normal approximation is justified. As for independence in the binomial test, under the null hypothesis every randomised trial would be uncorrelated, regardless of whether they involved the same participant or same study (think guessing a truly random coin toss). I also agree that the aggregate binomial test is not appropriate for meta-analysis. It was included only for completeness along with the binomial values for the individual studies in Section 2, and not used as part of the meta-analysis in Section 3.
The normal approximation may be justified, particularly for large numbers, but I think you need to show that. It is kind of beside the point, though. Why make those additional, potentially spurious, assumptions when it is easy to implement the correct analysis, modelled on the correct distribution, in freely available software? With respect to the aggregate binomial analysis being included for "completeness", wouldn't it have been more complete to actually put the correct estimate in there? The intra-individual trials are not like coin flips, in my opinion; there is a discrete set of perceptual apparatus that is unique to each individual that causes correlation between that individual's observations. If such correlations did not exist, nobody would ever score higher (or lower) than 50% in a sufficiently large number of trials.
With respect to publication bias, I never said you didn't consider it, only that you never mention, specifically, the implication about the type of study that is not reported based on that funnel plot. In any event, that is a lesser concern for me than the above. I certainly appreciate your comments here, and I hope you understand I am not trying to be a dick in anyway (this, after all, is the nature of scientific discourse), but your rebuttal doesn't much impact my previous assessment...
While you are here, on a somewhat related topic, can you comment on the Journal's review policy? The website says there is a "review board". Who comprises that board? How large is it? Do all reviewers come from this board or are outside experts brought in?
Thanks, I understood a good part of that.
Sorry about that! I'll try to make it a little more obtuse next time; maybe toss in some formulas...