i think what chris is really getting at is that if people are going to making money on modifying gear or selling boutique cables(referencing the hydra/van de haul thread), their tests should be bullet proof to ensure validity.
HOWEVER......
all we need to do is listen and make the decision ourselves. since this is a board about taping, in-field comps are genuinely good for us. I would still prefer that modified gear or boutique cables be thoroughly tested with the results and specs published.
I hardly ever get a chance to make and sell cables anymore and I've stopped doing mods, so hopefully people won't take this as some sort of way-biased comment, but:
I don't agree with you here. People on the board trying to make mods available or sell good cables at a good cost do not have the resources to make perfectly scientifically valid comparisons. And they almost certainly will not have the resources to publish any kind of meaningful specs. I can't get that level of information from a huge company like Monster Cable, why should I expect it from someone here?
As Brian Ska says, try to make the tests as valid as possible, let people know how they were done, and let people make their own judgements. Easier to make controlled living room tests, but people often like to hear tests done in the field -- meaning of PA recordings, so having those comps available makes sense. I'd rather 2 sets of high end mics be used rather than splitter cables and the ensuing effects on impedence, etc, but that's just me.
Not perfect, but still better than what you get from Monster Cable. Use the comps, seller reputation, word of mouth, feedback on ts.com and make a decision on whether to buy. Plus borrow a set of cables or a modded recorder and try them out for yourself. I know I've got a set of loaner cables somewhere floating around as part of the gear loaner library for people to try out, and I think others have done the same.
I think whatever comps we can get are a nice bonus, but it seems unrealistic to me to think we should have access to the results of scientifically valid tests and published specs, esp when you can't get the same from major cable making corporations.