wow! quite the pissing match we have going on here! Here is my opinion on comps. 2pr same brand mics are not close enough for a comp. mics sent through a consumer grade splitter not good enough. 2 pr identical mic cables not good enough. Setting levels by eye or via software to peak not good enough, living room taping not nearly good enough unless you plan on taping in your living room.
Here is what would make the test with all of these variables completely scientific and valid. Switch the variables! Run a couple of songs with mic a, and cable a into preamp a, and mic b cable b into preamp b, then after a couple of songs switch mic and cable a to pre b, then after a couple of songs mic a cable b into pre a and so on. There are 3 variables (4 if you include positioning which could never be repeated exactly, but could more easily be lumped into the mic variable)
If what you like about the comp is consistent across the board it is a completely valid and scientific comp IMO. If your favorite changes with the cable/mic switches then we know that the comp is tainted. It would be my guess that only thoes with very nice playback systems and impressive ears would hear much difference between the mic cable switches but it would be proof positive of the difference between the pre's.
Switching mic cables could potentially reposition the mics so I would say forgo that step as long as the cables were the same length and type. Another good wrinkle would be to use another brand of mic during the same show so that we could see the difference with different flavors of mics. Radical differences like ld's vs sd's, or the Neumans vs a very flat mic for example, or maybe just a very popular mic like the c-4's or whatever Chris feels is a common mic for his sales demographic. Using a single pair of mics and a splitter would be even better but I say still switch the outputs because the splitter or a cable in the chain could have some impact. As for setting levels, IMO one of the biggest differences between quality and average preamps is the transient response. One pre may record a far greater dynamic range than the other and it will show a peak far higher over the "meat" of the music than a lesser pre. When listening, each person would have to be able to make the determination between the two even though one seemed louder (this would probably be the lesser pre... in this regard anyway) and one would have to be able to make determinations despite this difference. Maybe two sets should be made for people to listen to... one with both sources normalized to a set rms level so they sound the same, and one set untouched.
Doing all of this would provide a comp that is real world, scientific, and easy to compare because they will seem at similar volume and utilize the gain settings representing the meat of the pre.
Take good notes, and snap pictures of the setup and IMO nobody could bitch about validity.
my .02
Matt