Topic: Taperssection - product listening comparisons *gold standard*

[digifish_music]

Aaronji asked a good question and I thought it deserved it's own topic...perhaps we can come up with a gold standard for product comparisons, and how some of the criteria may be achieved?

All 'sake oil' is 'essential oil' until it fails a well constructed blind listening test

digifish

Funny, when I made the "snake oil" crack, I was thinking of this thread http://taperssection.com/index.php/topic,109703.0.html, so I didn't really get your response.  Then I read the FR2LE thread, and it all made sense...

I am curious what people would consider constitutes a "well constructed blind listening test", though.  Seems like it would be difficult to simulate a concert setting at home and even more difficult to run a controlled test at a show.  Any thoughts?

A couple of basic things (all intended to improve listener sensitivity to differences)...

1. Levels need to be matched. Louder almost always sound better. The threshold for loudness differences is in the order of 0.5 dB (under the best listening conditions in an A/B back-to-back change. It rises to about 1.5 dB at 1 minute separation between the examples, again under ideal listening conditions). In the real world 1 dB is fairly close to the just noticeable limit.

2. The samples need to be spliced back-to-back with an instant (click free) transition between the recordings. The duration of the selections should ideally be around 5 seconds (however this is open to debate and will depend on the qualities being assessed. Generally, short selections A/B no gap usually produce the most sensitive discriminations).

3. The listener should be allowed repeated presentations of each comparison paring until they make their decision.

4. There should be an objective response indicator. That is they should be asked to choose based on preference - either 1st or 2nd half, or pick the modded one or whatever. But they should be given a A/B 'forced choice'.

5. The number of comparisons should exclude chance from playing a significant role in the outcome. It's typical to shoot for 95% confidence in the perceptual world. So that means a performance of ~7 consistent identifications out of 8. Ideally 10 randomized A/B or B/A comparisons would make a good standard.

5. The comparison should be done blind. That is the listener should not know which example they are listening to on any given session.

6. No feedback during the assessment session should be given. The internet makes this easy as you can download a file and listen to it without seeing the experimenter or interacting with them.

7 The recordings should be simultaneous from the same set of mics. Alternatively, the same the same microphones used in the same position recording the exact same sound. That is why the recording of a Hi-Fi source, ticking clocks, test signals or whatever is a great paradigm.

Note: One thing you should not do is compare A vs B with the knowledge of what you are listening to. Perception is just too influenced by expectation.

digifish

[Church-Audio]

Yeah but it must be a live band not a recording because the sound stage is not the same.

[it-goes-to-eleven]

Yeah but it must be a live band not a recording because the sound stage is not the same.

Whoa, we agree!

[Church-Audio]

Yeah but it must be a live band not a recording because the sound stage is not the same.

Whoa, we agree!

Sorry I was joking 

[digifish_music]

Yeah but it must be a live band not a recording because the sound stage is not the same.

In this case point 7 requires some compromise, options may be ...

A. Parallel recording rigs. Same brand/model of mics into the recorders. Mics located coincident as possible.
B. A mixer feeding the same signal to both recorders from the same set of mics simultaneously.
C. Sequential recordings of the same band using the same mics etc (not my preferred)

others?

Although I don't necessarily agree that recording a live band will reveal the differences most clearly between recorders, even if that is the ultimate usage. Perhaps test-signals or mechanical noises may be easier to hear differences?

digifish

[digifish_music]

5. The number of comparisons should exclude chance from playing a significant role in the outcome. It's typical to shoot for 95% confidence in the perceptual world. So that means a performance of ~7 consistent identifications out of 8. Ideally 10 comparisons would make a good standard.

95% confidence for statistical significance is not the same thing as a 95% selection rate.  Statistical significance is mainly influenced by sample size.

Anyway, you can have a very small difference in observed measurement and still have statistical significance; that simply means that the observed difference, while significant, had a small effect.

Lots more here:

http://en.wikipedia.org/wiki/Statistical_significance

I do agree that the first step is objective measurement of whether the source files can be discerned is helpful.  That can be done both by listening and analysis, ideally both.  Once statistical or analytical significance can be established in differentiation, then it becomes interesting to make subjective observations.

I would also add each sample shall be a .wav file of not more than 10MB.  There is simply no way I am ever going to download a 300MB test file, by the time it's done downloading I have probably lost interest, and I'm not going to devote 30 minutes+ to a listening test unless there is the potential for financial gain . . . which can't be realized with a 5 minute listening test

If you are to have a pool of subjects than any statistical test would be done on the pooled population. But I don't think we will ever get that far. We just need a criteria that can reasonably exclude chance from the listening session for individuals.

BTW: it would only take 5-10 minutes to work through 10 x 10 second clips and make A/B preference decisions. If the differences were pronounced, then you would do it in about 4.

digifish

[page]

In this case point 7 requires some compromise, options may be ...

A. Parallel recording rigs. Same brand/model of mics into the recorders. Mics located coincident as possible.
B. A mixer feeding the same signal to both recorders from the same set of mics simultaneously.
C. Sequential recordings of the same band using the same mics etc (not my preferred)

others?

Although I don't necessarily agree that recording a live band will reveal the differences most clearly between recorders, even if that is the ultimate usage. Perhaps test-signals or mechanical noises may be easier to hear differences?

digifish

1) I vote B, we just need to find a mixer that will do 2ch > 4ch. (I'm not familiar with one)
2) Why not do that, *and* the signal tests. I don't see a reason to cheapen the process.

[Gutbucket]

My pragmatic side urges me to mention that we all sort of come up with our own standards of rigor for comparison testing.  Some don't care to do any at all.  I've found it incredibly useful to run and compare two rigs myself.. more so than to download and listen to the comparisons of others if only because I know the original sound and all the dirty details of the comp.  I guess that invalidates my impartial observer status but makes applying what I hear to what I do next time clearer.

Curious if anyone has tired this comparison software yet.  I haven't had time to fool with it but the thought of tuning the typical  ABX comp upside down by listening to only the difference between recordings is intriguing.  It could also potentially 'train' the listener for what to listen for to hear a difference which might have been missed in an typical quick AB comp, but might begin to hear with repeated or long term listening or as her listening skills advance over time. After all, those Edison cylinders sounded just like the real thing to the first, less critical listeners, but I bet their standards of fidelity changed by the end of their lifetimes.

[it-goes-to-eleven]

I think the notion of a "*gold standard*" is overly presumptuous.

There are many useful methodologies depending on the gear and source available.  All will have compromises.

[digifish_music]

My pragmatic side urges me to mention that we all sort of come up with our own standards of rigor for comparison testing.  Some don't care to do any at all.  I've found it incredibly useful to run and compare two rigs myself.. more so than to download and listen to the comparisons of others if only because I know the original sound and all the dirty details of the comp.  I guess that invalidates my impartial observer status but makes applying what I hear to what I do next time clearer.

Curious if anyone has tired this comparison software yet.  I haven't had time to fool with it but the thought of tuning the typical  ABX comp upside down by listening to only the difference between recordings is intriguing.  It could also potentially 'train' the listener for what to listen for to hear a difference which might have been missed in an typical quick AB comp, but might begin to hear with repeated or long term listening or as her listening skills advance over time. After all, those Edison cylinders sounded just like the real thing to the first, less critical listeners, but I bet their standards of fidelity changed by the end of their lifetimes.

Interesting SW, I made some EQ changes to a file and did the difference thing and it showed what I had taken away, for listening. However it requires you have identical recordings - well, recordings of the same exact same thing.

Could be very useful. He has a presentation on it here...

http://libinst.com/Detecting%20Differences%20(slides).pdf

BTW: After installing the SW, go here and download the files on this page, they will allow you to hear the various tweaks/differences shown...

http://libinst.com/diffmaker_example_files.htm

It's also interesting in his test (bottom of the page) he chose 7 as a minimum number of A/B comparisons to perform.

digifish

[digifish_music]

I think the notion of a "*gold standard*" is overly presumptuous.

There are many useful methodologies depending on the gear and source available.  All will have compromises.

I presume nothing, it's a discussion, started with a list of criteria that I like...you are welcome to suggest your own. My criteria are all selected to make differences (based on my own experience and reading of the literature) more likely to be heard.

digifish

[Church-Audio]

The problem is this you have people that think using microphones in a live situation is different then sticking them in front of a pair of speakers and micing them..

That is the crux of the situation as long as people think that some "magic only happens live" you will never be able to make a standard. I agree 100% if there are differences they should be audible in any situation and if they are not then one must ask is this mod worth it?

I think some hide behind the fact that there is no standard test. The funny thing is we ALL benefit from a standard test we can all agree on.

You cant use separate mics even if they are close to the same axis they will sound different.. And these differences mean that some of the things being detected in a sample are skewed due to the fact that there are slightly different signal paths... You need to have ONE SOURCE ONE PAIR OF MICS ONE SET OF CABLES if you dont have this you have nothing but a skewed test.


So since we cant ask a band to play it again exactly the same way they played it the first time we need PRERECORDED MATERIAL.. Then and only then can we ever hope to have a test were any differences are with the item added to the signal chain and not mic position and changes in the bands performance.

There is no way in hell you can have two different sets of mics that will sound the same and you cant put them in the same space so you will not have two identical signal chains ever I test mics for a living you change the position of a mic by an inch its a huge difference in the way it sounds in relation to another pair of mics recording the same source less so if the mics are omni but if they are cardioid forget about it.


In the end I think that there is no way we can all agree on a standard test that's sad because there are a lot of companies making money from the fact that we cant agree and they are laughing all the way to the bank.



Chris

[digifish_music]

The problem is this you have people that think using microphones in a live situation is different then sticking them in front of a pair of speakers and micing them..

That is the crux of the situation as long as people think that some "magic only happens live" you will never be able to make a standard. I agree 100% if there are differences they should be audible in any situation and if they are not then one must ask is this mod worth it?

I think some hide behind the fact that there is no standard test. The funny thing is we ALL benefit from a standard test we can all agree on.

...

You cant use separate mics even if they are close to the same axis they will sound different.. And these differences mean that some of the things being detected in a sample are skewed due to the fact that there are slightly different signal paths... You need to have ONE SOURCE ONE PAIR OF MICS ONE SET OF CABLES if you dont have this you have nothing but a skewed test.

So since we cant ask a band to play it again exactly the same way they played it the first time we need PRERECORDED MATERIAL.. Then and only then can we ever hope to have a test were any differences are with the item added to the signal chain and not mic position and changes in the bands performance.

Chris

Yes identical signal chains up to the mic preamp are ideal and to be favoured. I thought the FR2LE test tenesejedd set up, that  I hijacked, was excellent...apart from the role of chance in the outcome. I would have probably chosen a studio recording of the highest quality tho and made sure to quiet the house down a little

Last time I did a test I used pre-recorded material that I synthesized to make sure it was free of (noticeable) hiss and had a decaying echo, so the sound could descend into the preamp noise.

As far as I can see, simple signals (spanning a wide range of frequencies, dynamics & transients) recorded in a quiet location should show differences more readily than in a wash of noise at a concert.

At the end of the day, not all need to agree, if you do the test properly, then those that don't like the methods can complain, but at least some solid data will be out there.

digifish

[crispin]

Somehow I like this thread ...  because it is trying to add some objective criteria to a subject that has become too subjective.  It was very informative when digifish divided up two recordings into 8 slices and reassembled them in little pieces for a blind side by side comparison - and those who clearly heard differences before (when they knew which was which) suddenly lost this ability.  On the technical level - probably there were still differences in the recorded waveform - but we as the listener could not hear the differences.  One can argue that it was an unfair comparison since the recordings were of music from a hi-fi system and not a live performance - but certainly this test tells most users that (a) differences are small (b) choosing the better/clearer/fuller (choose your own adjective) is subjective at best.

When I was choosing a portable recorder to record classical music .. I found the side by side comparisons on the WingField audio website incredibly helpful.  Clearly there were  large differences in the 'tone' of the music -  but who could say if it is better to have a drier sound that may match the original sound or a sound where the bass has been boosted?  If Digifish had divided up all these recordings into little clips for side by side comparison ...  and asked which clip sounds 'better' in a blind test ... certainly the results would be different than unblind tests.  There are also other differences besides the accuracy of the recording the waveform - and that is stereo separation - and since most of us are trying to record in stereo - this must be an important criterion - but how to objectively do comparisons?

So kudos for Digifish's efforts to bring objectivity to such a subjective science.

[wbrisette]

When I was choosing a portable recorder to record classical music .. I found the side by side comparisons on the WingField audio website incredibly helpful. 

What I tend to find is once you get over the low-end records, the recorder has less to do with the process and the microphones, placement, etc. have a lot more to do with it. Which is why there is mention of using same setup rigs here. I'll bet if you put the 7xx series, Nagra VI, Deva 4/5, and Cantar all next to each other all being fed the same signal you wouldn't be able to tell the difference. However, start putting mics in different locations, or even the same location at different times/events, and you may notice some changes. But those come from factors other than the recorder itself.

Wayne

[Roving Sign]

Aaronji asked a good question and I thought it deserved it's own topic...perhaps we can come up with a gold standard for product comparisons, and how some of the criteria may be achieved?

Let's not take for granted the way we do product comparisons here everyday...that seems to work pretty well - have "we" made any huge mistakes?

I get your drift - but this seems more like a contest than a comparison...

[Ozpeter]

Like I've said before, it seems to me necessary to agree on why we're testing. 

If the object is to do with personal satisfaction, then the kind of test being discussed would doubtless indicate whether listening to identical recordings made in an identical way can show up differences between equipment items, and we can congratulate ourselves on our system choices if proved audibly better than the alternatives.

But if we're talking about whether people listening to a recording made by Joe Bloggs using system X prefer them to recordings made by Fred Smith using system Y, which is the real world concern of professionals, then the real test is to record different music in different places.  That's how Fred and Joe's recordings will be compared by their potential clients.  They wouldn't get the chance to listen to the kind of comparisons being discussed here.  And of course the chances of Joe's recordings being preferred because of the preamps or recorder he uses compared to Fred's is pretty small, unless Fred is using something seriously naff.  What makes much more difference is the mics and where they are placed - and the general competence of Fred vs Joe.  Heck, if Joe's a really nice guy and good to work with, he might even find his recordings are preferred because people know that great guy Joe made them, rather than that pain in the butt Fred.

Anyway, here's a thought - Mackie mixers have little button on the back to switch their output to mic level.  So you could connect a good line level source (eg CD player) to a Mackie and feed that sequentially at mic level to two recorders, and compare the results - pretty easy test to do.  What's the objection to that method?  I must try that out on the R-44 and the H2 - the difference should be very plain, but actually under real music signal conditions, would it be??

[Church-Audio]

IMO in order to do a truly fair comparison the signal chain must be exactly the same minus the devices your trying to listen to. And if you cant do that then you dont have a fair comparison you have a guessing game and that is not fair to the vendor making the product or the consumer buying the product.



For example you were going to listen to mic cables you would want to clamp the mics so they could not move and replay the source wile only changing the mic cables otherwise what do you have? nothing because if you change anything other then the cables the objectiveness of the test is totaly lost.

Chris

[digifish_music]

Aaronji asked a good question and I thought it deserved it's own topic...perhaps we can come up with a gold standard for product comparisons, and how some of the criteria may be achieved?

Let's not take for granted the way we do product comparisons here everyday...that seems to work pretty well - have "we" made any huge mistakes?

I get your drift - but this seems more like a contest than a comparison...

I am not sure how you see this as a contest, it's called experimental control. It's mean to be a methodology for investigating the effects of components in the recording chain, for example I would think this would be good for comparing -

* Preamps
* Microphones
* Mods
* Cables
* Mic positioning
* Recorders

digifish

[digifish_music]

Like I've said before, it seems to me necessary to agree on why we're testing. 

If the object is to do with personal satisfaction, then the kind of test being discussed would doubtless indicate whether listening to identical recordings made in an identical way can show up differences between equipment items, and we can congratulate ourselves on our system choices if proved audibly better than the alternatives.

...

Anyway, here's a thought - Mackie mixers have little button on the back to switch their output to mic level.  So you could connect a good line level source (eg CD player) to a Mackie and feed that sequentially at mic level to two recorders, and compare the results - pretty easy test to do.  What's the objection to that method?  I must try that out on the R-44 and the H2 - the difference should be very plain, but actually under real music signal conditions, would it be??

I see there are several uses for a good blind methodology...

1. To determine if *anything* is audibly different (good for mic preamps, cables, mods, recorders, mics?).
2. Are you consistent in your preferences for a particular sound. So you post two tracks and prefer one for some reason. Are you consistent in identifying that 'sound' and preferring it? If so great, if not, you have learned something.

digifish

[digifish_music]

IMO in order to do a truly fair comparison the signal chain must be exactly the same minus the devices your trying to listen to. And if you cant do that then you dont have a fair comparison you have a guessing game and that is not fair to the vendor making the product or the consumer buying the product.



For example you were going to listen to mic cables you would want to clamp the mics so they could not move and replay the source wile only changing the mic cables otherwise what do you have? nothing because if you change anything other then the cables the objectiveness of the test is totaly lost.

Chris

I agree.

digifish

[digifish_music]

When I was choosing a portable recorder to record classical music .. I found the side by side comparisons on the WingField audio website incredibly helpful. 

What I tend to find is once you get over the low-end records, the recorder has less to do with the process and the microphones, placement, etc. have a lot more to do with it. ...
Wayne

I agree. IMO the main difference between the low and high end gear nowadays is the analog stage/s in front of the A/D converter (mic preamp). Throwing a quality mic preamp in fornt of most low end recorders makes them audibly indistinguishable from gear many times their cost. As I am sure you will be aware (since I keep evangelizing about the MixPre) I have been impressed by the MixPre + R09HR combo for quietude and solo instrument work. Actually that would be a good subject for the topic of this thread...

compare 7xx series, Nagra VI, Deva 4/5 VS (nice preamp + cheap recorder).

digifish

[it-goes-to-eleven]

The problem is this you have people that think using microphones in a live situation is different then sticking them in front of a pair of speakers and micing them..

I guess some of us just "think" a 4 piece jazz band sounds completely different live than in front of any pair of speakers.

[Gutbucket]

...a MIDI-controlled grand piano in a good space ought to do nicely.  Everybody's got one of those, right?  OK, me neither 

Imagining setting up a test in the lobby of the local fancy hotel.

[RobertNC]

I understand the motivation, and it is good in principle, but my 2 cents worth (it's in stocks not bonds by the way so you figure out the value).

First of all in terms of is it live or is it memorex, all I can say is that if you agree that "The problem is this you have people that think using microphones in a live situation is different then sticking them in front of a pair of speakers and micing them.." then all I can say is that is one helluva home entertainment system.  I would love to have it in my living room, but my primary home entertainment system is a woman - I don't think the two could peacefully co-exist.  (PM me if you are interested in a trade).

Second, statistics are not ears and brains.

Case in point.  Earlier this year I recorded a show next to Tony S with essentially the same back end chain, not in exactly the same location, but for the room, very comparable - two stands one seat between them.  I know, I know - definitely not a controlled experiment.  But still, by far the biggest difference was the mics - he ran DPAs, I ran Gefells.  I had been contemplating a set of DPAs for a long time at the time.

I listened to his recording later.  I listened to mine.  In my somewhat less than objective opinion, I will say in no uncertain terms his recording was "superior".   But I liked mine better.  I may still go for a pair of DPAs at some point in time but right now I've lost immediate interest and am piling up change for another set of caps for the Gefells instead.

Go figure, but it does not have to always make sense.

A waveform is a meaningful entity in terms of physics.  Physics can't define "music" though it can only measure a physical phenomenon.

Borrow gear.  Loan gear.  Buy gear.  Sell gear.

Recording music is just like music itself.   Music is an existential experience - stick your finger in the water and decide for yourself if it is fine or not.

[Gutbucket]

[digifish_music]

Second, statistics are not ears and brains.

Case in point.  Earlier this year I recorded a show next to Tony S with essentially the same back end chain, not in exactly the same location, but for the room, very comparable - two stands one seat between them.  I know, I know - definitely not a controlled experiment.  But still, by far the biggest difference was the mics - he ran DPAs, I ran Gefells.  I had been contemplating a set of DPAs for a long time at the time.

I listened to his recording later.  I listened to mine.  In my somewhat less than objective opinion, I will say in no uncertain terms his recording was "superior".   But I liked mine better. 

We are not at odds here. People will be using their ears and relying on their own perception. If what you say is really true, and you really do like the sound of your mics better, then you should choose 10 instances of recordings from yours when compared to the DPA's. If you can't do that, then you were suffering from some distorted perception caused by expectation. Surely that is a good thing to know. I don't see how the methodology affects the Zen of 'liking' things.

In other words the methodology has nothing to do with what is superior. Just what you can reliably identify and choose.

digifish

[Church-Audio]

Audio is audio.... If it can be recorded and there are differences then we can hear them period If there are no differences then we cant. I dont think the  microphones and the preamp and the cables know the difference.. SOUND IS SOUND........ Does live sound different then a pair of speaker YEAH but we are not hear to judge the sound of the recordings we are hear to judge the difference between the two devices. Are you going to tell me that the device knows where it is and will only reveal its "magic" in a live situation... Hell we better get all these idiots out of the studio and tell them to stop wasting there time recording one track at a time because the magic is not there unless its through a loud PA system this argument is lame and its fostered by people that make a living from ignorance.

Sorry for the rant.


BTW if you cant hear it when a mic is plugged in AND "music" is played then there is no difference.. If you want a fair test DONT DO A NEAR FEILD RECORDING do it from a distance with a pair of speakers in the room that should show any differences just like a real live situation recording would. Because the mics are picking up the "room" the only difference is SPL and again we record in all kinds of situations I personally would not pay for a mod that I had to be in a 110 db concert to hear  and nobody has yet to give me a explanation of the differences of a mod in a low SPL recording environment vs a high spl environment I am still waiting


 I have worked with mics for over 20 years in the studio and live and I think that if you cant hear the mod in all types of situations is it really worth spending all that money on?

I am not btw saying that some of the guys that do mods dont improve the performance of the gear they touch and I am not saying anything about anyone here at T.S why because I have never had the chance to evaluate a mod that anyone has done in this forum up close and personal.

Don't go thinking I am talking down to anyone here or anywhere that does mods.. But I am saying show me the money.... You do mods I want to see specs and I want an objective before and after recordings so I can make up my own mind and anyone that buys mods with out these two things is rolling the dice IMO.



Chris

[RobertNC]

I could have reliably chosen between the two source I described because I can even express pretty clearly what the differences were.  The DPAs had a smoother sound across the board and better detail.  The Gefells were a little brighter and had less detail on the bottom end.  I have no doubt I could have said 10 outta 10 times without knowing which was which "This recording is technically superior - it has more detail particular on the low end and better balance across the entire range".  I am also sure I coulda said 10 outta 10 times "This recording is definitely brighter, a little rougher, and the bottom end is lacking in the clarity and detail of the other recording.  But it has a raw sort of punchy in your face sound that makes me feel like I am at the show and I really dig that kind of sound over the other IMO reasonably objectively technically superior recording".

I'm not knocking the concept, I think it is great idea.  I am just questioning the feasibility.  You would have to not only have a lotta controlled experiments, you would have to have a lot that covered the pretty wide range of stuff people tape.  For what I  mostly tape and the sound I like, for rock to me the Gefells just absolutely rock.

I am sure that taping a different genre of music I would have liked the detail of DPAs better.  You would have to not only have a lotta controlled experiments, you would have to have a lotta controlled experments that covered the pretty wide range of stuff people tape too.  That is quite a library of experiments to capture and especially if you are really serious about controlling all the non-gear variables.

And then beyond that you still have real world to ultimately deal with.  IMO by far the biggest variables are two which I have in one case no control over and in the other case, often very limited if any control over - venue and location.  Those are major practical factors that are hard to incorporate into an organized program that already requires a lot of control over a lotta gear over a lotta genres to even be useful as a concept.

Let's face it, with any gear there are some situations so abysmal  Jesus could not pull a good tape.  But what about the practical taper like me who is often faced with a mix of sometimes pretty sweet spots, other times where location and or venue is definitley compromised, and the occasional situation where I go and put gear up because that is what I do, with the attitude of hey I like to see shows, like to tape shows, and do, but as far as the results, before I even hit record I already know what I am gonna get can be described as nothing more flattering than "I documented the show".

Add those variables, and not only does the matrix start becoming really huge, seriously how many of us have the desire, much less the time and opportunity to extend a large matrix of equipment and genres to also include "let's also set up as many controlled experiments with as wide a range as possible of gear in shitty location and or shitty rooms?

But that is not only a real consideration, but also really a primary consideration.

And to use the DPAs again as an example.  Please don't be offended anyone, but as far as the higher end gear I see people actually run regularly if it cames to say Scheops vs DPAs, it would not even be a decision for me. I think for the taper community and admittedly a broadly brushed stereotype, to me at least the DPAs are really the premier line i see in any sort of regular rotation. 

And don't interpret my next statement as me presenting as "fact", it is definitely not - I've never even owned a pair.  But I've heard the same thing from more than one taper I respect highly and that have run many kinds of gear that while DPAs make stellar recordings, higher quality also comes with less forgiveness.  Again that is pure anecdote, and second hand anecdote at that. But having heard the same thing and from the people I've heard the same thing from, I am by no means ready to just dismiss that there may be some merit to what they have said.

Don't get me wrong.  What you are proposing is very logical.  It has many genuinely useful merits.  And I respect that the level of effort of trying to follow though is truly admirable.

I'm just not convinced that concept of gold standards caneven be applied for the real world tapers in the real world environment faced with so many real scenarios.

But again, not trying to discourage, just suggesting that I think the approach of meeting fellow tapers at the kind of shows you tape, exchanging dialogue and then eventually exchanging gear and occasionally managing to just try to get together somewhere every once in a while and do some kind of basic collaboration at a show, while far from scientific, is still a strategy that while not yielding metrics, can nevertheless provide some genuinely valuable exposure to make hopefully what to to you personally prove to be equipment purchases that you feel were seriously worthwhile changes...

[RobertNC]

 I have worked with mics for over 20 years in the studio and live and I think that if you cant hear the mod in all types of situations is it really worth spending all that money on?

What you say makes scientific sense.  I am a hobbyist, not an enginerr.

I'm not talking about mods here.

However, when I bought my Gefells the first thing I did was set them up and record in the living room.  The  I set my ADK LDs up exactly same and recorded the exact same things.

Now these are really different microphones, not just in design, but in quality too.

I listened to the results numerous times.   My final analysis was that I was really glad I had a real show to tape the next week.  Because after this little experiment, I was thinking wow, that was a shitload of money, and I can hear a difference but just barely.  I sure hope I did not make that kind if purchase to essentially mostly lower my profile.

After I taped a real show, the money spent was a total non-issue.

No matter how unscientific, my opinion is (and again I'm not a sound engineer, but I am a degreed engineer and at least understand logical experimental design and control) , sorry, my firm opinion is that field and living room are apples and oranges.

[RobertNC]

One thought that came to me while I matched a batch of mics was how many of those who feel that audio is entirely subjective and gear must be tested on a live source, etc. own a matched pair of mics?  Do you feel that matching is important?  How do you think matching is done?  How would you do it?

I own both.  I feel like matching is a good benchmark, but less and less important as mic quality increases.

As far as how it is done, of course it is a measured response of two microphones to a single controlled signal with "matching" being accepted based on an acceptable statistical range of variations of two mics to a single source.  That is scientifically a basic principle of calibration.

It tells you that a pair of microphones will perform within an acceptable range under the same circumstances - the matching source circumstances.  If the matching charts I received represzent the entire data used to accept a "match" you don't even have enough data to extrapolate that they will therefore perform essentially the same in the field, and you have absolutely zero data about how your mics will perform in the field versus a living room - that is a totally different set of parameters.

[it-goes-to-eleven]

Let's say I want to buy a digital camera for a big vacation and want to get really good pics.  I could compare two models by displaying a picture of Yosemite on my LCD and taking a picture with each camera.  I might see some differences or I might not.  But regardless, that test would probably not tell me which camera would take better landscape pics on my vacation.

However, when I bought my Gefells the first thing I did was set them up and record in the living room.  The  I set my ADK LDs up exactly same and recorded the exact same things. Now these are really different microphones, not just in design, but in quality too.

I listened to the results numerous times.   My final analysis was that I was really glad I had a real show to tape the next week.  Because after this little experiment, I was thinking wow, that was a shitload of money, and I can hear a difference but just barely.  I sure hope I did not make that kind if purchase to essentially mostly lower my profile.

Same experience here.  I've owned or own C4's, DPAs, AT853's, MG200/210, Schoeps mk21/mk4/mk41.  The difference between the C4's and the higher end mics in front of my VR4's or Martin Logan CLS's is very small.   But in the field, the difference is far more dramatic (especially between cheap hypers and great hypers, but that's an extreme example).  Same can be said of my many pre-amps in front of speakers.

Again, I'm not saying the methodologies being proposed are useless. Far from it. These discussions are great. Though some here seem to be dismissing the significant, and often critical differences between a 'lab test' and the real world.

And, fwiw, for the past 20 years I have done quite a lot of test design.  Think thousands of result graphs over the past 10 years and individual test projects that saved $30 million in a single quarter.  A number of folks are using my datasets for their research papers and PhDs.  Just saying that methodology, testing, visualization and comparative analysis are more than hobbies for me.  But tapin' is a hobby

[Church-Audio]

Let's say I want to buy a digital camera for a big vacation and want to get really good pics.  I could compare two models by displaying a picture of Yosemite on my LCD and taking a picture with each camera.  I might see some differences or I might not.  But regardless, that test would probably not tell me which camera would take better landscape pics on my vacation.

However, when I bought my Gefells the first thing I did was set them up and record in the living room.  The  I set my ADK LDs up exactly same and recorded the exact same things. Now these are really different microphones, not just in design, but in quality too.

I listened to the results numerous times.   My final analysis was that I was really glad I had a real show to tape the next week.  Because after this little experiment, I was thinking wow, that was a shitload of money, and I can hear a difference but just barely.  I sure hope I did not make that kind if purchase to essentially mostly lower my profile.

Same experience here.  I've owned or own C4's, DPAs, AT853's, MG200/210, Schoeps mk21/mk4/mk41.  The difference between the C4's and the higher end mics in front of my VR4's or Martin Logan CLS's is very small.   But in the field, the difference is far more dramatic (especially between cheap hypers and great hypers, but that's an extreme example).  Same can be said of my many pre-amps in front of speakers.

Again, I'm not saying the methodologies being proposed are useless. Far from it. These discussions are great. Though some here seem to be dismissing the significant, and often critical differences between a 'lab test' and the real world.

And, fwiw, for the past 20 years I have done quite a lot of test design.  Think thousands of result graphs over the past 10 years and individual test projects that saved $30 million in a single quarter.  A number of folks are using my datasets for their research papers and PhDs.  Just saying that methodology, testing, visualization and comparative analysis are more than hobbies for me.  But tapin' is a hobby

Here is the problem.. You cant use two sets of mics to feed the signal chain. Because if you move the mics even an inch they will not sound the same. So unless we are willing to use an active splitter how can you record a live performance with one set of mics and split it into two signal chains.. If you cant do that then you cant have a 100% objective test.

That is the problem. Arguments could be made that an active splitter does not allow for the same type of load a pair of mics present to a preamp. So then you have problems with your signal chain not being 100% accurate between the two sets of signal chains and a Y cable presents the same problem of loading.

So the only way to use one set of mics is to have a source that can be played back precisely two times with only the device in question being switched out.

This is not rocket science. I build and test mics for a living and I am saying it is 100% impossible to have two sets of mics that sound exactly the same and its impossible to put two sets of mics in the same exact space. So you then introduce a guessing game of is it the fact that we have two sets of mics that are changing the sound or do we have a mod that is changing the sound making any meaningful results null and void.

To be fair we have to use a speaker or set of speakers in a fixed position to do a test like this. One way to do this might be to use a live venue with a cd source or prerecorded live show played through the PA to simulate the nuances caused by room acoustics of course not all of the complex reflections will be there because we dont have a complex sound source on stage adding to the total sound.


but how does this complexity translate into a situation where they "the complex reflections" must be there in order to evaluate gear? Nobody has answered that question as of yet to my satisfaction.


In the end sound is sound you can move big speakers into a big space and get a big sound. Or take them into a small space and get a more compressed sound but you still have something to listen to.

Two sets of mics = unfair evaluation of the gear in question its just that simple. The best test is live recorded music played through a good set of speakers how else do you get the same "exact" signal into two sets of signal chains? THE MILLION DOLLAR ANSWER IS YOU CANT this is not even grade 8 science this is basic understandings of the principals of sound and of microphones.


Chris

[RobertNC]

No I was saying I am definitely not a sound engineer, and can only offer my anecdotal opinions without any real scientific basis.  I am a chemical engineer, which is relevant only in terms of understanding the need for controls and experimental design.  But ChE is also by far the least scientific and most empirical engineering discipline, and coming from a heavy empirical background, basically what you are saying is what I am saying.  Yes, the microphones - in purely physical terms - respond the same - they know nothing about where they are.  But the empirical part is where I see the problem - the translation to real world performance.

If I was in a meeting and someone told me "we have carefully selected every component carefully tested and evaluated under strictly controlled uniform test conditions", I would say "And have you constructed a plant on that basis?"  When they said "Yes, absolute state of the art", I would say, "Sorry for having to leave so abruptly gentleman, but I really need to get moving and hopefully be in another timezone before you open the first valve."   

[Church-Audio]

No I was saying I am definitely not a sound engineer, and can only offer my anecdotal opinions without any real scientific basis.  I am a chemical engineer, which is relevant only in terms of understanding the need for controls and experimental design.  But ChE is also by far the least scientific and most empirical engineering discipline, and coming from a heavy empirical background, basically what you are saying is what I am saying.  Yes, the microphones - in purely physical terms - respond the same - they know nothing about where they are.  But the empirical part is where I see the problem - the translation to real world performance.

If I was in a meeting and someone told me "we have carefully selected every component carefully tested and evaluated under strictly controlled uniform test conditions", I would say "And have you constructed a plant on that basis?"  When they said "Yes, absolute state of the art", I would say, "Sorry for having to leave so abruptly gentleman, but I really need to get moving and hopefully be in another timezone before you open the first valve."   

As soon as I turn a piece of gear on I expect the "real world performance" to be the same no mater where I am if not then its crap gear.

I am not talking evaluating the gear in a lab I am still talking about using the things on ether side of your head. If we were strictly talking science here I would want lab tests on the gear as well but since were are not talking strictly lab tests here your argument does not apply. The simple fact is gear should function the same no mater where its placed. And that is a fact would I want a chemical plant setup based on some imaginary test conditions hell no... But we are talking about two sets of speakers in a living room vs a night club please explain the differences to me is the the lack of clanking beer bottles that makes my idea invalid? or the drunks yelling in the background. There are no environmental differences between some clubs and your living room... EXCEPT SIZE... and I have been and done sound at a few places that would make your night club look like a living room.......

[Busman Audio]

Acoustics make a huge difference on what your microphones pick up and how your gear sounds to your ears. If this doesn't make sense to you !!Wow!! 

There is no gold standard for testing as we have hashed this out so many times. 

Soundstage is the way your ears perceive the "image" of sound being produced that is how you can localize sounds made around you and how blind people see.  Its all about the ears!!!

[Church-Audio]

Acoustics make a huge difference on what your microphones pick up and how your gear sounds to your ears. If this doesn't make sense to you !!Wow!! 

There is no gold standard for testing as we have hashed this out so many times. 

Soundstage is the way your ears perceive the "image" of sound being produced that is how you can localize sounds made around you and how blind people see.  Its all about the ears!!!

You are talking about acoustics to someone that has spent half of his life studying them!! Sound in a live environment is different then a living room yes.. But if we all needed a live environment to hear the differences how can one judge the sound quality differences of a cd player? take it to a gig? 


Sound is sound you can capture it with a microphone the mic will pick up whats in the room how is that  different from the test we are talking about? it all comes back to sound is sound will live vs pre recorded sound different hell yeah but cant we listen to a prerecorded recording and judge the differences in gear? Or do we need a bigger "sound stage"? This argument is very dated I have heard it all before I think there is no gold standard because there are still alot of misinformed people who believe that the magic in the modifications they pay for only happen in at a live show....  If a mod is really good it should sound good anywhere if not how do you test all the gear you mod do you take each and every piece of gear to a live show and then ship to the customer??

I am fighting for a fair shake for all people not just for people that make money from modifications btw this is not personal this is about objectively evaluating sound so we can hear the differences between two pieces of gear. You sir with your rude comments are making in personal makes me think you have something to be upset about. Be professional dont take this as an attack on your business because its not. Its simply me stating what I think a fair evaluation method should be and somehow your translating this into an attack on you that is simply not the case.. I will say this again I dont know anything about your mods and how effective or none effective they are.


Chris

[RobertNC]

This is not rocket science. I build and test mics for a living and I am saying it is 100% impossible to have two sets of mics that sound exactly the same

If you are saying this them you are saying the whole concept is also 100% impossible.   If I buy a pair of mics based on a carefully controlled comparison, then I still did not buy THE microphones that were used in the comparison so in fact I don't have the same mics I made the decision on.

And after some field use there has been environmental and temperature exposure.  And man, even though I use windscreens religiously especially indoors at theatres for smoke screens if you've been to a Phil show recently in a small venue you know the fog of pot smoke is so dense it obscures the stage.

So by that argument I can buy a set of mics and in a short period of time no matter how much care I take with my gear (a lot, this stuff is not cheap and I'm not rich) I don't even still have the same mics I bought.

  

[Busman Audio]

You can just keep telling people you are right Chris and I will just keep telling them to listen to their ears.


sorry to offend you but I have a right to my opinion just like you do!

[Church-Audio]

You can just keep telling people you are right Chris and I will just keep telling them to listen to their ears.


sorry to offend you but I have a right to my opinion just like you do!

You absolutely have every right to your opinion. I dont agree with it and you dont agree with my opinion and that's ok. I would like to correct one thing.. I have never told anyone to not use ears as a major part of any evaluation EVER. But it would also be nice to know some of the folks doing mods are also using test gear to if anything make sure the mods they are doing are performing to the "spec" although unpublished for most is met dont you agree?
Chris

[it-goes-to-eleven]

Just as an aside, once you guys agree on the Gold Standard, I'm going to start a thread to develop the Platinum Standard

[Church-Audio]

This is not rocket science. I build and test mics for a living and I am saying it is 100% impossible to have two sets of mics that sound exactly the same

If you are saying this them you are saying the whole concept is also 100% impossible.   If I buy a pair of mics based on a carefully controlled comparison, then I still did not buy THE microphones that were used in the comparison so in fact I don't have the same mics I made the decision on.

And after some field use there has been environmental and temperature exposure.  And man, even though I use windscreens religiously especially indoors at theatres for smoke screens if you've been to a Phil show recently in a small venue you know the fog of pot smoke is so dense it obscures the stage.

So by that argument I can buy a set of mics and in a short period of time no matter how much care I take with my gear (a lot, this stuff is not cheap and I'm not rich) I don't even still have the same mics I bought.

  

Your saying your mics dont sound the same but what are you basing that on? There are lots of mics out there that dont drift very much over time the DPA 4006 comes to mind as an example.


We are not evaluating the quality of the microphones we are evaluating the quality of the modifications done to a device in the signal chain. And so since that's all we are evaluating the mics become irrelevant because the sound of the two samples comes from the same source. If you base your purchase of a mod on the overall sound of both recordings and not on the differences between the two recordings then you are wasting your money since the overall quality would be a direct product of the source in most cases. No two pairs of mics will sound the same No two pairs of matched mics will sound the same because there is no such thing as mics that are 100% perfectly matched they do not exist and it would take many many hours to find such a pair so most matching even by really good companies like DPA is based on a tolerance of anywhere from +- 3 db to + - 0.2 db. But even this might be based on sensitivity at one frequency and not broad band I would say broad band matching and finding  two identical microphones is way harder then winning the lottery.

Chris

[Church-Audio]

Just as an aside, once you guys agree on the Gold Standard, I'm going to start a thread to develop the Platinum Standard

No its the Diamond standard! 

[RobertNC]

I thought we were talking about product comparison, not just pre-amps.

In any event, I would like to address Chris's earlier comments.  First of all, I don't know your history, but I do know I have read this thread in it;s entirety and if you have found rudeness and personal attacks, I think that may be some kind of "history"

I am not questioning your knowledgeability, and this is not a personal attack, although it may be perceived that way.  I tend to be right some of the time, dead wrong some of the time, but brutally forthright all of the time, and I am used to people taking offense where none was intended.

So as far as modded stuff goes, it seems like we were talking about product comparisons but you seem to go to mods as a recurring theme.  While some of us here are openly admitting we are making subjective decisions, I have to start questioning some of your comments as being maybe not quite as objective as you seem to present them.

Having said that,  I personally will no longer touch any modded gear.  When someone asks me how I feel I give them my subjective but honest opinion.  If you are on a budget, modded gear can in my opinion genuinely give you a real upgrade with a lot of bang for your buck and put the quality of recording you can obtain on a totally different level than the same "prosumer" stock unit could provide.  There are problems however - you immediately not only void the warranty but if you have problems you have limited to resolutions, and practically never have an immediate option.  I know from personal experience.  Take modded gear and have it die on you even in a city like NY where you can find anything that exists, and you are still outta luck because anyone that would do quality work willl also not be willing to even touch a piece of non-functional modified gear.

That very unpleasant experience put an end to my willingness to run mod gear - the professional stuff is worth every penny.

I maybe wrong, but I had the distinct impression you were also in the business of modifying or at least building non standard gear yourself?   I have even steered a few people your way saying "Sorry, I don't do stealth so I can't advise but a lotta people that run stealth gear seem to have good results form this guy Church who I think provides a lotta stuff targeted at the stealth market - join taperssection and you'll find him.   

And as far as Busman goes, never had any dealings with him either, but people seem to have a high view of his mods and I've heard some good results.  And I understand the mod market exists and benefits many tapers on a budget, and I have had experience with non-stealth modded prosumer gear and I have steered people to him a number of times with the similar qualifier that I have not used his gear but many like it and unlike that other guy who's reputation is apparently beyond reproach and can rest on his laurels (i.e. to put it bluntly if you get a good mod, good for you, if you get one that has defects the other guy is not above totally fucking you over as an individual because he knows you have no real voice as far as his untarnishable rep), my impression is that Busman is OTOH actively trying to grow both his reputation and his business, so I would trust him to give better service.     

So what is with the whole mod issue and the perception of personal attacks?

Are we still all talking about differing views on product comparisons, of which modded and unmodded equipment would both be considerations, or have we moved to the realm of product business or some other personal business now?

[Gutbucket]

Discussion seems to have move on to mods and other things but here's a viewpoint on mic listening comps in particular that I typed earlier this morning-

Mics perform the same regardless of location but locations vary greatly.

There is testing procedure, there is data and there is the interpretation of that data.  We can get hung up on any of those three.

Let's keep in mind we are talking about listening comparisons not test measurements.  It can be very difficult to relate imperial measures to subjective qualities.  The measurement tool in this case is our ears and brains so we don't have to worry about relating measurements to our experience of the sound.  But we also can't make a blanket proclamation of which is objectively better or measures better.

The appropriate pragmatic testing procedure is the one that allows the user to make an informed decision as to which piece of gear will give him the results he desires in the appropriate application.   No test is perfect and we don't require a perfect test, just one that shows us which piece of gear performs better in our particular application.  A big problem is the potential for self deception.  How much rigor in constraining variables is necessary?  What is the true value of each variable.  Do I even recognize all the variables?

That problem is compounded when comparing recordings made by others, which requires answering the question, "Are her test procedures and standards the same as mine?" Answering that is difficult and requires more detailed communication and ground rules than many here are willing to commit to in order to rule out self deception.

On the applied aspect of living room testing, I'll note that not many are aiming to primarily record and reproduce the sound their living rooms, as live sound recording is the goal.  That complicates things even if we all were to agree that a recording test made in the living room could be made validly because, again, we're talking about a listening test and are going to be listening for the things we hear and value in a recording of a live event in a room that is probably very unlike a living room.  The reproduced sound in the living room is a small sub-set of the information at the actual event.
 
I hear or do not hear plenty of important things live that are either not available at all or more commonly are not reproducible in the same way via playback in my living room of my recordings.  That is not just limited to the bandwidth performance of my playback system, but things that would appear completely different from the mic's perspective at that live location.  For example, spatial information such as the off-axis ambience performance reproducing the direction and clarity of things like clinking glasses and a rattling register at the rear of the room at a jazz show; the sniffling patron in the squeaky seat two rows back at the string quartet date; the sound of the room reverberance off the coffered ceiling of the chapel; the bass modal response of the concrete walls of the club; etc. 

It's easier to make a meaningful listening assessment if we're listening to a recording made in the actual application.  We may also be able to extrapolate the results of a different test (living room or whatever) and project that on our final 'actual' use, but that is much harder and is closer in spirit and procedure to making a standardized measurement than doing a listening test.

[Church-Audio]

I thought we were talking about product comparison, not just pre-amps.

In any event, I would like to address Chris's earlier comments.  First of all, I don't know your history, but I do know I have read this thread in it;s entirety and if you have found rudeness and personal attacks, I think that may be some kind of "history"

I am not questioning your knowledgeability, and this is not a personal attack, although it may be perceived that way.  I tend to be right some of the time, dead wrong some of the time, but brutally forthright all of the time, and I am used to people taking offense where none was intended.

So as far as modded stuff goes, it seems like we were talking about product comparisons but you seem to go to mods as a recurring theme.  While some of us here are openly admitting we are making subjective decisions, I have to start questioning some of your comments as being maybe not quite as objective as you seem to present them.

Having said that,  I personally will no longer touch any modded gear.  When someone asks me how I feel I give them my subjective but honest opinion.  If you are on a budget, modded gear can in my opinion genuinely give you a real upgrade with a lot of bang for your buck and put the quality of recording you can obtain on a totally different level than the same "prosumer" stock unit could provide.  There are problems however - you immediately not only void the warranty but if you have problems you have limited to resolutions, and practically never have an immediate option.  I know from personal experience.  Take modded gear and have it die on you even in a city like NY where you can find anything that exists, and you are still outta luck because anyone that would do quality work willl also not be willing to even touch a piece of non-functional modified gear.

That very unpleasant experience put an end to my willingness to run mod gear - the professional stuff is worth every penny.

I maybe wrong, but I had the distinct impression you were also in the business of modifying or at least building non standard gear yourself?   I have even steered a few people your way saying "Sorry, I don't do stealth so I can't advise but a lotta people that run stealth gear seem to have good results form this guy Church who I think provides a lotta stuff targeted at the stealth market - join taperssection and you'll find him.   

And as far as Busman goes, never had any dealings with him either, but people seem to have a high view of his mods and I've heard some good results.  And I understand the mod market exists and benefits many tapers on a budget, and I have had experience with non-stealth modded prosumer gear and I have steered people to him a number of times with the similar qualifier that I have not used his gear but many like it and unlike that other guy who's reputation is apparently beyond reproach and can rest on his laurels (i.e. to put it bluntly if you get a good mod, good for you, if you get one that has defects the other guy is not above totally fucking you over as an individual because he knows you have no real voice as far as his untarnishable rep), my impression is that Busman is OTOH actively trying to grow both his reputation and his business, so I would trust him to give better service.     

So what is with the whole mod issue and the perception of personal attacks?

Are we still all talking about differing views on product comparisons, of which modded and unmodded equipment would both be considerations, or have we moved to the realm of product business or some other personal business now?

I am actually referring to the personal statements made by a member here and someone that is part of a thread that subsuquently deleted all of his posts that referenced me.

This thread has nothing to do with a "particular vendor" it has to do with the fact that all of the guys doing mods dont publish specs and dont have objective tests for people to base an educated opinion on the validity of the mods in question.  I think that's unfair and I think its misleading and I think that anyone buying mods from any company that cant provide published specs on his or her mod and provide a valid objective test must be questioned as to why these seemly basic things are missing. I dont know any electronics engineer that improves a product with out doing some kind of electronic test on the end result never mind listening tests I would have to question why that is. I am pushing for a standard because I believe there should be one. And people when we are talking about improving a circuit we are talking about electronics and if we are talking about electronics then we are talking about things that no only should be able to be measured but also heard. If not your just paying for someones best guess.
Its called spinning the wheel of parts substitution and not about making sure you can measure and hear the differences.

[Busman Audio]

if you referring to me having deleted a post I have not. 

Just keep hacking away at other businesses that sounds like a good way to make friends.




this was supposed to be about listening comparisons!

[Church-Audio]

if you referring to me having deleted a post I have not. 

Just keep hacking away at other businesses that sounds like a good way to make friends.




this was supposed to be about listening comparisons!

What are you talking about? anyway this thread has nothing to do with Busman Audio. I dont know anything about your mods and I have not examined any so it would be unfair for me to comment on them. I am making generalized statements about the industry of mods. As a whole its not directed at any individual.

[Roving Sign]

This thread has nothing to do with a "particular vendor" it has to do with the fact that all of the guys doing mods dont publish specs and dont have objective tests...

Im not sure we know any of that for sure...Im sure there are some simple things that are considered trade secrets...I dont blame them.

Chris Church - do you have any regrets about revealing the details of your high spl mic mod? Seems I remember other vendors copying you after that...personally I thought it was foolish of you to give that out. Why give away your competitive edge?

And to be fair - Edirol never published any THD specs on the UA5 - only the most basic specs appeared with that device's literature...so you cant just blame the modders for that situation.

One thing is for sure - a few of the small businessmen on this board are NOT their own best advocates...

[Roving Sign]

I once read an audiophile describe a tube power amp he had built with a switch to disable the negative feedback.  By his own admission, the THD of the system was much higher without the feedback, but he preferred the stereo imaging of the zero feedback version, except when the source had a lot of transient high-frequency content--he felt the NFB version was more detailed.  With the caveat of not having the bit of kit to test myself, I can conclude he felt that the smearing of high frequencies cause by the distortion of his system yielded better stereo imaging.  I don't conclude that is universally true for everyone, but if you can learn something like that about your own gear preferences, it does make your gear selection procedure much easier.

I too have heard that human ear likes a certain amount of distortion...old issue of Stereophile or something...

[Church-Audio]

This thread has nothing to do with a "particular vendor" it has to do with the fact that all of the guys doing mods dont publish specs and dont have objective tests...

Im not sure we know any of that for sure...Im sure there are some simple things that are considered trade secrets...I dont blame them.

Chris Church - do you have any regrets about revealing the details of your high spl mic mod? Seems I remember other vendors copying you after that...personally I thought it was foolish of you to give that out. Why give away your competitive edge?

And to be fair - Edirol never published any THD specs on the UA5 - only the most basic specs appeared with that device's literature...so you cant just blame the modders for that situation.

One thing is for sure - a few of the small businessmen on this board are NOT their own best advocates...

I dont think that reveling distortion levels or frequency or other tests and subjective listening tests would reveal to anyone how or what was modified inside unless you are psychic.

No regrets because most of the competition that is doing the mod is doing it wrong And I released it so that others could do it them selves because I did not have the time to mod all the mics that my competition sold that were distorting.....  

I think there are alot of small businessmen on this board that are not there own best advocates but as I explained this is not about any one single vendor its about getting a test that is objective all people who buy or sell mods would benefit from this. I dont understand the resistance pardon the pun  


And the reason why distortion was never revealed in the spec on the preamp your talking about is simple it was horrible that unit had no headroom and had very high distortion levels.


Chris

[chris319]

Here is my gold standard:

http://audio.rightmark.org/index_new.shtml

and another:

http://www.fesb.hr/~mateljan/arta/

The ear can be fooled.

[Church-Audio]

Well, if you have to have a live show, then bring two mics*, or two pairs of mics, and swap them between positions halfway through the show.  Then cut the samples into small bits as described by OP, and listen.  It's not perfect, because the source won't be exactly the same.  But what have we learned if you prefer one position or one half of a set rather than one mic over the other?  That other factors are more important than the selection of those two particular mics.  Since probably not too many people swap mics between songs in a set, you probably don't need to own both mics.

On the other hand, if you consistently pick one mic over the other, there you go.





* Gear doesn't really need to be compared in stereo.  Sure, it may be easier to discern differences with two channels of signal, since psychoacoustic effects will become apparent, but try this:  record one song with one pair, another song with a second pair.  Set those two songs aside.  Then record the rest with different mics for each channel (as close together as possible), and compare the mono tracks to each other.  Pick a favorite.  Now, go back to the stereo songs, ideally blind.  Pick a favorite based on your preference for stereo imaging.  If that's the same as your mono favorite, there, that wasn't too hard, was it?

If it's different from your mono preference, then listen to the mono files and try to discern what difference in mono caused you to prefer that mic in stereo.

I once read an audiophile describe a tube power amp he had built with a switch to disable the negative feedback.  By his own admission, the THD of the system was much higher without the feedback, but he preferred the stereo imaging of the zero feedback version, except when the source had a lot of transient high-frequency content--he felt the NFB version was more detailed.  With the caveat of not having the bit of kit to test myself, I can conclude he felt that the smearing of high frequencies cause by the distortion of his system yielded better stereo imaging.  I don't conclude that is universally true for everyone, but if you can learn something like that about your own gear preferences, it does make your gear selection procedure much easier.

The more you change in a test the more you confuse. Simple one set of mics 1 source that can be played back = objective test.

[Will_S]

if you referring to me having deleted a post I have not. 

If you didn't delete any post, what happened to the one referred to below?

http://taperssection.com/index.php/topic,113128.msg1517236.html#msg1517236

Graphs don't tell you how something sounds to your ears.  I hear differences in these files and that is a good thing since I do the mods.



Why don't you quit telling others what to do Chris.  You have done enough to ruin my business already with your constant comments whenever tests come up of my gear.

If I made graphs then you would say I doctored them somehow so just quit it just like you did when I made some critical sound sample tests with the R4 mods a long time ago.

Hell busman my ears tell me that after you did the mod on my ole' UA5 it sounded a hell of a lot better and the test you did on the R4 convinced me to buy a R4 w/ mod. Thanks for all you and for my $$ I will stick w/ the mod gear.
Peace

I noticed that this post vanished shortly after I saw you post this (of course it may have vanished sooner, that's just when I noticed it was gone):

http://taperssection.com/index.php/topic,113482.msg1519349.html#msg1519349

This test is not valid considering I make these mods to perform better for concert recording and not a home stereo recording of a studio album.

So I'm confused.  If you haven't deleted any posts, what happened to that one?

[Busman Audio]

You are a good researcher Will_S....    I was not talking to you in the first place and that post was not in this thread at all. 

[Teen Wolf Blitzer]

   How does it feel to know you, Core and SP are the most responsible for bootleg recordings being sold the world over?  Lay off Busman.  While you say your rants aren't directed at him...let's be honest.  There are 2 guys in the country modding gear.  Chris and Oade.  One is not a member here.  One is.

if you referring to me having deleted a post I have not. 

Just keep hacking away at other businesses that sounds like a good way to make friends.




this was supposed to be about listening comparisons!

What are you talking about? anyway this thread has nothing to do with Busman Audio. I dont know anything about your mods and I have not examined any so it would be unfair for me to comment on them. I am making generalized statements about the industry of mods. As a whole its not directed at any individual.

[Will_S]

Hoping this thread can somewhat get steered back on topic:

1. Levels need to be matched. Louder almost always sound better. The threshold for loudness differences is in the order of 0.5 dB (under the best listening conditions in an A/B back-to-back change. It rises to about 1.5 dB at 1 minute separation between the examples, again under ideal listening conditions). In the real world 1 dB is fairly close to the just noticeable limit.

This might not be an issue with very accurate recorders or preamps, but should the goal be to match peak or RMS levels?

I was going to whip up a blind comparision of a recording I made with my Audix Micro mics right next to a friend running Neumann KM140s - the idea being we should pretty much all agree that the two setups should sound audibly different, and so it would serve as a kind of test that the sort of blind test you propose either is or is not sensitive enough for the majority of posters here to actually correctly identify the different sources on their playback systems.  But evidently one set of mics (or something else in the respective recording chains) must compress transients or somesuch, as matching peak levels results in fairly different average loudness levels.

Potentially, some preamp/recorder mods might do something similar (perhaps even in an audibly pleasing way), which would make proper comparisons difficult.  How to proceed in such a case?

[Gutbucket]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

[Church-Audio]

  How does it feel to know you, Core and SP are the most responsible for bootleg recordings being sold the world over?  Lay off Busman.  While you say your rants aren't directed at him...let's be honest.  There are 2 guys in the country modding gear.  Chris and Oade.  One is not a member here.  One is.

if you referring to me having deleted a post I have not. 

Just keep hacking away at other businesses that sounds like a good way to make friends.




this was supposed to be about listening comparisons!

What are you talking about? anyway this thread has nothing to do with Busman Audio. I dont know anything about your mods and I have not examined any so it would be unfair for me to comment on them. I am making generalized statements about the industry of mods. As a whole its not directed at any individual.

  Are you crazy what does this have to do with bootleg recordings? Sure lots of people use my gear for bootlegs do you not think bootlegs happen with open taping? I am like Smith and Wesson I can't be held responsible for what some jackass does with my product anymore then a gun company can be held responsible for the dumb ass that shoots him self in the foot, Lame argument. I am not picking on Busman we actually spoke today and everything was cool. And to my knowledge still is.

My posts are not directed at any one individual at all they are directed against anyone that does mods and there are lots of people doing mods besides Chris and I am not going to name names... This is general information not directed at Chris or anyone in particular. But again I am getting sick of having to explain this.


Chris

[Belexes]

  How does it feel to know you, Core and SP are the most responsible for bootleg recordings being sold the world over? 

I've taped hundreds of shows with CS, SP, and CA mics and no one has pressed/sold any of my stuff. Please connect me to your all-knowing "stealth=bootlegs" connection so I can start getting rich.

 

[it-goes-to-eleven]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

RMS levels should match but peak levels *should* also..  When doing comps I tend to match peak levels and then expect the rms levels to also match.   If they don't then something's generally awry.

I always keep a written record when I master my shows. I have a very thick binder notebook. In addition to regular source info it includes seat location, pre-amp gain throughout the recording, and peak and rms levels for each section of the recording where different pre-amp gain was used.   And then, ultimately, any gain corrections applied.  Those notes are absolutely invaluable when I record a venue or performer in the future (how much gain and what will your final peak be with mk4's when you're 6 feet from Ravi and 20 feet from Roy's drum kit and the kick drum is facing you?).  Checking levels is so bush league ;-)

[SparkE!]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

RMS levels should match but peak levels *should* also..  When doing comps I tend to match peak levels and then expect the rms levels to also match.   If they don't then something's generally awry.

Your ears respond more to the average signal power level than they do to peaks.  That's the reason you can run a limiter and not severely affect the way the audio sounds.  Matching the rms levels is the way to go, in my opinion.  So, I would suggest matching rms levels and expect the peak levels to match also.  It's harder to do and still avoid running one of your peaks into the rail, but it really does help to avoid the problem where one source is preferred to the other due to signal power differences between the two sources.  If you match rms voltage levels, you automatically match the signal power levels, assuming you are playing both sources back on the same system.

[Church-Audio]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

Under most cases if the recorder has a digital knob for gain it should be easy if not then I suggest using a calibrator like the one I use for calibrating my microphones with a 1k tone. This is the middle of the spectrum all you are doing is calibrating the level settings of both devices under test a single tone will do that.

If the mics aren't perfectly matched it does not matter because both recordings will be made with the same microphones. And if both devices under test have detented attenuators then again it does not matter as long as both are set to the same value. The problem will arise with continuously variable gain controls if they are present then you MUST calibrate the input and output levels any differences on input and output levels relative to gain settings if they are detented should be attributed to the modification.


If you use a complex waveform like music it will be impossible to calibrate both levels unless you do a RMS level calculation on the spectrum the only problem with that is if there are any differences in the spectrum you will have changed them by changing the level "Fletcher munson curve" so a single tone must be used.

Remember we are just trying to electronically calibrate the input sensitivities.. so that both sets of devices under test are the same if both dont respond the same to 1k I say you have serious problems with one of the units and the test should be scraped.


Chris

[Church-Audio]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

RMS levels should match but peak levels *should* also..  When doing comps I tend to match peak levels and then expect the rms levels to also match.   If they don't then something's generally awry.

Your ears respond more to the average signal power level than they do to peaks.  That's the reason you can run a limiter and not severely affect the way the audio sounds.  Matching the rms levels is the way to go, in my opinion.  So, I would suggest matching rms levels and expect the peak levels to match also.  It's harder to do and still avoid running one of your peaks into the rail, but it really does help to avoid the problem where one source is preferred to the other due to signal power differences between the two sources.  If you match rms voltage levels, you automatically match the signal power levels, assuming you are playing both sources back on the same system.

No need to match levels if you run 1k through the gear before you run your tests. Then the signal chain is calibrated differences between left and right are a direct product of the mics and should be ok since these discrepancies will be on Both sets of recordings since the same pair of mics are being used.

Chris

[it-goes-to-eleven]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

RMS levels should match but peak levels *should* also..  When doing comps I tend to match peak levels and then expect the rms levels to also match.   If they don't then something's generally awry.

Your ears respond more to the average signal power level than they do to peaks.  That's the reason you can run a limiter and not severely affect the way the audio sounds.  Matching the rms levels is the way to go, in my opinion.  So, I would suggest matching rms levels and expect the peak levels to match also.  It's harder to do and still avoid running one of your peaks into the rail, but it really does help to avoid the problem where one source is preferred to the other due to signal power differences between the two sources.  If you match rms voltage levels, you automatically match the signal power levels, assuming you are playing both sources back on the same system.

No need to match levels if you run 1k through the gear before you run your tests. Then the signal chain is calibrated differences between left and right are a direct product of the mics and should be ok since these discrepancies will be on Both sets of recordings since the same pair of mics are being used.

That incorrectly assumes the frequency response of the gear under comparison is identical, including during transients.  Just because the levels match at a constant 1k doesn't assure they'll match at 50hz.  And it isn't practical on live sources.  Even if you calibrate that way during the recording, final differences still need to be measured and tweaked.

[Church-Audio]

I think matching RMS levels between samples should be the number one thing done before any audio comparison.

RMS levels should match but peak levels *should* also..  When doing comps I tend to match peak levels and then expect the rms levels to also match.   If they don't then something's generally awry.

Your ears respond more to the average signal power level than they do to peaks.  That's the reason you can run a limiter and not severely affect the way the audio sounds.  Matching the rms levels is the way to go, in my opinion.  So, I would suggest matching rms levels and expect the peak levels to match also.  It's harder to do and still avoid running one of your peaks into the rail, but it really does help to avoid the problem where one source is preferred to the other due to signal power differences between the two sources.  If you match rms voltage levels, you automatically match the signal power levels, assuming you are playing both sources back on the same system.

No need to match levels if you run 1k through the gear before you run your tests. Then the signal chain is calibrated differences between left and right are a direct product of the mics and should be ok since these discrepancies will be on Both sets of recordings since the same pair of mics are being used.

That incorrectly assumes the frequency response of the gear under comparison is identical, including during transients.  Just because the levels match at a constant 1k doesn't assure they'll match at 50hz.  And it isn't practical on live sources.  Even if you calibrate that way during the recording, final differences still need to be measured and tweaked.

This is my take on it.

1- Put 1k tone in and measure output of device under test. Or three tones at 1k 50hz and 10k
2- play source record source
3- Listen to source
4- evaluate differences

If you go and change RMS levels between samples you are fooling with the spectral differences between the samples. And now you might be erasing the differences between the samples and making them sound more the same.

If there are differences between the samples then say you have a bump on one device under test at 1k to 5k you the level of that sample will be louder if you go and change the rms level so the other sample is same amplitude then you lose your ability to hear the bump as well as you would have before. If you use a standard frequency to calibrate the inputs then you have a level playing field and you can now assume any differences are the MOD it self and not a side effect of rms level calculations and the resulting normalization that is applied to the wav file.

This is my take on it I agree level influences quality or perceived quality but we must make sure that we use ether a single 1k tone or say a 50hz 1k and 10k tones to make sure both decks or devices are even in basic level so we can hear the differences if there are any in the audio spectrum.

I do believe that these tests should be backed up with FFT measurements to make sure there is no dips in the test frequencies as this could be used to "fool" my test. But that these results should be viewed after the files are listened to and voted on.


Maybe I am wrong but we are just trying to make sure the gain settings are all equal after that we want to be able to hear the differences between the devices under test if that is so if you apply RMS and normalization you are changing the files and any spectral differences between the two files could be changed not to mention that fact that you might also be increasing the harmonic distortion of the file being "tweaked" with out even being able to see it but you might be able to hear it, Now you have changed the amplitude and distortion!

Chris

[Church-Audio]

That incorrectly assumes the frequency response of the gear under comparison is identical, including during transients.  Just because the levels match at a constant 1k doesn't assure they'll match at 50hz.  And it isn't practical on live sources.  Even if you calibrate that way during the recording, final differences still need to be measured and tweaked.

Assuming we are talking about preamps or recorders here, anything that doesn't test for flat frequency response from 20Hz to 20kHz* within +/-1dB should be disclosed as such in its specs, which is easily verifiable under test.  Transient response is not a huge issue here, normally you'd test frequency response with white noise, which is constantly transient.  You can also test with a frequency sweep, or several sine waves.  All of these should yield the same result in a reasonable quality amp.

For microphones, of course, that is not true at all, except for calibrated measurement mics.



* Filter behavior of a digital converter at 44.1kHz may result in slight attenuation of frequencies above 16kHz.  That is interesting to test and note for a particular device, but if we are concerned with its analog performance up to 20kHz, set the device at any higher sample rate.

Actually white noise is perfect most meters can see it correctly and it solves all of the issues I had with using music as a means of calibrating the device under test. Good idea... You can use a cd player and a set of patch cables into the xlrs on the preamp and press play then set the levels on the meter you can use a compressed white noise so its more constant less likely to jump around on the meters. Again we are not setting level we are just setting overall balance between both sets of devices under test.


Chris

[Ozpeter]

So long as the levels are set correctly, there is no difference to the amplifier whether its incoming analog signal is from a mic or previously recorded source

Which I think means that there would indeed be no major objection to my simple suggestion of using a mic-level pre-recorded source (eg CD player into Mackie mixer with mic-level output into recorders under test) as a test bed.  Of course you'd have the CD player's imperfections and the Mackie imperfections being heard, but they'd be heard on both recordings.  Must get around to that simple R-44 vs H2 test I promised several pages back!

The only time I did any kind of on-site comparison of kit was when I recorder one half of a classical chamber concert using a Sennheiser MS pair into a Motu Traveler (who preamps seem to be rated as quite good) into a laptop via firewire, and in the second half I substituted a Phonic firewire mixer whose preamps have no street cred at all - but personally I can't hear any great problem with them.

Listening afterwards, I couldn't pin down any significant difference between the two concert halves - in fact now I think I've forgotten which was which and certainly can't say that one is better than the other (and is therefore presumably the Traveler).  That included listening to the quality of background noise between movements as well as the music and the applause.  Of course they performed different works in each half, and so you can't compare like with like, but if there was a significant difference, you'd still hear it on such a test.

Of course it could be that there were differences and that I'm too deaf to hear them.  But that particular test left me quite happy with the humble Phonic mixer and with no desire to purchase the Traveler instead.

Maybe that would be a reasonable and practical way of testing under real world conditions.  Simply record one half with one set of kit, and the second with the comparison set, but use the same mics and don't move them.  One could then listen to bits of each and see if there was anything that would enable one to say which bits came from which kit set, A or B.  Then say whether you preferred A or B.  But saying that A or B was the best (= the most accurate) would be another whole layer of complication, unless there was a gross inadequacy in one of them.  How would you judge?  One might sound very sexy but might not actually be true to the original experience through your ears.

Which reminds me of another bit of testing I did - this involved some tests of a Sennheiser MKH series mic, a Naiant mic, an LSD2, and a Gefell studio mic.  They were each put up in from of a pair of Genelec monitors in a pro recording studio control room, recording the same prerecorded piece onto a multitrack via a Grace 8 channel preamp. so you could play back switching rapidly between the different versions by hitting each solo button in turn.  What I now recall was that the LSD2 sounded preferable to the others until one compared it with the original recording - the LSD2 added something nice, but not something accurate.  The Sennheiser sounded rather dark even though it's what I use for all my recordings in a main pair - but actually quite accurate as you would expect when compared to the source.  There was remarkably little to choose between the Naiant and the Gefell - I mean, a difference, but not much when you compared the price.

However, I'd still not base a purchasing decision on such a test unless I had no choice.   For whatever reason, there's no substitute for a real sound  in a real acoustic for evaluating mics.   But for preamps and recorders, using a pre-recorded source fed into the items under test at mic level seems to me to be a reasonable method.