Topic: Taperssection - product listening comparisons *gold standard*

[RobertNC]

One thought that came to me while I matched a batch of mics was how many of those who feel that audio is entirely subjective and gear must be tested on a live source, etc. own a matched pair of mics?  Do you feel that matching is important?  How do you think matching is done?  How would you do it?

I own both.  I feel like matching is a good benchmark, but less and less important as mic quality increases.

As far as how it is done, of course it is a measured response of two microphones to a single controlled signal with "matching" being accepted based on an acceptable statistical range of variations of two mics to a single source.  That is scientifically a basic principle of calibration.

It tells you that a pair of microphones will perform within an acceptable range under the same circumstances - the matching source circumstances.  If the matching charts I received represzent the entire data used to accept a "match" you don't even have enough data to extrapolate that they will therefore perform essentially the same in the field, and you have absolutely zero data about how your mics will perform in the field versus a living room - that is a totally different set of parameters.

[it-goes-to-eleven]

Let's say I want to buy a digital camera for a big vacation and want to get really good pics.  I could compare two models by displaying a picture of Yosemite on my LCD and taking a picture with each camera.  I might see some differences or I might not.  But regardless, that test would probably not tell me which camera would take better landscape pics on my vacation.

However, when I bought my Gefells the first thing I did was set them up and record in the living room.  The  I set my ADK LDs up exactly same and recorded the exact same things. Now these are really different microphones, not just in design, but in quality too.

I listened to the results numerous times.   My final analysis was that I was really glad I had a real show to tape the next week.  Because after this little experiment, I was thinking wow, that was a shitload of money, and I can hear a difference but just barely.  I sure hope I did not make that kind if purchase to essentially mostly lower my profile.

Same experience here.  I've owned or own C4's, DPAs, AT853's, MG200/210, Schoeps mk21/mk4/mk41.  The difference between the C4's and the higher end mics in front of my VR4's or Martin Logan CLS's is very small.   But in the field, the difference is far more dramatic (especially between cheap hypers and great hypers, but that's an extreme example).  Same can be said of my many pre-amps in front of speakers.

Again, I'm not saying the methodologies being proposed are useless. Far from it. These discussions are great. Though some here seem to be dismissing the significant, and often critical differences between a 'lab test' and the real world.

And, fwiw, for the past 20 years I have done quite a lot of test design.  Think thousands of result graphs over the past 10 years and individual test projects that saved $30 million in a single quarter.  A number of folks are using my datasets for their research papers and PhDs.  Just saying that methodology, testing, visualization and comparative analysis are more than hobbies for me.  But tapin' is a hobby

[Church-Audio]

Let's say I want to buy a digital camera for a big vacation and want to get really good pics.  I could compare two models by displaying a picture of Yosemite on my LCD and taking a picture with each camera.  I might see some differences or I might not.  But regardless, that test would probably not tell me which camera would take better landscape pics on my vacation.

However, when I bought my Gefells the first thing I did was set them up and record in the living room.  The  I set my ADK LDs up exactly same and recorded the exact same things. Now these are really different microphones, not just in design, but in quality too.

I listened to the results numerous times.   My final analysis was that I was really glad I had a real show to tape the next week.  Because after this little experiment, I was thinking wow, that was a shitload of money, and I can hear a difference but just barely.  I sure hope I did not make that kind if purchase to essentially mostly lower my profile.

Same experience here.  I've owned or own C4's, DPAs, AT853's, MG200/210, Schoeps mk21/mk4/mk41.  The difference between the C4's and the higher end mics in front of my VR4's or Martin Logan CLS's is very small.   But in the field, the difference is far more dramatic (especially between cheap hypers and great hypers, but that's an extreme example).  Same can be said of my many pre-amps in front of speakers.

Again, I'm not saying the methodologies being proposed are useless. Far from it. These discussions are great. Though some here seem to be dismissing the significant, and often critical differences between a 'lab test' and the real world.

And, fwiw, for the past 20 years I have done quite a lot of test design.  Think thousands of result graphs over the past 10 years and individual test projects that saved $30 million in a single quarter.  A number of folks are using my datasets for their research papers and PhDs.  Just saying that methodology, testing, visualization and comparative analysis are more than hobbies for me.  But tapin' is a hobby

Here is the problem.. You cant use two sets of mics to feed the signal chain. Because if you move the mics even an inch they will not sound the same. So unless we are willing to use an active splitter how can you record a live performance with one set of mics and split it into two signal chains.. If you cant do that then you cant have a 100% objective test.

That is the problem. Arguments could be made that an active splitter does not allow for the same type of load a pair of mics present to a preamp. So then you have problems with your signal chain not being 100% accurate between the two sets of signal chains and a Y cable presents the same problem of loading.

So the only way to use one set of mics is to have a source that can be played back precisely two times with only the device in question being switched out.

This is not rocket science. I build and test mics for a living and I am saying it is 100% impossible to have two sets of mics that sound exactly the same and its impossible to put two sets of mics in the same exact space. So you then introduce a guessing game of is it the fact that we have two sets of mics that are changing the sound or do we have a mod that is changing the sound making any meaningful results null and void.

To be fair we have to use a speaker or set of speakers in a fixed position to do a test like this. One way to do this might be to use a live venue with a cd source or prerecorded live show played through the PA to simulate the nuances caused by room acoustics of course not all of the complex reflections will be there because we dont have a complex sound source on stage adding to the total sound.


but how does this complexity translate into a situation where they "the complex reflections" must be there in order to evaluate gear? Nobody has answered that question as of yet to my satisfaction.


In the end sound is sound you can move big speakers into a big space and get a big sound. Or take them into a small space and get a more compressed sound but you still have something to listen to.

Two sets of mics = unfair evaluation of the gear in question its just that simple. The best test is live recorded music played through a good set of speakers how else do you get the same "exact" signal into two sets of signal chains? THE MILLION DOLLAR ANSWER IS YOU CANT this is not even grade 8 science this is basic understandings of the principals of sound and of microphones.


Chris

[RobertNC]

No I was saying I am definitely not a sound engineer, and can only offer my anecdotal opinions without any real scientific basis.  I am a chemical engineer, which is relevant only in terms of understanding the need for controls and experimental design.  But ChE is also by far the least scientific and most empirical engineering discipline, and coming from a heavy empirical background, basically what you are saying is what I am saying.  Yes, the microphones - in purely physical terms - respond the same - they know nothing about where they are.  But the empirical part is where I see the problem - the translation to real world performance.

If I was in a meeting and someone told me "we have carefully selected every component carefully tested and evaluated under strictly controlled uniform test conditions", I would say "And have you constructed a plant on that basis?"  When they said "Yes, absolute state of the art", I would say, "Sorry for having to leave so abruptly gentleman, but I really need to get moving and hopefully be in another timezone before you open the first valve."   

[Church-Audio]

No I was saying I am definitely not a sound engineer, and can only offer my anecdotal opinions without any real scientific basis.  I am a chemical engineer, which is relevant only in terms of understanding the need for controls and experimental design.  But ChE is also by far the least scientific and most empirical engineering discipline, and coming from a heavy empirical background, basically what you are saying is what I am saying.  Yes, the microphones - in purely physical terms - respond the same - they know nothing about where they are.  But the empirical part is where I see the problem - the translation to real world performance.

If I was in a meeting and someone told me "we have carefully selected every component carefully tested and evaluated under strictly controlled uniform test conditions", I would say "And have you constructed a plant on that basis?"  When they said "Yes, absolute state of the art", I would say, "Sorry for having to leave so abruptly gentleman, but I really need to get moving and hopefully be in another timezone before you open the first valve."   

As soon as I turn a piece of gear on I expect the "real world performance" to be the same no mater where I am if not then its crap gear.

I am not talking evaluating the gear in a lab I am still talking about using the things on ether side of your head. If we were strictly talking science here I would want lab tests on the gear as well but since were are not talking strictly lab tests here your argument does not apply. The simple fact is gear should function the same no mater where its placed. And that is a fact would I want a chemical plant setup based on some imaginary test conditions hell no... But we are talking about two sets of speakers in a living room vs a night club please explain the differences to me is the the lack of clanking beer bottles that makes my idea invalid? or the drunks yelling in the background. There are no environmental differences between some clubs and your living room... EXCEPT SIZE... and I have been and done sound at a few places that would make your night club look like a living room.......

[Busman Audio]

Acoustics make a huge difference on what your microphones pick up and how your gear sounds to your ears. If this doesn't make sense to you !!Wow!! 

There is no gold standard for testing as we have hashed this out so many times. 

Soundstage is the way your ears perceive the "image" of sound being produced that is how you can localize sounds made around you and how blind people see.  Its all about the ears!!!

[Church-Audio]

Acoustics make a huge difference on what your microphones pick up and how your gear sounds to your ears. If this doesn't make sense to you !!Wow!! 

There is no gold standard for testing as we have hashed this out so many times. 

Soundstage is the way your ears perceive the "image" of sound being produced that is how you can localize sounds made around you and how blind people see.  Its all about the ears!!!

You are talking about acoustics to someone that has spent half of his life studying them!! Sound in a live environment is different then a living room yes.. But if we all needed a live environment to hear the differences how can one judge the sound quality differences of a cd player? take it to a gig? 


Sound is sound you can capture it with a microphone the mic will pick up whats in the room how is that  different from the test we are talking about? it all comes back to sound is sound will live vs pre recorded sound different hell yeah but cant we listen to a prerecorded recording and judge the differences in gear? Or do we need a bigger "sound stage"? This argument is very dated I have heard it all before I think there is no gold standard because there are still alot of misinformed people who believe that the magic in the modifications they pay for only happen in at a live show....  If a mod is really good it should sound good anywhere if not how do you test all the gear you mod do you take each and every piece of gear to a live show and then ship to the customer??

I am fighting for a fair shake for all people not just for people that make money from modifications btw this is not personal this is about objectively evaluating sound so we can hear the differences between two pieces of gear. You sir with your rude comments are making in personal makes me think you have something to be upset about. Be professional dont take this as an attack on your business because its not. Its simply me stating what I think a fair evaluation method should be and somehow your translating this into an attack on you that is simply not the case.. I will say this again I dont know anything about your mods and how effective or none effective they are.


Chris

[RobertNC]

This is not rocket science. I build and test mics for a living and I am saying it is 100% impossible to have two sets of mics that sound exactly the same

If you are saying this them you are saying the whole concept is also 100% impossible.   If I buy a pair of mics based on a carefully controlled comparison, then I still did not buy THE microphones that were used in the comparison so in fact I don't have the same mics I made the decision on.

And after some field use there has been environmental and temperature exposure.  And man, even though I use windscreens religiously especially indoors at theatres for smoke screens if you've been to a Phil show recently in a small venue you know the fog of pot smoke is so dense it obscures the stage.

So by that argument I can buy a set of mics and in a short period of time no matter how much care I take with my gear (a lot, this stuff is not cheap and I'm not rich) I don't even still have the same mics I bought.

  

[Busman Audio]

You can just keep telling people you are right Chris and I will just keep telling them to listen to their ears.


sorry to offend you but I have a right to my opinion just like you do!

[Church-Audio]

You can just keep telling people you are right Chris and I will just keep telling them to listen to their ears.


sorry to offend you but I have a right to my opinion just like you do!

You absolutely have every right to your opinion. I dont agree with it and you dont agree with my opinion and that's ok. I would like to correct one thing.. I have never told anyone to not use ears as a major part of any evaluation EVER. But it would also be nice to know some of the folks doing mods are also using test gear to if anything make sure the mods they are doing are performing to the "spec" although unpublished for most is met dont you agree?
Chris

[it-goes-to-eleven]

Just as an aside, once you guys agree on the Gold Standard, I'm going to start a thread to develop the Platinum Standard

[Church-Audio]

This is not rocket science. I build and test mics for a living and I am saying it is 100% impossible to have two sets of mics that sound exactly the same

If you are saying this them you are saying the whole concept is also 100% impossible.   If I buy a pair of mics based on a carefully controlled comparison, then I still did not buy THE microphones that were used in the comparison so in fact I don't have the same mics I made the decision on.

And after some field use there has been environmental and temperature exposure.  And man, even though I use windscreens religiously especially indoors at theatres for smoke screens if you've been to a Phil show recently in a small venue you know the fog of pot smoke is so dense it obscures the stage.

So by that argument I can buy a set of mics and in a short period of time no matter how much care I take with my gear (a lot, this stuff is not cheap and I'm not rich) I don't even still have the same mics I bought.

  

Your saying your mics dont sound the same but what are you basing that on? There are lots of mics out there that dont drift very much over time the DPA 4006 comes to mind as an example.


We are not evaluating the quality of the microphones we are evaluating the quality of the modifications done to a device in the signal chain. And so since that's all we are evaluating the mics become irrelevant because the sound of the two samples comes from the same source. If you base your purchase of a mod on the overall sound of both recordings and not on the differences between the two recordings then you are wasting your money since the overall quality would be a direct product of the source in most cases. No two pairs of mics will sound the same No two pairs of matched mics will sound the same because there is no such thing as mics that are 100% perfectly matched they do not exist and it would take many many hours to find such a pair so most matching even by really good companies like DPA is based on a tolerance of anywhere from +- 3 db to + - 0.2 db. But even this might be based on sensitivity at one frequency and not broad band I would say broad band matching and finding  two identical microphones is way harder then winning the lottery.

Chris

[Church-Audio]

Just as an aside, once you guys agree on the Gold Standard, I'm going to start a thread to develop the Platinum Standard

No its the Diamond standard! 

[RobertNC]

I thought we were talking about product comparison, not just pre-amps.

In any event, I would like to address Chris's earlier comments.  First of all, I don't know your history, but I do know I have read this thread in it;s entirety and if you have found rudeness and personal attacks, I think that may be some kind of "history"

I am not questioning your knowledgeability, and this is not a personal attack, although it may be perceived that way.  I tend to be right some of the time, dead wrong some of the time, but brutally forthright all of the time, and I am used to people taking offense where none was intended.

So as far as modded stuff goes, it seems like we were talking about product comparisons but you seem to go to mods as a recurring theme.  While some of us here are openly admitting we are making subjective decisions, I have to start questioning some of your comments as being maybe not quite as objective as you seem to present them.

Having said that,  I personally will no longer touch any modded gear.  When someone asks me how I feel I give them my subjective but honest opinion.  If you are on a budget, modded gear can in my opinion genuinely give you a real upgrade with a lot of bang for your buck and put the quality of recording you can obtain on a totally different level than the same "prosumer" stock unit could provide.  There are problems however - you immediately not only void the warranty but if you have problems you have limited to resolutions, and practically never have an immediate option.  I know from personal experience.  Take modded gear and have it die on you even in a city like NY where you can find anything that exists, and you are still outta luck because anyone that would do quality work willl also not be willing to even touch a piece of non-functional modified gear.

That very unpleasant experience put an end to my willingness to run mod gear - the professional stuff is worth every penny.

I maybe wrong, but I had the distinct impression you were also in the business of modifying or at least building non standard gear yourself?   I have even steered a few people your way saying "Sorry, I don't do stealth so I can't advise but a lotta people that run stealth gear seem to have good results form this guy Church who I think provides a lotta stuff targeted at the stealth market - join taperssection and you'll find him.   

And as far as Busman goes, never had any dealings with him either, but people seem to have a high view of his mods and I've heard some good results.  And I understand the mod market exists and benefits many tapers on a budget, and I have had experience with non-stealth modded prosumer gear and I have steered people to him a number of times with the similar qualifier that I have not used his gear but many like it and unlike that other guy who's reputation is apparently beyond reproach and can rest on his laurels (i.e. to put it bluntly if you get a good mod, good for you, if you get one that has defects the other guy is not above totally fucking you over as an individual because he knows you have no real voice as far as his untarnishable rep), my impression is that Busman is OTOH actively trying to grow both his reputation and his business, so I would trust him to give better service.     

So what is with the whole mod issue and the perception of personal attacks?

Are we still all talking about differing views on product comparisons, of which modded and unmodded equipment would both be considerations, or have we moved to the realm of product business or some other personal business now?

[Gutbucket]

Discussion seems to have move on to mods and other things but here's a viewpoint on mic listening comps in particular that I typed earlier this morning-

Mics perform the same regardless of location but locations vary greatly.

There is testing procedure, there is data and there is the interpretation of that data.  We can get hung up on any of those three.

Let's keep in mind we are talking about listening comparisons not test measurements.  It can be very difficult to relate imperial measures to subjective qualities.  The measurement tool in this case is our ears and brains so we don't have to worry about relating measurements to our experience of the sound.  But we also can't make a blanket proclamation of which is objectively better or measures better.

The appropriate pragmatic testing procedure is the one that allows the user to make an informed decision as to which piece of gear will give him the results he desires in the appropriate application.   No test is perfect and we don't require a perfect test, just one that shows us which piece of gear performs better in our particular application.  A big problem is the potential for self deception.  How much rigor in constraining variables is necessary?  What is the true value of each variable.  Do I even recognize all the variables?

That problem is compounded when comparing recordings made by others, which requires answering the question, "Are her test procedures and standards the same as mine?" Answering that is difficult and requires more detailed communication and ground rules than many here are willing to commit to in order to rule out self deception.

On the applied aspect of living room testing, I'll note that not many are aiming to primarily record and reproduce the sound their living rooms, as live sound recording is the goal.  That complicates things even if we all were to agree that a recording test made in the living room could be made validly because, again, we're talking about a listening test and are going to be listening for the things we hear and value in a recording of a live event in a room that is probably very unlike a living room.  The reproduced sound in the living room is a small sub-set of the information at the actual event.
 
I hear or do not hear plenty of important things live that are either not available at all or more commonly are not reproducible in the same way via playback in my living room of my recordings.  That is not just limited to the bandwidth performance of my playback system, but things that would appear completely different from the mic's perspective at that live location.  For example, spatial information such as the off-axis ambience performance reproducing the direction and clarity of things like clinking glasses and a rattling register at the rear of the room at a jazz show; the sniffling patron in the squeaky seat two rows back at the string quartet date; the sound of the room reverberance off the coffered ceiling of the chapel; the bass modal response of the concrete walls of the club; etc. 

It's easier to make a meaningful listening assessment if we're listening to a recording made in the actual application.  We may also be able to extrapolate the results of a different test (living room or whatever) and project that on our final 'actual' use, but that is much harder and is closer in spirit and procedure to making a standardized measurement than doing a listening test.