Topic: Stock FR2LE vs. Busman T-Mod FR2LE - Blind listening experiment

[digifish_music]

This thread is an off-shoot of Stock FR2LE vs. Busman T-Mod FR2LE true side by side comparison.

The question ...

Can people really hear the difference between the two preamps A and B?

In identifying the modded and non modded recordings, there were 2 x 'two alternative forced choice' decisions possible. In this case there was a 25% chance of someone correctly picking the same mic preamp for both songs. That means 1 in 4 people here would (by chance alone) choose consistently. Too likely IMO, so I have set up a more objective test -

I took the Cold hard facts recordings  -

Cold Hard Facts-a 24-44.flac = mic preamp A
Cold Hard Facts-b 24-44.flac = mic preamp B

as uploaded by tenesejedd, since this was the song that several people said the differences between the recordings were most obvious.

I then made 8 files (24 bit wav) so that each file comes from a unique section of the song (CHF_1.wav is from the start through to CHF_8.wav, from the end). Within each file the same few bars are repeated as source A plays then source B OR source B plays then source A. The files are named -

CHF_1.wav
CHF_2.wav
CHF_3.wav
CHF_4.wav
CHF_5.wav
CHF_6.wav
CHF_7.wav
CHF_8.wav

So, in each file the mic preamps will appear in either of the following orders - 

A-B or B-A

...your mission, should you accept it, is to post for all 8 files, if you prefer the first or second half for each file, or which half is clearer or whatever criteria you decide, in the following fashion -

CHF_1.wav = X
CHF_2.wav = X
CHF_3.wav = X
CHF_4.wav = X
CHF_5.wav = X
CHF_6.wav = X
CHF_7.wav = X
CHF_8.wav = X

...where X = 1 or 2 depending on your preference, where 1 means you preferred the 1st half or 2 the 2nd half. There are no tricks each file consists of samples from both source recordings and the order has been randomized.

Probabilities

Given the task, the probability of choosing recording A or B 'exactly' the following number of times is -

0 = 0.003 (0.3%)
1 = 0.031 (3.1%)
2 = 0.109 (10.9%)
3 = 0.218 (21.8%)
4 = 0.273 (27.3%)
5 = 0.218 (21.8%)
6 = 0.109 (10.9%)
7 = 0.031 (3.1%)
8 = 0.003 (0.3%)

I'll just explain this a little more. For example, choosing preamp A - 0 times is the same result as choosing preamp B - 8 times. If A was chosen 7 times B must have been chosen 1 time, A - 6, B - 2 and so-on. In this way the probabilities are symmetrical about 4 times for either preamp. On this basis, I'd say anyone who chooses the same preamp 6 times is 'interesting', as we would expect only ~1 in 10 people to pick 6 A's or B's by chance alone, however, to be confident I'd nominate 7 or more as a convincing display of the ability to hear the difference between the preamps A and B.

This test is interesting in that it does not matter which you prefer, it only tests the hypothesis that you can consistently identify the same preamp, at a level that is demonstrably better than chance.

You can download the files in the following .zip

TapersSection_CHF_experiment.zip ~32 Meg

If people can clearly distinguish the modded and non-modded sound AND have a preference for one of them, they should clearly pick either source A or B all the way through the list.

You can download the answer key here (password protected) ...

Click here to download the answer key

I will post the password to the answer-key .zip file in a couple of days.

EDIT: ihavegoldenears (don't peek until AFTER you listen! )

Even if you don't want to post your results publicly, do write them down somewhere so you can check how you did when I do release the answer key.

digifish

[willndmb]

i don't get it
part of each file is from source a and part is from source b
we tell you which part is first and second

[digifish_music]

i don't get it
part of each file is from source a and part is from source b
we tell you which part is first and second

Yes correct. You got it

Actually I have changed this so that the tasks is simply to pick a preferred recording from the two halves. If people can clearly identify one source over another they should be consistent in picking the A or B source.

[stevetoney]

I listened to samples 1, 2, and 3 about 5 times each and honestly can't tell the difference between the A and the B samples on any of these three, so I gave up trying to do it for the rest of the samples. 

[digifish_music]

I listened to samples 1, 2, and 3 about 5 times each and honestly can't tell the difference between the A and the B samples on any of these three, so I gave up trying to do it for the rest of the samples. 

That was my experience, but others have said they can clearly hear that one recording was more distinct and had more clarity than the other, so let's see how this goes.

digifish

[Church-Audio]

I listened to samples 1, 2, and 3 about 5 times each and honestly can't tell the difference between the A and the B samples on any of these three, so I gave up trying to do it for the rest of the samples. 

That was my experience, but others have said they can clearly hear that one recording was more distinct and had more clarity than the other, so let's see how this goes.

digifish

I too have done the test and to my shock.... I could not tell the difference I swore up and down that there was a difference but now I cant hear it. I have to eat a bit of crow here as some of my friends said I was nuts and that there was no difference. I sent it to a friend of mine who is a mastering engineer he could not hear it. So I have to say what ever I was hearing was in my head. Good test.
That is not to say that the busman mod does not decrease noise floor or give better distortion specs I simply dont know if that is the case. I still dont know what sample is what if someone could tell me in pm that would be cool.

Chris

[Will_S]

I listened to samples 1, 2, and 3 about 5 times each and honestly can't tell the difference between the A and the B samples on any of these three, so I gave up trying to do it for the rest of the samples. 

That was my experience, but others have said they can clearly hear that one recording was more distinct and had more clarity than the other, so let's see how this goes.

digifish

I too have done the test and to my shock.... I could not tell the difference I swore up and down that there was a difference but now I cant hear it. I have to eat a bit of crow here as some of my friends said I was nuts and that there was no difference. I sent it to a friend of mine who is a mastering engineer he could not hear it. So I have to say what ever I was hearing was in my head. Good test.
That is not to say that the busman mod does not decrease noise floor or give better distortion specs I simply dont know if that is the case. I still dont know what sample is what if someone could tell me in pm that would be cool.

Chris

Wow, I didn't expect any of the "vendors" would risk giving an answer that turned out to be wrong, +T for having the balls to say what you heard (or didn't).

I doubt there will be many responses to this thread...as I stated in the other thread, I could convince myself I could hear differences between A and B when I knew which I was listening to, but when I tried to match up parts blindly after chopping them up myself I couldn't.

[Busman Audio]

This test is not valid considering I make these mods to perform better for concert recording and not a home stereo recording of a studio album.  There is a lot more to mods than you are considering.  The SPL levels of a concert produce many contributing factors to the differences the mod will give you from a stock recorder. I would appreciate if you are going to try to pick apart my business that you test other mods out there also and do it in the environment that these are meant to be used for.

THANK YOU TO MY CUSTOMERS

Harmonics and harmonic distortion are a big part of how music is heard and in a concert environment this is really comes into play when you have the high SPLs coming to your mics and your pre amp. The mod improves distortion in the high and low end along with allowing better transient response to give the recorder a more life like sound adding to the realism of the recording.  "the sound of being there"  Also the soundstage is improved which is hard to represent in a room with a home stereo that we do not have any idea how it is set up or the quality of the system.

I provide a service that is benefitting the audio community wether you hear it or not and I will continue to do this at a good price so I can still survive in this world and support my family. i will continue to develop new products and better mods and these will make a difference just like all of my past products have.

[digifish_music]

I listened to samples 1, 2, and 3 about 5 times each and honestly can't tell the difference between the A and the B samples on any of these three, so I gave up trying to do it for the rest of the samples. 

That was my experience, but others have said they can clearly hear that one recording was more distinct and had more clarity than the other, so let's see how this goes.

digifish

I too have done the test and to my shock.... I could not tell the difference I swore up and down that there was a difference but now I cant hear it. I have to eat a bit of crow here as some of my friends said I was nuts and that there was no difference. I sent it to a friend of mine who is a mastering engineer he could not hear it. So I have to say what ever I was hearing was in my head. Good test.
That is not to say that the busman mod does not decrease noise floor or give better distortion specs I simply dont know if that is the case. I still dont know what sample is what if someone could tell me in pm that would be cool.

Chris

Wow, I didn't expect any of the "vendors" would risk giving an answer that turned out to be wrong, +T for having the balls to say what you heard (or didn't).

I doubt there will be many responses to this thread...as I stated in the other thread, I could convince myself I could hear differences between A and B when I knew which I was listening to, but when I tried to match up parts blindly after chopping them up myself I couldn't.

As I previously noted, perception is a reconstructed & anticipatory process (this can't be over-stressed). Just as we see animals/faces in clouds, we can hear what we want in a sound if we listen to it long enough (even your washing machine will start talking to you ). That is, what we perceive is the combination of expectation (based on past experience, knowledge etc) and the real stimulus stream. Nothing beats blind A/B comparison for revealing what you can actually hear. You simply can't do a single A/B comparison (with a 50% chance) to determine what you think you should hear.

Interesting test for someone who has downloaded the file. Listen to CHF_7.wav, and focus on the off-beat banjo? strums. When I first did this I thought the second halff was noticably brighter...but it was a trick, the strums lose brightness across the section, so when the new part starts you get a sense of immediate crispness. It wasn't until I played the whole track in loop-mode I realised what was going on. However, for a while there I thought I had spotted a difference.

digifish.

[digifish_music]

This test is not valid considering I make these mods to perform better for concert recording and not a home stereo recording of a studio album.  There is a lot more to mods than you are considering.  The SPL levels of a concert produce many contributing factors to the differences the mod will give you from a stock recorder. I would appreciate if you are going to try to pick apart my business that you test other mods out there also and do it in the environment that these are meant to be used for.

THANK YOU TO MY CUSTOMERS

Harmonics and harmonic distortion are a big part of how music is heard and in a concert environment this is really comes into play when you have the high SPLs coming to your mics and your pre amp. The mod improves distortion in the high and low end along with allowing better transient response to give the recorder a more life like sound adding to the realism of the recording.  "the sound of being there"  Also the soundstage is improved which is hard to represent in a room with a home stereo that we do not have any idea how it is set up or the quality of the system.

I provide a service that is benefitting the audio community wether you hear it or not and I will continue to do this at a good price so I can still survive in this world and support my family. i will continue to develop new products and better mods and these will make a difference just like all of my past products have.

Greetings,

this test simply asks the question, can you reliably tell the difference between the modded and non modded preamp for this particular situation. What you are saying is that the preamp needs to be driven harder for this particular mod to be heard? Then, wouldn't turning up the stereo, using more sensitive mics, moving the mics closer to the speakers (or a combination of the three) achieve the same input levels?, after all loud concerts are created by speakers too.

Regardless of the validity of this test for the particular mod, we had a bunch of people here thinking they could hear differences between the recordings presented. I am just interested to know if they were really hearing something or not, because as hard as I listened I couldn't hear anything, while others would have confidently said otherwise.  What I am hoping to do, is to raise the bar a little, if we are to compare recordings then the method I have applied above is (IMO) a better test and not hard to construct.

I also agree this test has nothing to do with any of your other mods.

digifish

[page]

I would appreciate if you are going to try to pick apart my business that you test other mods out there also.

QFT, lets grab a stock unit, and an Oade unit and repeat all of these tests. I don't think favoritism (whether good or bad) should be shown in the search for answers, particularly when the results can have an economic impact.

What you are saying is that the preamp needs to be driven harder for this particular mod to be heard? Then, wouldn't turning up the stereo, using more sensitive mics, moving the mics closer to the speakers (or a combination of the three) achieve the same input levels?, after all loud concerts are created by speakers too.

That was one of the things mentioned in the other thread (overdrive). It may be that we are looking for A and B, while the effects are primarily centered around C (which we didn't test for). We know that some people can tell a difference, and some can't. We know that the noise floor is affected. We haven't seen what overdrive will do.

[Will_S]

I would appreciate if you are going to try to pick apart my business that you test other mods out there also.

QFT, lets grab a stock unit, and an Oade unit and repeat all of these tests. I don't think favoritism (whether good or bad) should be shown in the search for answers, particularly when the results can have an economic impact.

I agree...so far the folks doing the tests only seem to have had access to Busman modded units.  If anyone wants to loan me their Oade-modded FR2LE, I'd be glad to make some comparison recordings with it.

Although as a practical matter, pretty much anyone who gets an Oade-modded unit bought it from him in that state, since he won't mod decks purchased from others.  So in terms of community interest, there are probably far more people considering a Busman mod and looking for an objective description of how it differs from stock than there are people considering an Oade mod.

We know that some people can tell a difference, and some can't.

We don't really know this.  Some people claim to hear a difference, but no one has shown they can tell the samples apart in a blind test.

We do know that when hugely amplified, the background noise on the two tracks sounds different, but it really sounds to me like a fan or a refrigerator or something is switched on in another room for at least one of the tracks - the difference in both the level and tonal quality of the background noise is MUCH more different in the Pink Floyd tracks than it is in the Del McCoury tracks.

[Church-Audio]

This test is not valid considering I make these mods to perform better for concert recording and not a home stereo recording of a studio album.  There is a lot more to mods than you are considering.  The SPL levels of a concert produce many contributing factors to the differences the mod will give you from a stock recorder. I would appreciate if you are going to try to pick apart my business that you test other mods out there also and do it in the environment that these are meant to be used for.

THANK YOU TO MY CUSTOMERS

Harmonics and harmonic distortion are a big part of how music is heard and in a concert environment this is really comes into play when you have the high SPLs coming to your mics and your pre amp. The mod improves distortion in the high and low end along with allowing better transient response to give the recorder a more life like sound adding to the realism of the recording.  "the sound of being there"  Also the soundstage is improved which is hard to represent in a room with a home stereo that we do not have any idea how it is set up or the quality of the system.

I provide a service that is benefitting the audio community wether you hear it or not and I will continue to do this at a good price so I can still survive in this world and support my family. i will continue to develop new products and better mods and these will make a difference just like all of my past products have.

Nobody is picking apart your business. I don't agree with the way alot of these tests are done I don't agree with anybody that does mods that cant provide technical data to back them up. I dont care who it is its not personal its just one opinion. And I don't care who is doing the mods I would say the same thing. I have the benefit of really understanding electronics and I also know alot about modifications I do them all the time to high end recording consoles and live consoles. But I also provide before and after data for my customers. Anyway audio is very subjective we are never going to agree with everything. I think you do provide a good service and I wish you nothing but success I would have said the same thing no matter who's mods were being discussed so no need to take it personal. 

[digifish_music]

That was one of the things mentioned in the other thread (overdrive). It may be that we are looking for A and B, while the effects are primarily centered around C (which we didn't test for). We know that some people can tell a difference, and some can't. We know that the noise floor is affected. We haven't seen what overdrive will do.

While that may be true, I will return to the original issue, quite a few people claimed the differences were audible for the files posted. It seemed that people were quite happy to take these testimonials as evidence that there were differences between the A and B recordings. Think about this - when the results are made public of the first trial 50% of the people will have a result that supports their initial perception (by chance alone), and it will be very difficult to convince them that they were guessing.

So, I am simply running this to ground in a more controlled way, and as yet, no-one has offered their list of 1's and 2's for the files I posted.

BTW: this is not an attack on Mr Busman & his mods, it's about the validity of one-off testimonials based on listening to single recordings posted in this forum, and of this A vs B test in particular. I am hoping to learn whether or not there were differences or people were imagining things.

digifish

[page]

We know that some people can may be able to tell a difference, and some can't.

We don't really know this.  Some people claim to hear a difference, but no one has shown they can tell the samples apart in a blind test.

Misstep, my bad. Carry on.

[illconditioned]

Try recording live music and/or ambient sounds.

To Busman: Could you put that in your kitchen and record the same mics modded and umodded?  I think you'll hear more differences there.

I would offer to do this, but I no longer have an umodded unit!  I took the kool-aid and modded mine straight away  ,

  Richard

[Ozpeter]

And don't forget to keep your head in one place when listening to the test materials.  A slight movement of the head in the reverberant field of the listening environment is often enough to significantly change the listener's perception of what is being heard, due to the interactions of the sound waves bouncing round the room.  It is very, very difficult to conduct successful and valid listening tests unless the difference between the two systems under test are really significant, so that which is which is blindingly (or deafeningly) obvious.

I've got nothing against people going for these mods, and I wish anyone who undertakes them well - and in these tiny units, I greatly admire their brain-surgeon-like skills - but I would hope that people would say "I think it's better" or "I prefer it" rather than "it is better, as a matter of unchallengeable fact".

[chris319]

I swore up and down that there was a difference but now I can't hear it.

This supports something I have long suspected -- that there is a great deal of placebo effect in subjective listening tests.

The mod improves distortion in the high and low end along with allowing better transient response to give the recorder a more life like sound adding to the realism of the recording.  "the sound of being there"  Also the soundstage is improved

If you provided measurable, reproducible test results before and after your mods I would be more inclined to see the value in your service. The first thing I want to see is that your mods don't degrade the specs of a stock unit. There are free audio analysis programs available so cost is not the obstacle.

Your competitor offers a general description of what is done in the modification process. On your web page I see no such thing.

I provide a service that is benefitting the audio community wether you hear it or not

Well if you can't hear the difference ...

I too don't mean to "pick apart" your business, but if you're going to charge money for a product or service it should stand up to scrutiny IMO, and I applaud what digifish is doing. OTOH, if the people who pay for your mods are happy with them and feel they've received their money's worth, that's fine and dandy.

[Belexes]

I like my Busman mod and to me "it sounds better."  I can see why someone would opt not to get it given these results, but I personally have no regrets.

[page]

And don't forget to keep your head in one place when listening to the test materials.  A slight movement of the head in the reverberant field of the listening environment is often enough to significantly change the listener's perception of what is being heard, due to the interactions of the sound waves bouncing round the room.  It is very, very difficult to conduct successful and valid listening tests unless the difference between the two systems under test are really significant, so that which is which is blindingly (or deafeningly) obvious.

I gave up trying with speakers and use my IEM headphones instead.

If you provided measurable, reproducible test results before and after your mods I would be more inclined to see the value in your service. The first thing I want to see is that your mods don't degrade the specs of a stock unit. There are free audio analysis programs available so cost is not the obstacle.

Your competitor offers a general description of what is done in the modification process. On your web page I see no such thing.

IMHO, a general description is paltry, and if you arn't publishing specs and test results then to me you are saying nothing anyway. For example, Busman Audio could publish a description, but it doesn't change his mod, we are still back to the "can you hear the results" debate. Test results remove the largest barrier to know what the mods are doing, and I don't see Oade doing that either.

I can see why someone would opt not to get it given these results, but I personally have no regrets.

I don't regret getting mine done, but I use my unit for quiet material (nature recordings, cats, etc) on occasion and thats where I can hear a marked difference.

[willndmb]

i hear no differences
i did hear a difference in the vocals a little bit in the other comp where they were complete

[digifish_music]

I like my Busman mod and to me "it sounds better."  I can see why someone would opt not to get it given these results, but I personally have no regrets.

I'd just like to remind everyone this is a particular mod category on a particular brand of recorder, Busman makes quite a few mods on various models, so let's not generalize this particular case to all modding. Given that no-one is offering up preferences I will reveal the password to the TapersSection_CHF_experiment_Key.zip file.

ihavegoldenears



Something to do: Have a listen to the recordings again, if you know whether A and B was modded (I still don't) while looking at the answer key...can you hear differences now? Possibly...if you can, what does that tell you?

My main purpose here was not to jump up and down all over Busman, (or Oade) but to hopefully 1. find out if there was a difference in the recordings posted, and 2. perhaps teach the group something about perception.

I couldn't hear any differences, and I have been mixing and critical listening to music for a while now, so I was pretty sure there wasn't anything 'noticeable' as people were claiming in the original thread. Indeed the responses we have seen strongly suggests there was no audible differences between recordings from the two samples posted.

Let's get away from the 2 file, single decision comparisons. You pick the modded unit in blind listening once, that's chance, you do it 7 or more times, that's probably real.

digifish