That's the noise floor of the recording, which is likely to be audible only during the quiet parts. Likely that the noise floor you are hearing is the self-noise of the mics, but could be elsewhere in signal chain, like how the gain is set on a preamp verses the following recording stage. The first you can't do anything to prevent other than switching to mics with a lower self-noise spec, if the second, you may be able to set things differently so as to reduce the level of noise.