On most stereo microphone preamps a polarity switch, if it exists at all, affects only one channel. Each channel may have such a switch, or (often with portable equipment) only one channel has one. If there's only one switch and it affects only one channel, then it's there so that you can compensate for a miswired cable, or for mikes that are placed far enough apart to produce phase conflicts when their signals are heard together.
Not very many portable, low-cost preamps have polarity switches for both channels because, rightly or wrongly, "absolute polarity" isn't widely considered to be an issue. If you invert the polarity of both channels of a stereo recording, the very great majority of listeners (I'm talking ~90% or more) will notice no difference. Human hearing is just not very sensitive to this issue most of the time. Among those who do report hearing a difference, it seems to be mostly in the sound of a kick drum (or a bass drum in an orchestral recording). Some of these people can pass controlled, blind A/B tests quite handily, so this is definitely a real thing; it just doesn't affect most listeners and most program material. (This is apart from the possibility of a defective or overdriven woofer in one or both of your loudspeakers, which could make a rather different sound depending on the polarity of a strong signal.)
If it tells you anything, the very highly regarded (appropriately IMO) Grace Lunatec V3 preamp had inverse polarity for years, and no one noticed it until this fact was discovered by testing for it specifically--whereupon the manufacturer adjusted the circuit, and made a retrofit mod available on request. I'll bet most V3 owners didn't bother, though. And a fair amount of regular (as well as high end) hi-fi equipment has always been polarity-inverting, without the oh-so-sensitive "golden ears" in the audiophile press ever making any fuss about it.
If you're mixing signals from different sources (what people here call "matrixing") (incorrectly in my opinion), the same acoustical events reach your recorder channels with differing amounts of delay in the different sources, reinforcing some frequencies and weakening others due to the resulting phase differences. Whenever there are multiple pickups of the same acoustical event from different miking distances, this happens (e.g. it already affects the soundboard mix even before the addition of the signals from your microphones). So you may find that one setting of the switches sounds better to you than the other--but if so, it's not predictable which setting would sound better to you.
Plus you might well make a different choice if you had the luxury of monitoring over a good pair of loudspeakers as opposed to headphones. Mixing signal sources live in this way, when you can't possibly have a good monitoring setup, is always a huge gamble, and I am frankly surprised that anyone would do it. Recording the soundboard signals on a separate channel or pair of channels is a whole other matter--you can pick and choose afterward, and decide whether some form of mixture improves the recording or not, and make adjustments at your leisure. In that situation, the polarity of the signals from the soundboard is definitely one variable that you'd want to play with.