Actually, a small but significant minority of listeners can detect absolute polarity of audio signals at low frequencies. For them, inverted recordings can sound distinctly unnatural and uncomfortable to listen to, depending on what kinds of sonic events are in the recorded material (e.g. an inverted string quartet or chorus might well be no problem, but an inverted percussion ensemble or a kick drum could sound quite strange to them). It's so easy to maintain correct polarity when working with balanced microphones, etc., that I've always made a point of doing so.
I've also sometimes wondered how much this phenomenon may have skewed listening tests where it wasn't controlled for because people didn't think that it mattered, or even people's opinions of an entire medium or genre of recording. In the early days of CD, for example, ISTR that some players had outputs that were inverted from true polarity. For that matter I have no idea whether most consumer audio these days maintains correct polarity or not; some kinds of amplifier stages are inherently polarity-inverting, and as a circuit designer you have to take care about the total result; it doesn't take care of itself.
--best regards