noam, just as a thought experiment, if you have a recording that gives you listener fatigue, and while you're playing it back I come in and make a (hypothetically) perfect recording of that sound, then presumably you would also get fatigue from listening to my recording through a perfect playback system.
What this tells me is that unfortunately, there probably won't be any one clear, authoritative answer to your question since it can be due to something in the original sound, in the recording method or recording equipment, in the way the recording is processed before it is played back, and/or in the playback equipment.
For example, a radio announcer could be speaking into a high-quality cardioid microphone that has some proximity effect, causing a moderate exaggeration of the low and low-mid frequencies of his voice. This could then be going through a board that has the gain set too high, causing the radio station's compressor to kick in (raising the average levels).
And then this signal could be going through another, faster-acting multi-band compressor, and my upstairs neighbors could be listening to it with their speakers standing directly on the floor, and with the "loudness" switch on their receiver turned on because they don't know what it's for. Meanwhile I'm beneath them and all I hear is the energy below about 250 Hz from the announcer's voice, at an almost constant level and yes--I'm getting pretty fatigued by that.
By itself, no one link in that chain is inherently evil or technically inappropriate; it's the succession and the combination of all those things which creates the aggravation.
But most people, I think, think of distortion (particularly with IM products in the 1-to-several-kHz region, as someone else suggested) as a big culprit, too. Basically whatever people do to make their songs "grab your attention" can easily become the very same things which make them tiring to listen to.
--best regards