If you're doing a lot of headphone listening, read up on super-stereoism.
It is the opposite of binaural recording. Two track is designed (usually) for speaker playback.
Hence in headphones the channels are too isolated for long, confortable playback.
A very good headphone amp will slightly mix left and right channels for simulation of "open-air" leakage.
I'm still learning about psychoacoustics, so take my interpretation as a launchpad for getting solid info.