Acoustically it won't matter either way - regardless of whether the recording was made with omnis, spaced or baffled, or directional micrphones, coincident or spaced.. or is a soundboard patch. this is no different than flipping your head-phones around, or swapping the cables between the two speakers.
Psychoacoustically it can matter though. It can of course if the listener was present at the performance and retains some memory of how things were arranged, or simply how it sounded. Or if the listener was not there but there is some expected orientation, such as the drummer always being off to one side and the organist on the other or whatever. Orchestra recordings are like this. most often with the violin sections to the left and the cello and bass to the right.
Time differences create stereo imaging with omnis, so, personally, I think it matters. If they are baffled (by your head, for example), that makes it even more important.
Because the same time of arrival delay is retained if one swaps channels, it doesn't matter. You are correct that time of arrival information is an important aspect imparted by any non-coincident stereo microphone arrangement - with any recording made with acoustically-significant spacing between stereo microphone pair, some time of arrival difference between channels will occur for any sound sources that are not centered, or more specifically, for any wavefront arrival that is not perpendicular to the medial plane. The [edit]
potential confusion here is drawing a connection between what are two different issues.