This is correct. It's the relative difference in any two clocks that determines how long it will take for the difference to become noticable.
Interstingly, if you playback two sources simultaneously independantly, using the same machines the recordings were made on, and manually sync the two at the begining, they will stay in that relative sync for a very, very long time. In that case the difference in timing between the two clocks is self-compensated for the most part since the same clock is being used for recording and playback, leaving only the absolute variation within each clock to cause things to drift, and that is miniscule.