I've done several sync's with alternative audio sources, aud's and SBD releases. I usually use the cam aud as a guide to sync the alternate audio source. But I have never noticed any drift. There have been several instances where I thought there was drift, when in fact it was actually due to dropped frames during the transfer, resulting in a loss of audio samples as well. Most of the time the give away is the fact that the difference (or drift in your case..maybe) should be getting longer as the set progresses. I have also run into a couple situations where the sbd release did several seconds of overlap between discs that was not noticable, and conversely, a case where the release actually cut out seconds of banter/tuning in between tracks. All of the above scenarios totally threw me off and took me days before I realized what was really going on.
I don't have a solution for any of you, just thought I'd point out some possible scenarios of where things are going off. But I do believe drift happens, I noticed when I was matrixing phish auds and sbds before.