[..]I time align in the middle of recordings or sets. So if I did it would be worse at the beginning or end?[snip]
Here's the answer (which I realize you weren't really looking for) to the above question.
Aligning in the middle of the recording without stretching or shrinking one to sync perfectly with the other will "split the difference" and cause any mis-synchronization to manifest equally at both start and end, but with only half the offset that would otherwise occur at the end if the files were aligned at the start instead (or vice versa). Because doing so effectively halves the alignment error, this makes for a good strategy to avoid stretching/shrinking, as long as it successfully keeps the mis-sync at either end from being perceivable or otherwise messing with something.
Specifically, the mis-sync will manifest as the slightly shorter file (the one recorded with the slightly faster clock reference) trailing the other maximally at the start, that error being gradually reduced until the sync point, then increasing again to the same amount at the end, but with the shorter file pulling ahead of the other in time after the sync-point.
With the different cameras I have, and different recorders and such, Ive never had to time stretch any audio to sync with video. I always use 24/48 so Im not sure if that makes a difference, but what I thought was getting lucky for a while in the beginning has become pretty common place.
Sample rate and bit-depth won't make a difference. It's probably more modern recording gear incorporating more accurate clocks, good luck, and recordings that may seem long but are not long enough for the problem to become noticeable. If a recording were to run for many many hours you would be more likely to notice it.
As Guitard noted, syncing audio and video is less problematic by two orders of magnitude in comparison to the accuracy needed for mixing two closely-correlated audio sources without phase problems occurring when the sources are mixed. If the two audio sources are not highly correlated its more like one order of magnitude before the delay will begin to become apparent. Yet recorders which include time-code have historically tended to use more accurate clocks to begin with, partly as a way of allowing for longer free-running of time-code without problems.