I don't know what the tolerances are for running that digital signal around but in terms of which wires to keep short, I'd go with the jumper that runs from the ADC. The USB controller might be the better choice for the longer wiring - that way if there are any noise effects, they'll occur on USB playback and not your standalone recording.
Also, you want a DPDT switch - it can simulataneously turn one off while turning the other on.