Yeah 24/96 almost maxes out the transfer rate of the USB 1.1 bus, which is why you can't do full duplex 24/96 - you have to set the UA5 to either 24/96 record OR 24/96 playback.
Using a USB2.0 controller might acheive better stability... Obviously the UA-5 will still operate at 11Mbps max, but the USB2 bus has a higher maximum bandwidth.
Good luck!