I'm not familiar with your particular driver, but I can speak to a few general issues with computer based interfaces... the first one is latency. Latency is basically "delay for the signal to work it's way through this system". If you were running what comes into your computer back out to the band's monitors (or studio headphones), they would really hate it if there was a delay there. Since you almost surely aren't doing that, you can turn the latency allowance all the way up, and who cares if it gets written to the hard drive 1/4 second later. Some drivers have a little "driver control panel" where you can set latency.
Then...
- defrag your hard drive. And make sure the drive isn't almost full. Cluttered drives are slow drives.
- Make sure the settings in that software panel (if there is one) match the hardware switches on the UA-5. Any time you make a change to those settings, restart Audacity.
- make sure your USB port is set to USB2.0 and not USB1.1 in the BIOS or elsewhere.
- If you are trying to record at 24/96 and have glitches, but it works fine at 16/44 or 24/48, you may just have to face the fact that you can't run at that speed with this hardware.
Good luck.