Your approach sounds right, if I'm following (and my assumptions about the UA-1EX are correct):
- Playback on SV-3800 using optical / toslink OUT
- Optical / Toslink IN on UA-1EX
- UA-1EX performs purely digital conversion of toslink / optical signal to USB
- UA-1EX USB OUT
- USB IN on Macbook
- Capture USB input via Macbook / Audacity
The only unknown for me in this case is how the UA-1EX treats the optical / toslink > USB conversion: does it resample, is it bit-accurate, etc. Ideally, the UA-1EX would not resample and be bit-accurate. In the grand scheme of things, it's not the end of the world if it doesn't, but it's definitely preferable.
If Audacity isn't picking up the signal, I'd first confirm there is, in fact, a signal. Try using other software to confirm the UA-1EX is actually passing a signal -- for example an audio player to play back the signal from the UA-1EX in real-time, or another recording app.
Once the signal is confirmed, the issue is then configuring Audacity to recognize said signal.
Unfortunately, I can't help beyond any of this as I'm not familiar with Mac.