Hrmm...sounds like an odd one. edit: actually...maybe not so odd. I just remebered that during initial testing for rockbox optical recording there were a few punters who had issues with freezing in the recording screen. Invariably it was due to a dodgy optical cable not sending a correct signal, so rocbox can't pickup a sample-rate so it kinda freaks out. Do you have any other cables you can test with? /edit
If its definitely not that then.....:
Try what Diamond_D suggested. Like he said tho, that should be fixed.
Am I reading it right that the iriver freezes as soon as you plug in the optical cable...irrespective of what screen you're on? Does it freeze immediately? Or does it just freeze when you go into the recording screen?
Again, this probably won't work but you might like to try resetting your iriver settings (A-B button > Manage Settings > reset settings)
Once you've done that, and if its still an issue, just for giggles also try enabling optical out (A-B > General Settings > Playback > Optical Output). Optical out needs to be enabled for optical in to work...but it should be auto-enabled when you enter the recording screen.
Have you got any gear to do isolation tests with? ie can you run UA-5 > optical > computer or jb3?
What happens when you try running the rig with the iriver f/w? (hold record when you press play to start the iriver)
Are you sending a 44.1/16 signal from the UA-5? (again...most variations shouldn't be a prob)
If all else fails, you could perhaps try jumping onto the rockbox irc channel (available via their website) and asking amiconn or preglow if they have any ideas. They did much of the work on getting optical recording up and running.
I really hope you don't have a buggered iriver, but quite a few people are running this rig fine (including myself) so its not likely that its rockbox related.