As you note, you can set up the 680 to record digitally on the "ST" channels (it's easier to me to think of these as channels 7 and 8 ), or you can record digitally to channels 5 and 6.
If you are recording to channels 7 & 8, I think you left out the most important part:
To record on 7/8, you need to go the the recording menu off the main menu, and set "ST REC" to "DIN", meaning digital input (as opposed to "MIX", for recording a mix of channels 1-6 on the 7/8 channels).
This is all you need to record digitally on 7/8. According to the manual, if set to "DIN" on "ST REC", on the I/O menu, it will not matter if you select "ANALOG" on INPUT SEL" or if you select "DIGITAL", either way, the 680 will record the analog channels on 5/6, since you have set it up in the REC menu to record the digital channels on 7/8.
I don't think the DIN MON settings or D OUT MODE have anything to do with whether you can record digitally. They will affect whether you can monitor your digi input channels on the headphone, or output the digi input to the spdif coax output (for a patcher, etc), but I don't think they affect whether you can actually record off the digital input.
For recording the digital input on channels 5/6, the "ST REC" setting on the recording menu must be set to "MIX" (sending the 1-6 mixed signal to 7/8), and then also set on the I/O menu the "INPUT SEL" to "DIGITAL".