What signal does the digital out send? Just the digital input, or can it send a mix? Is there any way to send individual channels (analoge out set to "direct") then send two specific channels through the digital out?
The digital out sends whatever monitor mix you set as a panned and level adjusted mix of the 6 inputs. If you set the machine for 'direct' output of each individual channel to it's analog RCA out, set the monitor mix levels to only output the two specific channels you want, one hardpanned left and the other right, with the other channel levels set to zero, it will do what you describe.
Also, if I set analoge out to "mix", and have a board feed into channel 5/6, then set all the other channels in the mix at 0, I would be have 3 separate outs giving the board feed, correct?
Assuming you don't have very old firmware, you would actually have board feed to all outputs. If you don't pan anything you'd get the identical mono mix of ch5 & 6 on all outputs. If you pan ch5 hardleft and ch6 hardright in the monitor mix, ch 5 would be routed to all the odd numbered RCA outs and ch6 to all the even ones. The digital out would consist of ch5 left / ch6 right.
If you set the analog out to 'direct' you'd get the ch5 input on the ch5 output, the ch6 input on the ch6 output (regardless of monitor panning or levels), and 5 & 6 on the digital out, panned however you set that in the monitor mix.