Phantom power is strictly for powering your microphones...
In your case - you are not running any mics...just taking the feed from the board (their mics) - no need for phantom...
And as others have noted - it can damage the sending soundboard...so make sure its off...
Question for UA5 users:
Can you send a line level signal to those front XLR inputs, or are they just for microphones?