Phantom power shouldn't affect mics that don't take phantom power if it is turned on, I think. I think the name "phantom power" comes from the fact that its "phantom" to mics that don't use it, such as dynamic mics. I mean, when you have a mixing board, and have dynamic mics plugged in via xlr, and condenscer mics that require phantom power plugged in via xlr, and you turn on the phantom power, it doesn't hurt the mics or the board.
that being said, I wouldn't test this theory on my own UA-5... lol...
phantom power requires xlr. no phantom power on the 1/4" inputs.