the sample rate on the PMD-661 has to set to match the ouput of whatever is upstream. If the sample rate on the 661 doesn't match the upstream gear, it won't lock on the digi signal. 16 bit / 24 bit doesn't matter as much. if it's set to 16 bit and the upstream gear is sending 24, the signal will be truncated by the 661, but it'll still lock. Likewise, if the deck is set to 24 bit and the upstream gear is sending 16 bit, you'll just be wasting disc space.
anyway, my guess is that the digital signal from your DVR wasn't stereo LPCM, and that's why it sounds like static.