I have no experience with either deck, so take this with a grain of salt...
My thought is that you are not actually adding 30 dB of gain with the 663. This is all very back of the envelope (for real; I scribbled this out on an actual envelope), but maybe worth a look. According to DPA's Microphone University, "sound pressure level inside a concert grand piano can exceed 130 dB SPL peak less than eight inches (20 cm) over the strings". From your description, your mics were about 5 feet from the piano, so if it was being played at near peak volume, it would be about 112 dBSPL at the caps. Those mics are quite sensitive (50 mV/Pa), so at 112 they are putting out about -6 dBu. SD specs the maximum mic input as 0 dBu, so I think you would likely be overloading at 30 dB gain. Probably the -6 dBu figure is inflated, as I only used your estimate of horizontal distance (the extra height would make it further, per Pythagoras, but probably not a whole lot) and because the actual SPLs at the piano probably didn't reach that 130 dB extreme. So probably in the ball park of the F8 gain setting would get you touching zero.
If possible, I would send a signal of known SPL into the 663 and see how much gain is added at that 9 o'clock setting.
[EDIT: Assuming I did the math correctly...]