On Vanark's point about finding the most neutral input setting, I think that is a good idea to do, I just don't know if it is necessarily the unity gain point. Like itgoes11 says, the way to get there might better be to listen to what sounds best.
The signal path in a recorder or preamp could be going through a number of capacitors, passive attenuators/resistors, active gain stages, etc. Who knows how the signal is affected by all those components, and what sounds best? I wouldn't say that it isn't necessarily a bad starting point to begin setting levels at unity gain, and listening to what it sounds like there and then at other gain settings, but I also don't know or assume that the unity gain point is where it will sound most neutral.
Ok, here goes the uber-geeky stuff, but I'll try to walk through it slowly (and I'll use pictures!). Like I said, there could be lots of elements in the signal path, but consider the op amp gain stage (probably most/all of the recorders we use will have at least one active gain stage using an op amp or something akin to it).
There are lots of factors affecting the sound of an op amp, and most of these are characterized by the manufacturers. Among other things, this might include noise performance, slew rate, signal response, etc, and these characteristics might/will change depending on what gain setting you have on the op amp -- that is, unity gain (gain factor of 1, or 0db gain), gain of 2x (6db gain), gain of 10x (20db gain).
Below I've got some graphs and oscilloscope outputs provided in the datasheets for the Burr-Brown OPA2132 op amp and Analog Devices AD823 op amp (good op amps, though probably a bit dated). These graphics represent how the op amp manufacturers characterize the response of their op amps.
First: noise performance. In the figure below you can see that at a gain of 1 (unity gain, 0db gain), the noise performance is about 0.0001%, whereas at a gain of 10 (20db of gain), the noise performance is about 0.001%. That is, at 20db of gain, the noise performance is 10 times worse than it is at unity gain -- though the noise performance in either case is very, very good (and you'd be unlikely to discern a difference).
Next up: slew rate. Slew rate represents how fast the output voltage changes in response to changes in the voltage of the input signal. Higher values are better.
In the figure below, you can see that the slew rate at a gain of +10 (20db gain) is higher (better) than it is at a gain of +2 (6db of gain), which are both higher than the slew rate at a gain of 1 (unity gain). In all cases, the slew rate is sufficiently high that the differences wouldn't have any discernible difference in sound (I wouldn't think anyway), but the slew rate response is better at higher gains than it is at unity gain.
Next figures: small and large signal response, and overshoot. If for instance you send an input of a square wave, you would ideally want to get an output of a square wave.
The figure below is an oscilloscope output of a small signal response of an op amp set to unity gain. As you can see, what should be a square wave isn't perfectly square: The signal rise and fall points should be straight up and down, but instead they are actually a steep ramp up and a steep ramp down. On top of that, you can see the results of "overshoot" where the output signal rises too far and needs to correct itself, and falls too far and needs to correct itself.
The figure below shows overshoot as a percentage for small signal responses, for gains of 1 (unity) and 10 (20db) -- higher means more overshoot, or worse performance. As you can see, performance is worse at unity gain than it is for a gain of 10, or a 20db gain.
Finally, large signal response. Again, these use square wave inputs (on top) and show the resulting output (on bottom), which ideally should also be a square wave. The first figure is the large signal response at a gain of 1 (unity gain), the second figure shows the large signal response at a gain of 10 (20db gain). As you can see, the large signal response is better (more like a square wave) when the op amp is set for a +20db gain than when it is set for unity gain.
Bottom line, the info from the manufacturers spec sheets indicate that the op amps, not surprisingly, aren't perfectly ideal -- as in, perfectly neutral. But the op amps are more neutral if you will, when considering signal noise when set for unity gain than when set for a gain of 10 (20db of gain). On the other hand, the op amps are better/more neutral when set at higher gains rather than being set at unity when you are looking at slew rate, overshoot, and small and large signal response. I'd guess though that in all cases, it is all just a matter of degree and that for all these performance metrics the op amps perform perfectly good regardless of whether they are run at unity gain or at +20db gain.
So which of the different op amp characteristics most impact the resulting sound, and lead to it sounding more neutral? I couldn't tell you, but it seems like op amp response isn't universally better at unity gain than it is when set for a gain of +20db. So I wouldn't assume it will sound more neutral at unity gain -- need to just listen for yourself at unity and at different gain settings and make your own judgements.
Hopefully I've got all that right with interpreting the spec sheets and mshilarious won't come in here saying I've screwed it all up.