Some sort of pre amp will improve your sound. I understand the philosophy. I bought into it to at one point in time. Adjustable output controls on CD players and dacs are controlled with attenuators. When an adjustable output is available, it's best to have the control wide open (no attenuation). Not doing so reduces overall fidelity of the system. An adjustable output is not the same as a volume control, which uses amplification. The setup you describe will work, but I think you should look at a preamp first when the upgrade bug hits you.
Chris