Your ears will tell you when you've reached the limit. If you have to amp so much that the noise floor starts to get overly audible, if the music still sounds good just use a filter to get rid of the noise. FWIW, as long as you recorded in 24bit, you could amplify ALOT and still have a reasonable recording, so 20db is not too much as long as it still sounds good and the noise isn't overly audible.