kidrocklive, you can generally look at any recorder or signal processor as a set of blocks or "stages" of circuitry. Each stage must be able to handle the signals that are passing into and out of it without overloading. According to your description it seems as if the very first stage (the "input stage") of your recorder overloaded when you switched it to "mic in" and set the sensitivity to "high."
When you switch any recorder to its microphone inputs, you're setting its input stage to be more sensitive than it would be if you set the switch to "line." The first stage is then performing a higher degree of amplification on the signals you send in--or it's trying to, anyway. Every amplifier stage has its limits, and if you try to make a circuit stage amplify a signal more than it can, it will overload (a/k/a "clipping" or "brickwalling"). The active device in the circuit stage will be trying to pump out more current than it can--it's like what happens if you tell a human being to run at 20 miles per hour (when the maximum is more like 15), and then you tell the person that they have to run at 30 miles per hour--this will not make them move any faster, but the difference between what they're actually managing to accomplish and what is being demanded of them will be greater.
The thing is, in most recorders you have the first stage, and its gain (degree of amplification) is controlled by a mike/line switch (which may have high or low sensitivity settings as a variation, or may not)--and only then does the signal pass through the record level knob. So if you set the input stage sensitivity too high for the signal levels that are coming in, clipping (overload distortion) will occur in the first stage, and THEN that clipped, distorted, amplified signal will be set to whatever level you choose via the knob. Similarly the meters will only show you the levels of that clipped, distorted signal--because they, too, are located at a "later" point in the circuit, after the signal has already passed through the input stage for better or for worse.
If you intend to make live recordings, you need to know a few things in advance in order to avoid problems like this. You need to know the maximum sound pressure level that your microphones can handle without overloading themselves. You also need to know what signal level (voltage) the microphones produce at that sound pressure level. And you need to know the maximum voltage that your recorder's inputs can take without clipping, for each of the possible input settings that are available (e.g. line, mike, "low sensitivity," "pad" or whatever options there are). Keep in mind that the record level controls do not (or only very rarely) affect this overload limit, and that the meter or LED readings do not (or only very rarely) reflect what is going on in the input stage of any preamp, mixer or recorder.
With consumer or semi-pro portable recording equipment, it's extremely common that the mike inputs of a preamp or recorder or sound card can be overloaded by the signals coming out of a condenser microphone at high sound pressure levels. This is the manufacturers' fault up to a point; past that point, it's our fault for buying such equipment, but there sure is a lot of it out there. Anyway, once you have bought the equipment, the situation can only be remedied by choosing a lower-sensitivity setting for the input circuit (via the mike/line switch and whatever alternatives it may offer) or by using in-line resistive pads at the inputs of the recorder to reduce the signal levels coming in.
Or you can pay attention to these issues when shopping for a preamp, mixer, recorder, interface, etc. and make sure that it's designed so that at the signal levels your microphones can produce, the inputs of your recording equipment cannot possibly be overloaded before their outputs are. That's a topic for another message if you're interested, but what it amounts to is that, for example, with the preamp I use most often for live recording, if I set it so that the signal peaks just reach (say) 2 dB below full scale, the input stage of the preamp will be able to handle any signals that my microphones (which I know have a maximum output voltage of about 900 mV) can possibly put out. So all I have to do is set the recording levels, and I will never have a problem of input overload with those microphones and that preamp.
With a little specific attention when selecting equipment, the problem can be avoided altogether. Unfortunately when people combine their equipment without paying attention to this issue (or when the manufacturers refuse to talk straight about the limits of their products), the problem can occur during live recordings and the meters or overload LEDs will never show it--it will only be evident after the concert, when it's too late to do much about it.
--best regards