I'm wondering why do you have the bass filter and limiter on? I never use those and I don't think any tapers on TS rely on those either.
I use the bass filter because the battery box I'm using at the moment has no bass rolloff. The Sound Professionals mic has a much wider frequency response (20-20k) than the Olympus (100-15k), so some bass rolloff is needed. The recordings I made without any rolloff were unlistenable, and I hate using EQ in post. Admittedly, the Sony's filter is supposedely rather high (200Hz), so that could be an issue.
As for the limiter, I'm not aware of any reason not to use it. I set the levels at Sony's suggested -12dB, so I guess I don't really need it. But what would be the disadvantage of using it? Isn't a limiter desirable to prevent clipping from an unexpected noise?
Still, no reason to do this. You should adjust bass in post. The bass rolloff on those small decks is just not any good; also, once it's gone, you can't really put it back. Rolling off at 200Hz, especially depending on the curve it's using, is massive overkill.
Not all limiters are created equal as I understand it. For example, the Sound Devices 722 that page mentions below is a superior quality audiophile deck designed for recording music and FX - its limiter probably works with those applications in mind. The primary purpose of a Sony M10 and units like it is recording speech and that is what its settings such as the limiter and the bass rolloff are set for. When used with music, the results can be unpredictable. It may well be the case that the "limiter" is not working like a typical hard limiter that just prevents clipping, but actually starts compressing and limiting your signal prior to hitting the clip point; in fact, I'd bet it probably does. That is why you'll find pretty much universal agreement on this board that with these small decks like the M10, which are very good when used properly, you should never use these various "automatic" settings the deck has.
As to your original problem, it sounds like one of placement or program material or both. In general with a lot of rock recordings made in the audience, you are going to find that guitars don't come through as clearly as vocals and drums. I find that even running high-end gear, I have some recordings that I am not terribly happy with the guitar on.