Hi Ed: welcome. finding a comfortable balance between quality and convenience is always a struggle. I think the suggestion to explore in-ear mics is a good one. I don't know much about in-ear / binaural mics, but it's worth investigating. If you're making stealth recordings the in-ear mics might look a lot like earplugs and shouldn't attract attention.
As for recorders, I think you have a lot of very good options. If you really want an all-in one unit and definitely do not plan to use external mics I think a Sony D-50 is worth consideration. Start-up is very fast, external mics are good (and they are adjustable); background hiss is low and it's extremely well built. It will also support external mics if you choose to go that way at some point. The D-50 is discontinued, so you would need to find one used.
i concur this is one of the best options for recorders with built in mics
i am actually selling the one i just bought from bass_ur_face as i bought it for digi in and it doesnt play nice with my mytek....