OK, I've tested along those lines.
I made an H2e recording with the device lying on the bed in a quiet room. Then I put a duvet, rolled up, over it, and recorded some more. Then I took the recorder into the toilet adjacent to that bedroom, which has no windows, and recorded some more silence. Finally I put the duvet over the H2e in the toilet - yes, really - and made a final recording of silence.
Playing those back, normalised equally, I could hear no difference with the duvet over the recorder, compared to it not being over, and actually to my surprise there was a very slight increase in noise in the toilet. All in all, I would say that what I recorded was very close to 100% system noise.
Next, I put together a plot of the noise in the park and the noise in the bedroom. With luck it is attached. The noise in the park is in red. In the bedroom, probably just system noise, it's green, and it is a smooth curve indicating that no significant variable environmental noise was present. Note that the two lines converge at 9kHz. The noise from the park, which included environmental noise on top of system noise, is a maximum of about 9dB greater than in the bedroom. The level of system noise goes down from about -72dB at 400kHz to about -101dB at 20kHz. Remember that the recordings were made at the same analog level, as it's not adjustable.
So having now got a recording which I'm confident is system noise from this device, I went back to considering under what circumstances it would matter. I spent some time in Audition multitrack, overlaying commercial classical recordings with the system noise. I chose works which have significant dynamic range, and finally settled on Taverner's choral work "God is With Us", where the final chord in the organ is full scale loud, and where other parts are pianissimo. I adjusted playback to threshold of pain when playing the end of the work!
Interestingly, after the final chord, there are at least ten seconds of what seems to be 'live silence' before the original track is faded. With the H2e system noise added, I couldn't hear that noise at all. If I raised the H2e noise playback level by about 6dB I was just about able to hear it - with headphones, and remember the music peaks were maximal. With the H2e noise muted, the original recording's noise in the end silence was about 12dB louder than the H2e noise on its own, though it may well have included some environmental noise from the cathedral where it was recorded.
So, I go back to the suggestion that even this inexpensive modern recorder's system noise (and that from other such devices) is unlikely to be a significant problem, so long as when using it you make sure that the signal to noise ratio is optimal. You can't do that with digital gain setting, but you can do it with positioning. Make sure using appropriate proximity that the level reaching the mic, the signal, is as high as possible, with the proviso that when using the device for acoustic recording and the location's reverberation is important, you are not too close. The end result, in terms of system noise audibility, should be fine in the context of the likely uses of such a device. And if it's amplified music, I would be really surprised if system noise is any kind of problem at all.