^^^ Some devices are more finicky than others. I would really try a different card and see if you have the same problem. A class 4 card (sustained write at 4 MB/s) should be amply fast enough for stereo recording, even at 24/96 (~ 0.55 MB/s)...
For what it's worth, I have been using class 4 cards in my M10 for several years with no issues (mostly at 24/48 but I have made some recordings at 24/96 too). Also, I think the Memory Stick Micro, Sony's proprietary card, clocks in around 3 MB/s.