FAT32 has a 4 GiB file size limit (2^32 = 4GB). Whatever they do, M-Audio would have to a have a pretty large buffer (let's say 24 bits x 96 ksps x 5 seconds to save/restart recording = 1440000 bytes = 1.37 MiB) to ever seamlessly exceed 4 GiB. I suspect that the buffer is already pretty large, and that a good portion of the "Writing file" save time is actually just flushing that buffer. Probably just not large enough for seamless splits. Or their programmers are lazy.
A workaround for FAT32 limitations would be to use a different filesystem, but a) they're probably using a 3rd party chip/driver to handle the flash card, so it'd take a lot of hackery to do that, and b) that would force us to use M-Audio's proprietary software to read the data on the flash card, which I think everyone would loathe.
The 2 GiB limit, if they really have decided not to fix in this hardware, is, if it's at all related to hardware, more likely a limitation of whatever chip they're using to interface with the flash card. Maybe it steals that last bit as a control or parity bit. Maybe some lazy engineer couldn't find the space for the last address line. Who knows. We're probably stuck with it. I record my stuff as MP3 anyway because my equipment/the recording environment is never going to be good enough for me to notice the difference between 24bit/96kHz and 320kbps/48kHz, so my biggest limitation is always that stupid permanent internal battery. And the flakey dollar store batteries I've been using to externally power the deal.