Thinking about it more, if it helps with the camera analogy --
More megapixels are better, so you'd rather a 20mp camera over a 10mp camera. But what if the 20mp camera mfg couldn't really make a reliable sensor, and with their 20mp sensor, they randomly would get 5% of the pixels that would show up pink, no matter what you were taking a picture of. Now, would you rather have the 20mp camera or the 10mp camera? What if the camera had a button for 10mp operation (and the pixels were all accurate) and a setting for 20mp operation, but again, you'd get a whoe bunch of pink pixels. Which settign would you choose?
The issue is actual, real world operation, not theoretical operation. None of us really knows whther our recorders or A/D's actually perform worse when set to perform at 24/96, and there is a lot of info out there that getting performance at 96 to be better than 48 is no easy task.