Some time has passed since the question was posed...I suspect that nobody has a firm opinion on this. Frankly I don't either. I think the answer is that you should keep whatever you feel you need to keep for whatever you think you might want to do with your data in the future. Question...you said you record in 96/24. Why? You must have a reason why you record in this format, so the answer probably should figure into your strategy for long-term storage and archiving.
If you think that you'll always stay 'cutting edge' and the reason you record in 96/24 is so that you might be able, in the future to hear a difference or because you might want to somehow take advantage of a technology that uses this resolution, then you should probably archive this way also.
Personally, I'm not that into it all because my view is that live ambient recordings are inherently faulty. My ears enjoy a good sounding live recording whether it's recorded in at 44.1/16 or anything higher...IOW, I subscribe to the philosophy that if it sounds good, it is good. The incremental sound quality difference gained by higher bitrates, while I do acknowledge is there, for me I don't consider it significant...again because my enjoyment of the music derives from the performance quality and the basic elements of what is in the input chain (mics, preamps, location of mics, etc), far more than the resolution of the recording.
That said, I keep my recordings archived in flac'ed and verified versions in 24/48 and don't save the redundant 16/44.1 versions. I'm now using a 16 bit recorder, so I archive those in 16/44.1. This is what I keep as my 'master' (without the crap before and after the sets). Generally, I don't keep the raw unprocessed and untracked file. IOW, the only FLACs I keep are those that I have applied processing to. My reasoning is that it's all data and I personally see no difference between what gets laid down originally or what gets tweeked in post. (I know I'm in the minority here...it's just my preference). The exception to this is if I needed to do alot of processing on the original file...in that case I do keep the original raw source file.
Right now, I'm exposed in that all of my flacs are on several smaller harddrives and the only backup is the CDRs of the show. However, I will be buying a second 1TB drive for redundant backup of the entire collection.