Taperssection.com
Gear / Technical Help => Post-Processing, Computer / Streaming / Internet Devices & Related Activity => Topic started by: stevetoney on November 07, 2010, 09:06:40 AM
-
I'm reassessing my storage practices of my raw master recordings and looking for feedback. For the last several years, after uploading my 16 and 24 bit shows to archive or etree, for long term storage I've redundantly saved my 24bit flac files and the raw master file on two separate hard drives (second hard drive is kept at work). I've got hundreds of master recordings saved this way, but after several years of using this practice, I'm realizing that I'm never returning to my shows to tweek them or re-master them. I'm thinking about going back and deleting the raw master files and just keeping the flacs since obviously if I did ever want to do any tweeking, I could still do so to the FLAC files. With so many masters of audio and video stored and something like 5 or 6TB of harddrive space, I'm thinking it would be nice to free up 2 or 3 TB by deleting redundant raw files that I've never really needed and probably won't need.
Has anyone else found a profound need or reason to keep the raw masters, over and above what the flac's provide you?
Thanks!
-
For the cost of storage these days, I couldn't bring myself to deleting anything. The 2-3 TB you're considering reclaiming is worth, what, about $150 at today's prices? What's the peace of mind of having those hundreds of raw files archived worth to you?
-
For the cost of storage these days, I couldn't bring myself to deleting anything. The 2-3 TB you're considering reclaiming is worth, what, about $150 at today's prices? What's the peace of mind of having those hundreds of raw files archived worth to you?
I fully agree with your comment, but if I never do anything with the raw files, what are storing them providing me that the flacs aren't also providing?
For example, can anyone give me some feedback about whether the raw files were more valuable or better to work with than unpacked flacs if you wanted to go back and tweek?
-
If nothing else, having the raw files provide another backup in the event of data corruption.
-
The way I look at it is in the event that I lost my edited files I could always go back and re-track my favorite shows. I may never re-track them all, but at least I can go back and listen to my favorite shows.
Hopefully I will never need to do that. I just bought a 2TB hard drive and external case for $102 shipped. Even if they just sit there for eternity it's nice to know they are there.
-
I've said it many times before, but I'll say it again.... Damaged flac files are extremely difficult to extract back into wav files. A wav file is simple pcm. It is the data with a trivial header that can be easily constructed. If you have a media failure and are left with 85% of a wav file, you have 85% of the music. But 85% of a flac file is potentially useless.
Flac is more robust than it was in the past. It will continue to extract through errors that would have previously caused it to abort entirely. I haven't done any recent tests, ymmv.
Another consideration is file deletion vs. formatting. It is vastly easier to recover files from media that was formatted before being populated. That causes files that are copied to be contiguous. If there have been deletions, your files are more likely fragmented across the media. So whenever possible, format, rather than delete, before using (especially on field recorders).
-
all i save are my 24 bit tracked flacs. the rest seeded and then deleted. raw files get deleted
-
I have the raw wav files and the 16 bit tracked out flacs, but I've gone back and tinkered with the raw files on a number of occasions later.
-
If you've never re-edited raw wave files after you're finished with it, I don't see the point of keeping it. Perhaps you can setup a schedule to delete only a certain amount at a time just in case you decide to re-edit or if the flacs cause issues.
2-3 TB is a lot of space and you would also want backup of your backups which could be 4-6 TB.
-
I wish there was, and maybe there is, a way to save the changes to an audio file in a non-destructive way. In the video world, I can save my project file in Final Cut Pro, but the video remains virgin. I can also save an EDL (edit decision list), which is portable among video editing apps. If you could do this in an audio editing app, then you could save the raw files, and just archive the relatively small EDL, which could be called upon for future edits. Make sense?
Maybe this is possible in apps like ProTools, etc., but this board seems focused on Audacity and mostly other low end apps.
keytohwy
-
I've said it many times before, but I'll say it again.... Damaged flac files are extremely difficult to extract back into wav files. A wav file is simple pcm. It is the data with a trivial header that can be easily constructed. If you have a media failure and are left with 85% of a wav file, you have 85% of the music. But 85% of a flac file is potentially useless.
Flac is more robust than it was in the past. It will continue to extract through errors that would have previously caused it to abort entirely. I haven't done any recent tests, ymmv.
I thought about this. What you don't say though is whether you've actually had experience getting lots of damaged FLAC files. I'm not challenging your claim, but asking for additional information.
FWIW, I've have had experience with damaged discs the caused my FLAC not to extract. It was the reason I stopped archiving to CDs and DVDs...because they don't last.
However, now I'm using harddrives with redundant backups. Since I have a redundant copy of the FLAC file on my redundant hard drive, I would be able to recover the damaged file on the second drive, so I don't think the damaged FLAC file is a concern since I'm already redundant. That said, I suppose a damaged FLAC could be damaged at creation and pass the file integrity check, then it would be damaged on both my source and backup drives.
Thoughts, comments?
-
I would stop archiving in flac and keep the raw files. I have had flac/shn files (not masters as far as i can remember) not want to open before.
-
Has anyone else found a profound need or reason to keep the raw masters, over and above what the flac's provide you?
In your situation if one of the drives is damaged by flood, fire, tornado, etc. you have a better chance of having a library of your audio recordings.
You have to weigh the value of keeping them vs the cost. But consider the fact that cost of storage has steadily decreased and likely will continue to do so.
-
I wish there was, and maybe there is, a way to save the changes to an audio file in a non-destructive way. In the video world, I can save my project file in Final Cut Pro, but the video remains virgin. I can also save an EDL (edit decision list), which is portable among video editing apps. If you could do this in an audio editing app, then you could save the raw files, and just archive the relatively small EDL, which could be called upon for future edits. Make sense?
Maybe this is possible in apps like ProTools, etc., but this board seems focused on Audacity and mostly other low end apps.
keytohwy
Reaper.fm...... all for $60us IF you decide to buy.
-
I wish there was, and maybe there is, a way to save the changes to an audio file in a non-destructive way. In the video world, I can save my project file in Final Cut Pro, but the video remains virgin. I can also save an EDL (edit decision list), which is portable among video editing apps. If you could do this in an audio editing app, then you could save the raw files, and just archive the relatively small EDL, which could be called upon for future edits. Make sense?
Maybe this is possible in apps like ProTools, etc., but this board seems focused on Audacity and mostly other low end apps.
keytohwy
Yes, this would be nice. I do save my cue lists when I set my tracks in CDWave, but that only lets you repeat track splits.
-
I burn my master waves, 24 bit and 16 bit flacs to dvd. I keep one set of both the 24 bit and 16 bit flacs in my music folders/hard drives to listen to thru my playback. I keep a backup copy of my music (albums, downloads and my recordings) offsite.
-
I wish there was, and maybe there is, a way to save the changes to an audio file in a non-destructive way.
FWIW, Samplitude does.
-
I burn 16bit, 24bit, & untracked flacs to dvd (all quality TY disc). 16 and 24 bit flacs are also backed up to at least two externals harddrives (kept in fire and h20 proof safe) in addition to the internals where I use them for the squeezebox. I too am thinking about dropping the burning of the untracked files. I have never gone back and needed them. for shows that I was the only taper I think I should keep them but once I seed the 16&24 bit flacs so they are out there without errors then it's starting to seem pointless keep the raw files.
-
all i save are my 24 bit tracked flacs. the rest seeded and then deleted. raw files get deleted
it's a free country so do whatever you want of course, but just in case anyone else sees this post and thinks that your approach is wise...deleting everything but tracked FLACs seems like a risky and borderline boneheaded approach that would not be recommended by many. what if you ever want to go back and remaster the full show? then you have to piece it back together or do it track by track. what if the FLACs get corrupted somehow in a way that a WAV wouldn't be? seems like that is a strategy bound to bite you in the ass someday.
-
see my post above. been taping with mics since 2003 and have NEVER had to go back to my raw files. I still have them though but am thinking about not keeping them anymore.
-
all i save are my 24 bit tracked flacs. the rest seeded and then deleted. raw files get deleted
it's a free country so do whatever you want of course, but just in case anyone else sees this post and thinks that your approach is wise...deleting everything but tracked FLACs seems like a risky and borderline boneheaded approach that would not be recommended by many. what if you ever want to go back and remaster the full show? then you have to piece it back together or do it track by track. what if the FLACs get corrupted somehow in a way that a WAV wouldn't be? seems like that is a strategy bound to bite you in the ass someday.
if i lose it all, i lose it all ;) i am really not that worried about it. i may go back and listen to maybe 20% of my recordings more than once or twice. there will always be more music to collect down the road.
-
if i lose it all, i lose it all ;) i am really not that worried about it. i may go back and listen to maybe 20% of my recordings more than once or twice. there will always be more music to collect down the road.
Yeah, and performers die. Sometimes there is only one recording of their last performance, or none. Stuff like that.
Most of my masters are 24/96. I don't consider the 16/44 versions to be the 'be all, end all'. So preserving the originals is important, to me.
-
How often do people actually have issues unpacking FLACs? I've mentioned earlier that I had issues, but that ended up being due to corrupt media discs. Otherwise I've never personally had an issue with the actual FLAC file unpacking. I'm not saying it's not an issue though...I'm asking if this has happened to people.
For those that this may have happened to, do you double-check the integrity of your FLACs with your MD5 or FFP immediately after you create them to ensure the FLACs were properly built?
-
How often do people actually have issues unpacking FLACs? I've mentioned earlier that I had issues, but that ended up being due to corrupt media discs. Otherwise I've never personally had an issue with the actual FLAC file unpacking. I'm not saying it's not an issue though...I'm asking if this has happened to people.
For those that this may have happened to, do you double-check the integrity of your FLACs with your MD5 or FFP immediately after you create them to ensure the FLACs were properly built?
had one of my masters sit on an hdd once and it wouldn't decode after about 2 years. I ended up figuring out where the bad sector was, decoding up till that point and I lost the last 15 minutes of the show (which had the best song... bah).
I verify all flacs upon creation, mine was something went wrong on the drive over time.
-
How often do people actually have issues unpacking FLACs? I've mentioned earlier that I had issues, but that ended up being due to corrupt media discs. Otherwise I've never personally had an issue with the actual FLAC file unpacking. I'm not saying it's not an issue though...I'm asking if this has happened to people.
For those that this may have happened to, do you double-check the integrity of your FLACs with your MD5 or FFP immediately after you create them to ensure the FLACs were properly built?
had one of my masters sit on an hdd once and it wouldn't decode after about 2 years. I ended up figuring out where the bad sector was, decoding up till that point and I lost the last 15 minutes of the show (which had the best song... bah).
I verify all flacs upon creation, mine was something went wrong on the drive over time.
Right. Again bad media...in this case bad sectors on your harddrive. I'm thinking that having redundancy addresses that concern as long as you verify that the FLACs were properly created at the start. IOW, just go to your redundant drive and grab the same file which of course isn't ruined due to sector errors.
Curious, in the case of having a bad sector on a .wav file, does that just result in a momentary drop out of the music? I'm thinking so based on the fact that if you listen to a CD with a scratch it will keep playing past the scratch as long as the scratch isn't bad.
-
I have made it a point to keep my raw files going forward. I almost always do 4 channel, which requires "matrixing" I can't help but think that either there will be technology in the future that will enhance the recordings I have now, or my processing methods will change/improve to the extent that I might wish I had those raw files to work from.
-
How often do people actually have issues unpacking FLACs? I've mentioned earlier that I had issues, but that ended up being due to corrupt media discs. Otherwise I've never personally had an issue with the actual FLAC file unpacking. I'm not saying it's not an issue though...I'm asking if this has happened to people.
For those that this may have happened to, do you double-check the integrity of your FLACs with your MD5 or FFP immediately after you create them to ensure the FLACs were properly built?
had one of my masters sit on an hdd once and it wouldn't decode after about 2 years. I ended up figuring out where the bad sector was, decoding up till that point and I lost the last 15 minutes of the show (which had the best song... bah).
I verify all flacs upon creation, mine was something went wrong on the drive over time.
Right. Again bad media...in this case bad sectors on your harddrive. I'm thinking that having redundancy addresses that concern as long as you verify that the FLACs were properly created at the start. IOW, just go to your redundant drive and grab the same file which of course isn't ruined due to sector errors.
Curious, in the case of having a bad sector on a .wav file, does that just result in a momentary drop out of the music? I'm thinking so based on the fact that if you listen to a CD with a scratch it will keep playing past the scratch as long as the scratch isn't bad.
While true, in my experience, having a bad media issue with a flac file is much more disastrous then a bad media problem with a wav file (was really the point I was trying to articulate, my appologies if that didn't come through).
-
as far as redundancy goes if you only have one drive or burn to dvd (relying on media) you may want to look into creating parity volumes with a program like quickpar (http://www.quickpar.org.uk/). no affil but a decent program to save you if your media 'has a bad sector' like above..
may take up a bit more space but usually when burning to dvd you have a bit of free space. I used to burn off a bunch of music to dvd. Had issues with dumping the music from dvd back onto the machine (media failure). So for awhile there I was rar'n up everything then creating 20% parity so if I was only able to recover 80% of the files from the dvd + the par files I was able to restore 100% of the music using quickpar.
neil
-
My shows are mastered so perfectly and with such precision, that I can not imagine anyone, least of all me, wanting to re-master them. ;D
So I keep the split flac files only. And have three (3) backups.
-
And have three (3) backups.
What medium do you like for your backups?
-
I save my raw files... right now only 2 copies on identical hard drives. In the future I hope to get a third hard drive, and also keep them in separate machines. Occasionally I've gone back to the raw files after I had an edited version (mostly on shows where I mixed multiple sources that I did when I was still pretty new to taping, and the final mix wasn't as good sounding as I can do now).
The edited version I have 2 to 3 identical copies on separate hard drives. But I have another plan that is my last resort if I encounter data loss across all local copies with any of the edited versions: I try to post them online so that other people have a copy and I would try to find someone to provide a copy back to me. ;)
-
Sparky--
I have a 4 Bay SATA enclosure which houses the weekly (or more frequent) regular backups via an esata card and cable.
I have a cheap 4tb usb external usb external drive which backs up the back up about once a month.
I have an identical 4tb drive which backs up the 2nd back up about every other month.
All back ups are handled via folderclone which I like to do manually, usually on Sunday. Sunday is backup day.