Topic: Archiving: Do You Save Your Raw Master File, Just the FLACs, or Both?

[stevetoney]

For archiving my masters, I've been saving my raw 24bit master along with the FLAC files of the final 24 bit master and then back both up on a separate harddrive which I store in an off-site location.  I don't save the 16bit versions since they're derivatives from the 24bit master anyway and can be re-created later if need be.

It occurred to me today that after several years of using this practice, I don't think I've ever once gone back to my raw master file and re-used it for any reason.  I've used the FLACs (mostly to make 16bit versions for burning CDs), but not the raw master. 

When I originally saved the raw master, I had the idea that I should keep the raw master in case I ever wanted to re-master the show.

Now since half of my archive space is being taken up by these raw masters, I'm wondering if it's a waste of harddrive space to save the original raw master since I have all the FLACs in two places anyway.  If I needed to, I could re-build an un-cut version of the show, but of course it wouldn't always be the same as the raw master although I try not to do too much post-processing to my shows. 

I'd probably still keep all of my M/S raw masters (where one channel is the mid and the other is the side), since those can't be reverse engineered very easily.

Anybody else have strong feelings one way or the other about keeping their raw masters? 

If you keep them, are you like me in that you never use them?

[fuzzypants]

granted I have no where near as many masters as most of you do what I've done is kept the master wavs and burn them to dvd then put the dvds in box then in my closet..nothing fancy but it keeps them off site on something that hopefully will stand the test of time.

[yug du nord]

Keep the raw....  takes up space.... but??  If you trash anything, I'd trash everything but the raw master.  Keep something to listen to that's a finished product....  but keep the raw!  That's my opinion.

[Javier Cinakowski]

If I don't do anything more than normalizing and track splitting, I see no need to keep the orignal one track master.  If I eq'ed and otherwise tweeked with the show then yes, I would keep the master...

[imgoinmad]

hard drives are cheap

[stevetoney]

granted I have no where near as many masters as most of you do what I've done is kept the master wavs and burn them to dvd then put the dvds in box then in my closet..nothing fancy but it keeps them off site on something that hopefully will stand the test of time.

Fuzzy...unfortunately, DVD media can be amongst the most expensive and least reliable forms of long term storage.  I've got MANY DVDs and CDRs failing to read after less than 5 years...many only 1 or 2 years old. 

[fuzzypants]

Fuzzy...unfortunately, DVD media can be amongst the most expensive and least reliable forms of long term storage.  I've got MANY DVDs and CDRs failing to read after less than 5 years...many only 1 or 2 years old.

really? welp guess I'm saving up for another HD lol

thanks for the heads up

[stevetoney]

Fuzzy...unfortunately, DVD media can be amongst the most expensive and least reliable forms of long term storage.  I've got MANY DVDs and CDRs failing to read after less than 5 years...many only 1 or 2 years old.

really? welp guess I'm saving up for another HD lol

thanks for the heads up

Yup...thats really what you want to do.  Two hard drives with info backed up from one to the other is considered one of the best ways to do it.  When one of the drives starts acting up, time to get a third drive and just keep that cycle going. 

If I'm not mistaken, the problem with DVDR and CDR is that, over time, the 'burn' kinda seeps so that the tracks on the disc can't be read reliably.  I have no idea if this is what happens, but in my mind I see it kinda like an aging tatoo that loses it's crispness on the edges.

Some people use the logic that hard media is more reliable than harddrives because harddrives fail.  Well, it's true that harddrives fail, but the thing you should NEVER do is rely on a single harddrive to keep your important data.  Redundant storage is the answer to reliability.  I go the extra step and keep the second drive in a separate location in case of catastrophe such as fire or flood.  Then monthly or so, I'll synch my two drives. 

Like I said, first clue that one of my drives is going on the fritz (which hasn't happened yet) I'll be out there buying a new harddrive.

[fuzzypants]

never thought of it like that thanks for the heads up..now to wait a month for black friday

[Gordon]

I flac the untracked edited mater (not raw) and burn them to dvd.   as for regular archiving I keep 24 and 16 bit flacs on 2 different harddrives and I burn those to dvd as well.  I need to get a fire and waterproof safe to store them all in (drives and dvds).

[twatts (pants are so over-rated...)]

My workflow:

Convert raw master to 16/44 with Audacity and save new 16/44 file.
FLAC raw master.
Burn raw master FLAC with text, FFP, and WAV MD5 to DVDR.
Copy raw master FLAC with text, FFP, and WAV MD5 to external HDD.
Open new 16/44 file in CDWave, cut tracks and save.
Flac new 16/44 tracks.
Burn new 16/44 FLACs with text, FFP, and WAV MD5 to DVDR.
Copy new 16/44 FLACs with text, FFP, and WAV MD5 to external HDD.
Seed to BT.ETREE.ORG as appropriate.

Terry

[danlynch]

I back up all the raw wav files on a DVD, then back up the flacs, 24bit untracked and 16bit untracked files on another DVD.  I had a 320 GB drive, and its almost full but that's mostly recent stuff.  There's not enough room to store everything on one external drive.

I have gone back and retrieved old raw files when I felt that something needed a serious improvement in post-production. 

[bhtoque]

I've got 2TB of RAID for my recordings. I don't keep anything backed up on optical media. Some stuff I burn audio cd's of, but those are disposable in my mind. I usually listen once or twice and give them away.

As far as what to keep, unless I do a lot of processing or a multi-track recording, the original 24/96 files get erased once I've checked the tracked 24/96.

I use Soundforge to join and then put marks into the original file. I save the tracks in 24/96 then resample and dither to 16/44.1 (mostly just so I can seed) then save the tracks again for the lo-rez version. I MD5 both sets of wav's, then flac it all then make a 2nd md5 for the flacs. Once that's done the raw files get canned.

It is also important to me to keep the file structure organized. Each band has it's own directory, with sub directories for each show date. For each show the folder has the flacs, wav and flac md5's and a txt file, then a sub folder with the 16 bit fileset.

JAson

[meatballs]

i stash all of my recordings (24/48) as raw masters on an external H.D. (Terra) -- Staples on sale for $120.00!

I don't know what else to do with them -- they live quietly in their own cave as 24/48 raw masters.

sometimes -- I don't even listen to them

[Belexes]

hard drives are cheap

Agreed.  I don't even FLAC my masters. I keep 'em raw and archive them to a hard drive.

[JackHenry]

This might seem like an easy question to answer, but I've never converted to flac. The question is, why convert from wav to flac?? Why not just go direct from 24 bit wav dithered down for CD production?

If there is no real need to convert to flac, saving backup space by not doing it seems like the obvious thing to do. But as I said, I've never used flac so I don't know if there is an advantage or not.

[meatballs]

My philosophy is simple 24/48 is the max that the human ear can signify a difference(so they say). Anything above this you would have to be genetically /grafted (island of doctor meuro) mutated with a dog, bat, whale, or some other animal that can realistically, hear, determine the difference in the higher frequencies of sound.

  similar to vitamins you can take 500 mg of vitamin C, however, the human body can't process more than that -- i.e. 1000 mg grams of Vitamin C is 500 mg of waste -- all living things have this problem i.e. even plants.

That said raw masters saved and archived in 24/48 is as good as it gets for the human capacity. relatively speaking from what I know and understand -- we are fooling ourselves i.e. if you master at 192/24 yes it is more transfers per second and takes up more space but the human ear is not capable of distinguishing this.

I could be wrong, however, my arguement is based on sampling that I can remember i.e CD players first came out with 8x over sampling the moved up to24 48 56 yadah yadah however the human ear is not capable of distinguishing the difference abover 24x oversampling.


That said 24/48 is the ceiling -- I could be wrong but I'm befinetly not a whale as much as I like to sing with them   Note, please correct me if I'm misguided about this.

[Gordon]

This might seem like an easy question to answer, but I've never converted to flac. The question is, why convert from wav to flac?? Why not just go direct from 24 bit wav dithered down for CD production?

If there is no real need to convert to flac, saving backup space by not doing it seems like the obvious thing to do. But as I said, I've never used flac so I don't know if there is an advantage or not.

I know when I batch process the 24/48 files to 16/44 that the 16/44 will have header errors and sometimes have sbe's even using cd wave.  flacing fixes all of this.  and I have no need for the wave files so they get deleted.  I save and listen to the flac files only.  so by flacing and getting rid of the wavs you are saving a lot of space.

[anhisr]

I keep them raw on an external HD and I make a raw copy on a DVD first.  Then I open Wavelab and go to work.

[dmonkey]

I burn the raw master WAV files onto DVD. I put all my mixed/tracked/edited shows onto a RAID server in FLAC format.

[stevetoney]

This might seem like an easy question to answer, but I've never converted to flac. The question is, why convert from wav to flac?? Why not just go direct from 24 bit wav dithered down for CD production?

If there is no real need to convert to flac, saving backup space by not doing it seems like the obvious thing to do. But as I said, I've never used flac so I don't know if there is an advantage or not.

From a practical perspective, I can't see anything wrong with the logic you've stated.  I think most people prefer archiving to FLAC for the obvious benefits of saving harddrive space, saving money if they archive to DVD, or saving bandwidth when/if those files need to be electronically transferred.

[admkrk]

most of my shows are on dat even if the original transfer was from hdd. 

[Xontar]

Fuzzy...unfortunately, DVD media can be amongst the most expensive and least reliable forms of long term storage.  I've got MANY DVDs and CDRs failing to read after less than 5 years...many only 1 or 2 years old.

That's not true. I've only ever found 2 bad discs in my collection out of several hundred (one of which was a cheap DVD9 I'd expect to go bad, and was only bad in the 2-unreadable-tracks sense). I stick at least 2-3 a week in my drive for reading, so it's unlikely a significant fraction are bad. Cost per unit of storage for DVDs is less than for hard drives (I just did a quick calculation, it's about 11 cents per gigabyte for Verbatim DVDs, about 22 for the $80 250GB hard drive I just bought).

[stevetoney]

Fuzzy...unfortunately, DVD media can be amongst the most expensive and least reliable forms of long term storage.  I've got MANY DVDs and CDRs failing to read after less than 5 years...many only 1 or 2 years old.

That's not true. I've only ever found 2 bad discs in my collection out of several hundred (one of which was a cheap DVD9 I'd expect to go bad, and was only bad in the 2-unreadable-tracks sense). I stick at least 2-3 a week in my drive for reading, so it's unlikely a significant fraction are bad. Cost per unit of storage for DVDs is less than for hard drives (I just did a quick calculation, it's about 11 cents per gigabyte for Verbatim DVDs, about 22 for the $80 250GB hard drive I just bought).

Well, 2 bad discs is 2 bad discs.  To me, that's unacceptable performance.

My personal experience is that I've lost maybe 1 disc in a hundred...maybe a little bit more maybe less.  I don't really know the numbers.  To me that is WAAAY too many.

The key point though is that, as time passes, I lose more and more data.  It's a well publicized fact that optical discs have a shelf life.

As far as cost, a spindle of 50 DVDRs in my area costs about $20.
$20 / (4.7GB/disc x 50 discs) = 8 cents a gb

I've bought three 1TB drives in the last year for $100 or less...
$100  / 931gb = 11 cents a gb

So, I guess I did overstate that the cost of DVDR media is more expensive, but given the unreliability of the optical media, I'm sticking with my method of saving everything onto harddrives and backing up with redundant harddrives.

Consider this...if optical media is reliable, then why have virtually 100% of companies stopped using optical and tape-based methods for saving their crucial commercial data?  Instead they use redundant drives with nightly backups, where every night the data is backed up with off-site machines that have guaranteed power sources.

I'm not saying you should abandon your method of storage.  If you're happy with it, then that's great. 

I'm just reinforcing why I personally choose the method that I use.

[Scooter123]

I have a full TB of past shows.  Don't separately store the masters. I store the finished product. 

[gmm6797]

I have made many adjustments to my archval process, but this is what I do:
raw files to an old HD
raw files FLAC'd up to a DVD
edited raw files FLAC'd to a different DVD (normalized, etc)
24bit WAV files FLAC'd to a different DVD
16bit WAV files FLAC'd to a different DVD
16bit audio CDs burned

Over kill?  Maybe, but this way I have multiple copies of my work that can not be re-produced and/or recorded again, since the show is already over

[midside]

I keep 2 NAS (network attached storage) systems connected on my network, one in the house and one in the office space (in another building outside the house).  Each NAS box (Thecus N5200) currently has (4) 1TB drives, 3 in RAID5 with 1 hot spare.  Every night (morning) the other box turns on to backup new files daily.  Deleted/Altered files are not deleted on the backup system.  Once in a while I compare using Syncback and clean up the backup unit.  I feel like I am doing pretty good and everything is automated...no more long and frequent drawn out data copy/backup days for me.  I also keep a single 1TB drive disconnected and update this once in a while for extra security, but it is now full so I either need another or a 1.5.  But, one thing I am not completely safeguarded against is file corruption.  So, I really feel like I need an LTO tape scheme as well so I can go back in time if need be. EVERYTHING connected to ANYTHING on the network is either surge suppressed or plugged in though a UPS.
BTW, a lot of this is business stuff.  So, not only would I lose years or hard work, I would also be out of a job for a while.
p.s. I used to burn discs of stuff..NO WAY I am going to go back down that road.

[mosquito]

I'm a packrat and paranoid about hardware failure.

I save the original data, at least one intermediate version of from when I'm working on it (if there's substantial work), and a copy of each final product.  I use matched & mirrored pairs of drives for a RAID.  If / when there are issues with the drives, I set up a second pair in a RAID and move all of the data there.  Then I wipe (and sometimes zero) the first pair of drives and try to get them to work in a RAID again.  This means I always have two copies of the data.  One day I'll get around to keeping a copy of the RAIDs off site too.  I don't burn 'cause of the additional time.  Using RAID means I always have two copies and the RAID controller does the work of making sure the data matches between the two copies.

Then again, my mother says I get obsessive about things. 

[Lil Kim Jong-Il]

I've only ever found 2 bad discs in my collection out of several hundred (one of which was a cheap DVD9 I'd expect to go bad, and was only bad in the 2-unreadable-tracks sense).

I started a a project going back to recover all my old masters to HDD from DVD and DAT.  I'm starting to find that my masters aren't all recoverable from DVD.  A 2005 Ratdog show will never be recoverable.  Fortunately around early 2006 I started using two different types of media and making duplicate backups of the master in addition to the authored copies.  I've tossed more tha  a few discs since this project began but was able to read one disk so no other losses yet.

To the original question, yes I save the masters.  I have recordings with little issues and as I improve with the editing tools, I occasionally go back to the master and re edit when my earlier fix was not as clean.  Masters go to a RAID and optical.   Authored tracks go to media server and optical.

[bgalizio]

I do the following:

1. Master WAV to DVD
2. Master FLAC to DVD
3. Final FLAC to DVD (24bit and 16bit)
4. Final FLAC to hard drive (24bit and 16bit)
5. Hard Drive backed up to another hard drive via Time Machine
6. Upload/share final FLAC (24bit and 16bit)

[stevetoney]

I've only ever found 2 bad discs in my collection out of several hundred (one of which was a cheap DVD9 I'd expect to go bad, and was only bad in the 2-unreadable-tracks sense).

I'm starting to find that my masters aren't all recoverable from DVD. 

I've tossed more tha  a few discs since this project began but was able to read one disk so no other losses yet.

That's really one of the important messages I think needs to get out to people.  Everyone is probably gonna have varying success in archive to DVDR and CDR, but I just don't see DVDR and CDR being a long term solution...

One thing I've convinced myself is that my CDR and DVDR are never gonna have the shelf life of...say...an analog Maxell XLII from back in the day. 

[kindms]

I do the following:

1) record show
2) copy show to DAW
3) 24/48 raw file (dead air removed) flac'd
3) 24/48 mastered > Flac'd
4) 16/44.1 derivative of #3 > Flac'd
5) all 3 versions burned to DVD ( I used to do a DVD-A copy as well but stopped that recently)
6) DVD media stored in dbl disc thin DVD cases
7) once all is complete I then move the 24/48 mastered file to my masters folder for playback.

I worry about my HDD's tho. I need to come up with a good solution for redundant back up. The amount of time to copy everything again including my silverback collection is not anything I want to do again.

[midside]

I have a CD/DVD duplication business and go through thousands of discs a year.  The media, while some is much better than others, is made cheaply.  Sometimes I can go through 500 or more discs without a reject but other times they can have up to a 7-8% failure rate...and that is when they are new.  As someone who uses this media frequently and for a long time, I would not trust it for archiving.  Also, using so many discs is a PITA IMO.  At the same time, there are so many ways a HDD can go bad, and they do, frequently.  As they say, and as I believe, there are 2 kinds of hard drives, ones that have gone bad and ones that will go bad.  If you use hard drives and you NEED your data and you do not subscribe to this way of thinking, then you are setting yourself up for disaster.  Because of these issues with both forms of media, I have opted to use hard drives for their convenience, but in a double, 2 location, RAID5 system as I explained in an earlier post.  But, I know I need to add LTO (tape) backup to safeguard against file corruption or accidental deletion.
On another note, I really think it is overkill or save 24 bit and 16 bit versions of masters along with flac in 24 and 16 and final version and final flac, etc....
Just use a program that does non-destructive editing like samplitude.  Then, you only need to save the original master files.  But, I also save a few other versions sometimes for convenience like a CD image file for duplication for instance or mp3 for portability, etc.

[gmm6797]

midside: any brands of DVDs you use more often with less failures?  Any info you can give, even if only personal experience, would be good to know

[midside]

Short answer = Taiyo Yuden

Do not buy name brand or 'branded' media unless you know what it is.  You see, there are only a handful of quality disc manufacturers and companies buy what is available for a cheap price (for the most part) then put their name on it.  So, frequently, you have no real way to know what disc they are using and how the quality of that batch is.  You are much better off buying 'unbranded' discs straight from a supplier.  I highly recommend Taiyo Yuden for their consistency and quality.  Be careful of TY discs on ebay, there are LOTS of fakes out there.  Buy them from a reputable distributor.

[Lil Kim Jong-Il]

Media is important yes but the health of your drive makes a difference too.  The discs I'm losing were some of the ones I burned on my old drive just before it finally couldn't verify more than 50% of the time I tried to burn a disc.  Even TYs will fail if the burner is ailing.

[yousef]

For what it's worth, my solution is to have three hard drives each of which have 24bit unedited original recordings for archive purposes and 16 bit tracked out FLACs (that may or may not have had eq etc applied) for ease of making 'listening' copies. There are also copies of my info files, md5s, FLAC fingerprints etc on each drive.

As soon as one drive fails, I'll be copying everything across to a new one - short of a house fire or some sort of catastrophic buffoonery on my part, I can't imagine wiping out three hard drives at once, so I think I'm relatively safe.

And in the event of a house fire, my collection of unedited 24bit files will probably be some considerable way down my list of priorities.

I don't really see a place for dvd-r media in terms of cost, space considerations or reliability.

One thing I've convinced myself is that my CDR and DVDR are never gonna have the shelf life of...say...an analog Maxell XLII from back in the day.

Dunno if you're just playing devil's advocate there but the way I see it is that even though an analogue source can be left in a box for years and still be playable/listenable (though far from original condition), the price of being able to have an absolutely perfect digital clone of a recording decades down the line is that we have to take greater care of our 'masters'. It's a trade off but one I'm more than happy to enter into.

[Lil Kim Jong-Il]

For what it's worth, my solution is to have three hard drives

<snip>
 short of a house fire or some sort of catastrophic buffoonery on my part, I can't imagine wiping out three hard drives at once

If you have three hard drives, put one at a buddy's house.  That will provide you with a local back-up for immediate access and the security of an off-site backup for the catastrophic case. 

[Craig T]

Duplicate hard drives (2: one at home, one at my office) - FLAC only, post-processing/tracking/etc.

[yousef]

If you have three hard drives, put one at a buddy's house.  That will provide you with a local back-up for immediate access and the security of an off-site backup for the catastrophic case.

You're absolutely right and it's something I've been intending to do for months.

Perhaps what is stopping me is the belief that in leaving the drive elsewhere, I am accepting that my house is going to burn down.

[mdogbucket]

I know when I batch process the 24/48 files to 16/44 that the 16/44 will have header errors and sometimes have sbe's even using cd wave.  flacing fixes all of this.  and I have no need for the wave files so they get deleted.  I save and listen to the flac files only.  so by flacing and getting rid of the wavs you are saving a lot of space.

Forgive me but what are "header errors" and "sbe's"?

I save masters as 24 bit/96k wav files and then save edited tracks as 16bit/44.1 wavs for burning CDs.

I have never understood a reason for saving a flac version when 1tb hard drives are available for under $100 unless.... the flac conversion process cleans things up in some way?

[lpmaskman]

I archive the raw wav and the mastered flac also to a removable HDD.

[kindms]

I know when I batch process the 24/48 files to 16/44 that the 16/44 will have header errors and sometimes have sbe's even using cd wave.  flacing fixes all of this.  and I have no need for the wave files so they get deleted.  I save and listen to the flac files only.  so by flacing and getting rid of the wavs you are saving a lot of space.

Forgive me but what are "header errors" and "sbe's"?

I save masters as 24 bit/96k wav files and then save edited tracks as 16bit/44.1 wavs for burning CDs.

I have never understood a reason for saving a flac version when 1tb hard drives are available for under $100 unless.... the flac conversion process cleans things up in some way?

sbe's = sector boundary errors and only potentially effect 16/44.1 wavs. I am surprised that the above poster has issues with sbe's if they are using CD WAV Editor. My understanding was that tracking in cd wav editor prevents SBE's.

Header errors could be alot of things but I think he is referring to a conversion where the header doesn't "record" the correct bit depth / sampling rate. It is easy enough to correct but can be timely if you have a lot of them to correct

If you stay in the 24 bit realm then you generally will not encounter a lot of these issues.

I archive in FLAC as it adds a checksum to the file. This way if it is altered corrupted etc. When you convert back to wav, it would alert you to a problem etc. with a raw wav file you have no quick easy way to determine if there was a problem copying & pasting or moving files around etc. Your wav file may playback fine for the most part but errors can be introduced and you wouldn't have a way to no this quickly etc. More or less a peace of mind thing IMHO

Also of course it saves space on the HDD and smaller files transfer faster etc etc

[yawnfactory]

hard drive and multiple dvd copies of both the raw and completed show. a little overkill, but losing a master that has not been shared would be bad news.

[Fried Chicken Boy]

Short answer = Taiyo Yuden

Qft.  If you're serious about using burnable media for archival purposes, TY is the only way to go.  That being said, there's no method/medium for storage that's fail-safe.  I've personally had just as many harddrives fail on me as CD's and DVD's that have become unreadable.  Difference being that when a HD fails, you have the potential to lose a lot more than just a few recordings; the size of a CD/DVD limits the amount of information you can put on and possibly lose on one disc.  I've gotten in the habit of burning two copies of my original masters to DVD's with accompanying MD5 checksums and then verifying the checksums periodically.  If one set of the masters fail (hasn't happened yet, knock on wood), I have the backup and can burn another.  Not a perfect solution but helps me sleep at night. 

[RobertNC]

i stash all of my recordings (24/48) as raw masters on an external H.D. (Terra) -- Staples on sale for $120.00!

If it is the typical book style external hard drive with a plastic enclosure and a USB/Firewire connection, I would advise switching to something else.  These drives just don't have the shock resistance and there is no fan.  They have a notoriously high failure rate and especially when you are talking about high density like 1 TB.

I had a couple of these drives die on me.

I advise checking out NewEgg.  You can buy an external enclosure with a fan and better mounting, and then load your own.  I am now using a 2 bay cooled enclosure with 2 Western Digital 1 TB Black Caviar drives in it.

[gmm6797]

speaking of TY media, I have some "bulk CDs" that are not labeled, and I cant remember if those are TY or not... does anyone know of an app that reads the media to see who made it? I know there used to be an app like that out there, but i cant find it

[Brian Skalinder]

does anyone know of an app that reads the media to see who made it?

Can't remember which one I used.  Take your pick:  http://www.cdmediaworld.com/hardware/cdrom/cd_utils_2.shtml