Topic: Archiving master WAVs to FLAC?

[BlingFree]

I tried to search for this topic already so forgive me if this has already been discussed.

Lately I've been archiving 24/48 wav of my band's practice sessions to FLAC... something I never considered for my recordings of pro bands.

Recording in 24/96 is becoming too rich for my blood and I'd MUCH rather archive to 1 dvd rather than 2-3.

SO...

I'm sure someone will be happy to point this out as a sin but...

if an md5 checks out on a master wave and an uncompressed flac then... no loss, right? thus making it safe and maybe even practical to archive to FLAC??

An official ruling on this would be great. It would save precious archival space and tons of bandwidth.

[OFOTD]

I tried to search for this topic already so forgive me if this has already been discussed.

Lately I've been archiving 24/48 wav of my band's practice sessions to FLAC... something I never considered for my recordings of pro bands.

Recording in 24/96 is becoming too rich for my blood and I'd MUCH rather archive to 1 dvd rather than 2-3.

SO...

I'm sure someone will be happy to point this out as a sin but...

if an md5 checks out on a master wave and an uncompressed flac then... no loss, right? thus making it safe and maybe even practical to archive to FLAC??

An official ruling on this would be great. It would save precious archival space and tons of bandwidth.

Yes and Yes

[datbrad]

If you are dropping to 24/48, you can still get over 4 hours at full wave on a recordable DVD. Given this, you can make a raw clone of the original file as recorded at the show on DVD. Then, for a second archive you could convert the raw wave files to flac and keep them on an external drive and get more storage capacity. But, my personal view is that at least one raw archival copy without any post processing or file conversion/compression of any kind should be made representing a "master" recording.

[stevetoney]

But, my personal view is that at least one raw archival copy without any post processing or file conversion/compression of any kind should be made representing a "master" recording.

datbrad...so is it traditional that the raw unretouched file is called the 'master'.  I'm just curious about terms, because I was maybe wrongly thinking that the 'master' was the term used for the thing that results after the engineer does all of his magic to it, such as EQ, filtering clicks/digi crap, etc. 

[datbrad]

But, my personal view is that at least one raw archival copy without any post processing or file conversion/compression of any kind should be made representing a "master" recording.

datbrad...so is it traditional that the raw unretouched file is called the 'master'.  I'm just curious about terms, because I was maybe wrongly thinking that the 'master' was the term used for the thing that results after the engineer does all of his magic to it, such as EQ, filtering clicks/digi crap, etc. 

You are correct that the term "master" used for studio and live recordings made for commercial release typically refers to the final 2 track mix down of a multi-track audio recording. This 2 track "master" is what is copied and sent to the production facilities for printing CDs/DVDs. The copies are also sometimes refered to as "masters".

For the regular hobby taping world, the media used at the actual show to make 2 track audience recordings, analog tapes and eventually DATs, were always considered raw "masters". Copies made from them were just that, copies of the "masters" made at the show. This was really a big deal during the cassette period, when generations of copies from the "master" had artifacts such as increased tape hiss, making the "master" always the best version.

Since we no longer keep recordings on the media that was used to initially capture them, true "masters" no longer exist. However, to me, the original raw file represents the "master" and I always back up every show I record directly from the CF card to 2 DVDs, one DVD-R and one DVD+R. Then I copy the file from the CF to an external drive on my PC for whatever post work I choose to do, and only then erase the card for it's next use.

So, I consider the DVDs to be "masters" since they were burned directly from the CF card with no changes or copied stages through intermediary hard drives. The DVD folders are identical to the folders that were on the card. This is just my own piece of mind method, and I am in no way saying it's the best or only way to go, just the way I have my head wrapped around the process today.

[BlingFree]

+1's for all.

Thanks for responding. Sorta darned if I do darned if I dont.

Since no one has officially said no then... I guess I'll give it a try and see what gives.

If anyone has a real strong negative arguement please feel free to encourage me. I'm just trying to save a buck really.

[adrianf74]

Hmm... This has always been a bit of a bone of contention between the two sides. I've been taping since my WM-D3 days (although I've been on a bit of a sabbatical the past four) so I feel I have some insight on this.

I mean, I remember the days where many of us were backing old analog tapes > DAT at either 48k, 44.1k, or even 32k and there was a huge divide on what the "correct" sample rate was (I used 32k for a lot of stuff just because the audience analog tapes I had didn't really have the dynamic range to warrant wasting tapes).

Regardless of my sidetracking, I think a valid point was made.  As long as your md5 file has a matching crc between your WAV and FLAC files, you're pretty much assured that "nothing" has happened to your original file.

That's not to say that you shouldn't keep a copy of the original non-FLAC'ed WAV somewhere else as it's a "true master".  Probably the idea of archiving your FLACs to an external hard-drive makes more sense than burning those to a DVD as well (if you're burning the originals).

DVD's can be finicky from one drive to the next (my wife's Toshiba Laptop burns discs that my HP Laptop tends to get nervous over - and vice versa - and both drives are made by Panasonic).  The best way to do this is to back the files up on a physical drive (for instance, I'm fortunate that I can pick up a 250GB WD 5400 RPM "Passport" external drive for $120 at Costco in Canada).  That's A LOT of FLAC files.

On another note, I recorded a lot of shows in between 2001 and 2004 on MiniDisc (with lossy compression).  The recordings were then transferred via Analog to my PC (using a relatively hi-end audio card).  All of my shows were burned to CD using CDRWin and were track-split to WAVs to create FLACs which were archived on Mitsumi DVD-R media.   I have no real problems, personally, with using FLAC.  I like the fact that it shrinks file sizes without affecting playback.  The only other option you have is to look at Apple Lossless (http://en.wikipedia.org/wiki/Apple_Lossless) which a friend of mine is using to archive his entire CD collection to a 1TB hard drive.  You can play these files back on your iPod without issue as well which makes it convenient.

Hope I've added some food for thought.

[boyacrobat]

my masters dont see flac frontend at all
i dont trust flac to be faithful to the concept of sound design.

my drives and me are worthy of wavs in storage.
extra bucks but a pure mind and peace to  just listen to a wav as clean as i can.

i feel free more like this
just  a  me  thing

g

[rowjimmytour]

If you are dropping to 24/48, you can still get over 4 hours at full wave on a recordable DVD. Given this, you can make a raw clone of the original file as recorded at the show on DVD. Then, for a second archive you could convert the raw wave files to flac and keep them on an external drive and get more storage capacity. But, my personal view is that at least one raw archival copy without any post processing or file conversion/compression of any kind should be made representing a "master" recording.

I make a DVD-R w/ the virgin 24/48 wav files sometimes 4 tracks and depending the length of the show I can some times put the 24 bit flac files w/ ffp and text on one disc but usually have to use two. Then I slip the disc(s) in a plastic sleeve in a metal file in my closet for storage.

[emalvido]

I asked myself the same question some months ago, and most of the fellow tapers in here recomend wav to DVD-R
Yep......archive your recordings directly to a wav to DVD-R
Ed 

[dgale]

I guess I have a different perspective on this - once you burn the wav file to DVD, transfer it to another HD or otherwise move or copy it, how do you confirm that this move/process occurred in such a way that the new file (yes, it's actually a new file now that you've moved it, copied it, burned it etc. - it is no longer the "master" WAV file once you move or alter it from the drive or card on which it was originally recorded) it a bit-for-bit accurate copy of the original file?  In the somewhat older days people when to extra lengths to use EAC to ensure that as WAV files were burned to discs and then extracted later that this was done without any uncertainty as to the integrity of the file as it is burned/extracted along the way.  Why would burning a master WAV to DVD and then at some point in the future pulling it back off that DVD for some sort of use be any different? 

When you convert your master WAV>FLAC, you get ffps (and if you choose to create them, MD5s as well) that you can use to verify that the file remains bit-for-bit accurate as it is burned to disc, pulled off the disc, copied to drives, transfered electronically etc.  Seems like most people take the ffps for granted these days and don't even bother to test them as files are copied/transfered, which is a mistake as that is the point.  Personally I could care less that FLAC compresses the files to save space - I use it for the certainty that the files remain true to the original source. 

I used to be paranoid and would back-up my 24 WAV files both as a FLAC on DVD and also just burn the WAV straight to DVD, until I realized it was pointless IMO and that down the road I would never use the WAV version - I would always use the FLAC version as I could run test the ffps, run the MD5s and know that the file had not been corrupted in anyway.

[DSatz]

> "i dont trust flac to be faithful to the concept of sound design"

Lossless compression is just that--lossless. The resulting file is smaller, but it contains EXACTLY the same data as the original. There are no sonic side-effects whatsoever; this is not even a matter of opinion or "who says so." The musical content is demonstrably, bit-for-bit, 100.0% identical before and after compression.

On the other hand, data errors occasionally develop in a storage medium due to accidents or defects. The net effect of these errors (all other things being equal) is less when data is uncompressed. Since we're talking about archival storage here, that is something to consider. Personally, I never compress files when I archive them; blank discs are relatively cheap nowadays and I just don't think there's enough benefit to it.

--best regards

P.S.: As for the term "master"--in the recording industry it generally means the recording from which the commercial product is made. The terminology comes from the analog era, when each copy added noise and distortion; in the digital era that is no longer the case, and a copy can even be "better" (have fewer bit errors) than whatever it was copied from.

In the analog era a master tape was almost never the original session or live recording--that was kept safely in the vault, and the (generally) mixed, equalized and producer-approved copy from which the lacquer masters were cut was then "the master tape." That copy might well have been third- or fourth-generation.

The main exception was a few audio fanatics who recorded directly to two-track tape--mixing "live" if there were more than two microphones--and who then edited that original tape and risked bringing or sending the result to the lacquer mastering studio. I can think of exactly two engineers besides myself who operated that way: Marc Aubort (famous for his Nonesuch recordings with Joanna Nickrenz) and the late Ralph Dopmeyer (Titanic Records). The benefit in sonic clarity was immediately obvious. Ralph didn't even used to use Dolby noise reduction, yet his recordings sounded quieter and cleaner than most from major labels. Anyway, only in cases of this kind would the "master" be the same as the "original."

--best regards

[RebelRebel]

Marc Aubort (famous for his Nonesuch recordings with Joanna Nickrenz)

- still going strong too. (had a great article in Tape OP last year or the year before) with his 221Bs and a mackie mixer of some sort.

I love his recordings and grab them whenever I see him credited. /ot

[dgale]

> "i dont trust flac to be faithful to the concept of sound design"

Lossless compression is just that--lossless. The resulting file is smaller, but it contains EXACTLY the same data as the original. There are no sonic side-effects whatsoever; this is not even a matter of opinion or "who says so." The musical content is demonstrably, bit-for-bit, 100.0% identical before and after compression.

Thanks for saying it better than I could - As for the above quoted comment about FLAC not being "faithful to the concept of sound design" - what exactly is that supposed to mean?  Indeed, there is no debate about "FLACing" WAV files - it is a bit-for-bit perfect copy of the original file and thanks to the ffps (and md5s if you make them), you also have the abilty to verify that the files remain bit-for-bit accurate as they get copyed,  electronically transfered etc.

On the other hand, data errors occasionally develop in a storage medium due to accidents or defects. The net effect of these errors (all other things being equal) is less when data is uncompressed. Since we're talking about archival storage here, that is something to consider. Personally, I never compress files when I archive them; blank discs are relatively cheap nowadays and I just don't think there's enough benefit to it.

I feel that the compression side of FLAC is a secondary benefit and agree that blank media is cheap, so it really is not that big a deal.  If one is concerned about the compression and the magnitude of a potential flaw vs the same flaw on an uncompressed file (as stated above), then lower the level of comression when you create the FLACs (i.e. choose level 1 when you create the files).  The bonus of FLAC is that if such an error were to occur in your file down the road due to accidents or defects, then at least with FLAC you can run the ffps and you would know your file or disc had developed a flaw...how would you necessarily know this with a WAV burned to DVD or stored on a HD?  A big flaw would probably be obvious but small flaw(s) that might only alter a handful of bits here and there would likely not be detectable by listening or viewing the WAV.  ffps on the otherhand would fail for the tracks that were not bit-for-bit accurate.

I resolve the concern about having flaws develop down the road by making sure I back up irreplacable files in more than one place.  I typically make extra copies of irreplacable files and give them to one or more friends who keep them for safekeeping and I do the same for them...FLAC or WAV, what are you going to do when the house burns down and your DVDs or HDs are all a melted pile of goo?

[JasonSobel]

The bonus of FLAC is that if such an error were to occur in your file down the road due to accidents or defects, then at least with FLAC you can run the ffps and you would know your file or disc had developed a flaw...how would you necessarily know this with a WAV burned to DVD or stored on a HD?  A big flaw would probably be obvious but small flaw(s) that might only alter a handful of bits here and there would likely not be detectable by listening or viewing the WAV.  ffps on the otherhand would fail for the tracks that were not bit-for-bit accurate.

you've mentioned a couple of times the advantage of FFP's and md5's to verify, bit for bit, the integrity of FLAC files.  and that's definitely a good thing and should be done.  But there's nothing that says md5 checksums are exclusive to FLAC or SHN.   personally, I create an md5 checksum for all my "master" .WAV files before I burn to data DVD.  After they are burned, I verify the md5 and I know that the burned copy is identical, bit for bit.

[dgale]

you've mentioned a couple of times the advantage of FFP's and md5's to verify, bit for bit, the integrity of FLAC files.  and that's definitely a good thing and should be done.  But there's nothing that says md5 checksums are exclusive to FLAC or SHN.   personally, I create an md5 checksum for all my "master" .WAV files before I burn to data DVD.  After they are burned, I verify the md5 and I know that the burned copy is identical, bit for bit.

That's true and point well taken, although FLAC and SHN files are data files and if I recall correctly from trying to understand the purpose and benefits of SHN over WAV in the early days of SHN, data files more readily handle copying, moving around in Windows etc. then do audio WAV files - perhaps I am mistaken about this but that was my recollection from when their purpose was first explained to me...back in the day when you had audio CDs and wanted to copy them, you had to extract the WAVs via EAC or similar extraction program (at least if you wanted to minimize errors you did it this way) to get them from CDR to HD, while SHN (and subsequently FLAC) files could be dragged and dropped in Windows Explorer.  Of course you could do this with WAV files as well but my understanding was they were more prone to having errors introduced doing this over when you properly extracted them, while it was much more of a non-issue with data files.  Someone much more in the know about such computer format issues should chime in here.  Basically, it was explained to me that the benefits of SHN/FLAC were three-fold - lossless compression (which was more of a big deal IMO back in the days before DVD-Rs and huge HDs), ffp/md5 (and yes you can create md5s for any files), and the fact that they were converted to data files instead of WAV audio format.   

[rowjimmytour]

I guess I have a different perspective on this - once you burn the wav file to DVD, transfer it to another HD or otherwise move or copy it, how do you confirm that this move/process occurred in such a way that the new file (yes, it's actually a new file now that you've moved it, copied it, burned it etc. - it is no longer the "master" WAV file once you move or alter it from the drive or card on which it was originally recorded) it a bit-for-bit accurate copy of the original file?  In the somewhat older days people when to extra lengths to use EAC to ensure that as WAV files were burned to discs and then extracted later that this was done without any uncertainty as to the integrity of the file as it is burned/extracted along the way.  Why would burning a master WAV to DVD and then at some point in the future pulling it back off that DVD for some sort of use be any different? 

When you convert your master WAV>FLAC, you get ffps (and if you choose to create them, MD5s as well) that you can use to verify that the file remains bit-for-bit accurate as it is burned to disc, pulled off the disc, copied to drives, transfered electronically etc.  Seems like most people take the ffps for granted these days and don't even bother to test them as files are copied/transfered, which is a mistake as that is the point.  Personally I could care less that FLAC compresses the files to save space - I use it for the certainty that the files remain true to the original source. 

I used to be paranoid and would back-up my 24 WAV files both as a FLAC on DVD and also just burn the WAV straight to DVD, until I realized it was pointless IMO and that down the road I would never use the WAV version - I would always use the FLAC version as I could run test the ffps, run the MD5s and know that the file had not been corrupted in anyway.

My main reason to burn the untouched wav files to DVD-R is for matrix and MS recordings but I still do them all. I figure I might be able to come up w/ a better mix or edit later from my self or some one else so I can easily access these files and mix or edit away.
Peace

[JasonSobel]

you've mentioned a couple of times the advantage of FFP's and md5's to verify, bit for bit, the integrity of FLAC files.  and that's definitely a good thing and should be done.  But there's nothing that says md5 checksums are exclusive to FLAC or SHN.   personally, I create an md5 checksum for all my "master" .WAV files before I burn to data DVD.  After they are burned, I verify the md5 and I know that the burned copy is identical, bit for bit.

That's true and point well taken, although FLAC and SHN files are data files and if I recall correctly from trying to understand the purpose and benefits of SHN over WAV in the early days of SHN, data files more readily handle copying, moving around in Windows etc. then do audio WAV files - perhaps I am mistaken about this but that was my recollection from when their purpose was first explained to me...back in the day when you had audio CDs and wanted to copy them, you had to extract the WAVs via EAC or similar extraction program (at least if you wanted to minimize errors you did it this way) to get them from CDR to HD, while SHN (and subsequently FLAC) files could be dragged and dropped in Windows Explorer.  Of course you could do this with WAV files as well but my understanding was they were more prone to having errors introduced doing this over when you properly extracted them, while it was much more of a non-issue with data files.  Someone much more in the know about such computer format issues should chime in here.  Basically, it was explained to me that the benefits of SHN/FLAC were three-fold - lossless compression (which was more of a big deal IMO back in the days before DVD-Rs and huge HDs), ffp/md5 (and yes you can create md5s for any files), and the fact that they were converted to data files instead of WAV audio format.   

you are correct about the audio CD format, in that they have to be extracted with EAC or some other similar program, and that they are much more prone to errors.  But we're all talking about burning data CDs and data DVD's.  If you burn a .wav file to a data DVD (or CD), then it's just like burning any other file (FLAC or SHN or whatever), and you can copy the .wav files back to the HD with the "drag and drop", and it'll be just as error-free as any FLAC or SHN file.  The key here is to not burn an audio CD, but keep everything as a data disc.

[BlingFree]

> "i dont trust flac to be faithful to the concept of sound design"

I'm wondering if you mean Sound Design like the 722? IN which case I understand. A wav file transfered from a 722 and then FLAC'd right away produces errors while encoding. I'm assuming because SD puts non-musical data in the wav file for later. Time code maybe or other data. I'm not sure and personally would be happy if it didn't do that anymore.

So in essence the md5s, in this case, wouldn't match between the original and the FLAC'd because of the missing data bits.

[morst]

if an md5 checks out on a master wave and an uncompressed flac then... no loss, right? thus making it safe and maybe even practical to archive to FLAC??

If you want to be really sure, you could create an MD5 for each original WAV file, then losslessly compress using FLAC, but SAVE THE WAV's MD5 along with the FLAC files. That way whenever you decode the FLAC files, you can check the resulting WAV against the original MD5 signature to make sure it's still there.

Seems like overkill to me, but it's possible that it could help at some point. Of course if you want to be really finicky, you will want to decode each flac you encode, and check the MD5 before creating the archival backups, just to make sure it encoded correctly!

Personally, I do this:
1- copy original files to a hard drive on my computer
2- FLAC encode the files
3- load the original file into audacity and do any processing needed, including tracking. I prefer to store the audacity session on a separate hard drive from the original file, just in case.
4- burn 2 identical DVD-R discs with the FLAC's of the original unprocessed files (renamed for clarity) as well as the final output of the processing.
5- share the music. The more people have it, the less likely it is to disappear!

I need to start keeping my second set of backups in my storage unit, just in case! Really, making two backups is pretty much a RAID mirror.
By the way, I think DVD-R is more reliable for long term storage than DVD+R. The +R discs have faster burn rates, but the -R discs are supposed to last longer.

[DSatz]

As a reply to the last several messages in this thread: The advice to store the wave audio data as data files rather than in a directly playable audio disc format is well taken. The Orange Book standard for CD-ROM and all standards for DVD-ROM formats (DVD-R, DVD+R, etc.) include checksumming for the data of each sector. The Red Book standard for audio CDs does not. Basically if you can transfer a data file from a DVD-R, DVD+R or CD-ROM without encountering read error messages from your operating system, you should have the original data.

In addition, just focusing for a moment on CDs (audio versus CD-ROM), while the CD-ROM data format is based on the physical format of the audio CD (due to the fact that the medium was invented for audio use, and its use for data came only later when the audio medium had established itself), audio CDs were not designed to be "randomly accessed" nor necessarily to give bit-accurate reading. The logical data sectors on an audio CD aren't numbered; a player can only infer where it is reading from. This is why software such as EAC is necessary for high-speed digital audio extraction if you don't have a Plextor or other drive with firmware that has the equivalent logic built in. If the player's output data buffer fills up, the disc keeps spinning but the reading stops until there is room in the buffer. Then reading has to resume, but exactly where?

All disc media have this same problem, but the other ones (such as your hard drive) cope with it by numbering their logical sectors so that they can seek to a particular sector and "know" where they are. With audio CDs, the position can only be estimated and contents compared, etc., to establish exact head location. With CD-ROMs the logical sectors contain additional data which include sector numbers, so that precise seeking is possible and data are not repeated or skipped during an interrupted transfer (the normal kind at high speeds).

CD-ROM was designed as a distribution medium for software as well as a carrier for enterprise business data; undetected, uncorrected bit errors are of course unacceptable in those applications. In the playback of audio CDs a set of techniques known as "error concealment" based on psychoacoustic assumptions is used; these techniques cannot be used on CD-ROM. Thus an additional layer of error detection and error correction data is included in the Orange Book format, such that the bit error rate is reduced by a factor of 1,024 over audio CD, and the error detection is powerful enough that it is mathematically very unlikely to be fooled even once during the estimated service lifetime of a player.

--best regards

[morst]

music > microphones > a recorder of some sort

DSatz- your posts are always awesome. Glad you're here. +T bro!

[digifish_music]

Flac results in ~30-50(max)% compression. I am not convinced that it's worth the hassle given 500 Gb drives are so cheap...and as are DVDs?

digifish

[datbrad]

Also, when burning data DVDs, remember there is a data verification step where the freshly burned disc is immediately read and the data on it compared to the source file. If there are any differences, the disc ejects with a message that the disc "failed data verification". If you get the "successful" message with a green check mark or whatever, this means the file on the DVD matches the data of the original file exactly.