Topic: How do you store your raw/master files?

[page]

I have a bunch of external hard drives and I'm loathe to spend more cash on another set right now so I'm tempted to redo how I store my master files to reclaim enough space to last me till the end of the year (when I'll be more apt to get new drives). How do you store your stuff?

[twatts (pants are so over-rated...)]

I'm doing mostly DAT Transfers, but regardless the idea is the same:

I copy my Master WAV off the Flash Card onto my PC.
I cut off any extra silence with CDWave and save as "bandname2013-06-25 - 1648.wav"
I create a WAV MD5.
I FLAC. (lvl 8 for least amount of space used)
I create FLAC MD5.

I save it all in a "master" folder along with the 1644 Master (FLAC w/ WAV and FLAC MD5s), CUE Sheets, and anything else I have for the show (pix, scan of ticketstubs, etc.).  This "master" folder lives inside the folder I make for FLACs I Seed on ETREE (or however).  It ends up taking up a ton of space...

I save all the FLACs on HDDs...  I'm slowly running out of space, I've had to delete some dusty SHN/FLAC collections that never got played anymore (bye-bye Zeppelin and Clapton)... 

Terry


ETA I ahve approx 6TBs external and 1.5TB internal...  I think my next upgrade will be a pair of 3TBs internal...  That should set me up for at least another year or so! 

[hoppedup]

flac it, put it on two hard drives. I have some files that are on three or more hard drives. I've been meaning to store the redundant hard drives off-site, but haven't gotten around to it yet.

I will pm my address to the first person who responds that storage is so cheap these days that we should just store the .wav files. Then they can send me as many 2TB drives as they can afford.

[kindms]

I burn to physical DVD media. I can usually do 16/44.1 24/48 and master recording (ie whatever file came out of my recorder).

rename and flac all of it creating ffp files for each version

I keep whatever is the highest bit rate recording on HDD for playback purposes.

[page]

has anyone who stores flacs tried to do test decodes of older material and run into issues? I did once but it was off of optical media. I'm curious if anyone's had a problem with hdds (aside from crashes).

[bombdiggity]

I save the original wav/s. 

I save the final edit 24bit (usually as wav again) and if I do a 16bit edit that too but in final aligned flac. 

I try to have all the masters on at least two drives. 

A friend had converted all his masters to flac for storage.  At least one of those conversions was corrupt.  It will still somehow play through but cannot be decoded by any means either of us can find.  That made me not trust doing that with the source files (tests and checks notwithstanding).  I don't know how he processed or qc'd it though... 

Given that I've got upwards of 20 drives around here with all the bittorrent/trades I'm past the point of worrying how much space it takes (in a relative sense). 

[bombdiggity]

has anyone who stores flacs tried to do test decodes of older material and run into issues? I did once but it was off of optical media. I'm curious if anyone's had a problem with hdds (aside from crashes).

Not directly but as I'd typed above I know someone who has. 

Optical media is NOT reliable after a certain number of years (that varies with brand as well as any number of random factors).  It will degrade. 

Redundancy and circulation are the only answers to that issue. 

[Ultfris101]

Hmmm, i haven't had any decode problems with old FLACs but the oldest ones I have are ones I torrented down and didn't record myself. I may need to rethink the value of saving space at the risk of undecodable flacs.

I've been saving raw files in a "raw" directory as wav until I'm done working with the show and then I've been flac'ing them. The directory they stay in is covered by carbonite but takes a while to get up there. I also copy to another local internal drive periodically. I usually keep the CUE sheets but delete the final processed full file, in lieu of saving the 24 bit flac/tracked set.

For my longer standing collection of recordings (4000+ shows) I cracked open all the external drives I had and made a Freenas 4+2 Raidz2 volume (4 data discs + 2 parity. I can lose up to any two discs and still be ok). Currently just 1tb drives but plan to upgrade them all to 2tb at some point to get about 7tb usable. I'll use that as long term storage eventually. I need a mirrored system as a backup too tho.

I'm not bothering with DVD backups anymore. local and remote mirrored  hard drives (ideally some economical cloud solutions eventually). LMA helps a lot with peace of mind for stuff that can be stored there.

[F.O.Bean]

I burn the raw master in data WAVS and FLAC on separate DVDRs!

[Chuck]

I store the WAVs and FLACs separately on DVD-R's and two mirrored hard drives.

[adrianf74]

I store the WAVs and FLACs separately on DVD-R's and two mirrored hard drives.

I don't go that far but store the raw FLAC files in one folder and the mastered files in another.  They exist on one drive (main media computer) that has a parity drive associated with it (so it's, in theory, backed up).  I then have the files stored on two portable USB 3.0 hard drives (one stored on site and one off site which comes back to me every 3-4 months for updating).

I don't think I can be more safe than that.

[F.O.Bean]

I have four copies of every set I record, all onto HQ Verbatim DVDRs! Raw WAV/Raw FLAC/Tracked FLAC 24 bit and Tracked FLAC 16 bit I think I'm good to go

[adrianf74]

I have four copies of every set I record, all onto HQ Verbatim DVDRs! Raw WAV/Raw FLAC/Tracked FLAC 24 bit and Tracked FLAC 16 bit I think I'm good to go

I did that once upon a time... on DVD-R's and ran into some serious read issues with even though they were certified Taiyo Yuden discs and were stored properly. I'd sooner store shows on hard drives.

[F.O.Bean]

I have four copies of every set I record, all onto HQ Verbatim DVDRs! Raw WAV/Raw FLAC/Tracked FLAC 24 bit and Tracked FLAC 16 bit I think I'm good to go

I did that once upon a time... on DVD-R's and ran into some serious read issues with even though they were certified Taiyo Yuden discs and were stored properly. I'd sooner store shows on hard drives.

I just went from 10-12 year old Verbatim CDRs>INHDD and everyone transferred perfectly. I know some problems might arise, but so far, so good. And that's why I burn 4 separate copies. Also, 99% of what I record goes to archive.org, and the other 1% goes to bt.etree.org, so if I ever needed anything because of a bad DVDRS, I always have those to back me up as well

[Chuck]

I have had good luck with 10 year+ old CD-R's too. My DVD-R's are not as old, but I haven't found any "bad" discs when I go in and re-seed old copies of shows I've recorded years ago. I store them in Tyvek envelopes in plastic bins in a dark closet.

[adrianf74]

CD-R's were never a problem.   The DVD-R's, however, were a nightmare.  I vowed to never do that again (but that was my experience).

[Datfly]

My name is Datfly and I am an addict.  Hard drives are the only way for me twice.....
My Datfly Master HD's are in the back left. They contain the original wav. & a FLAC copy
on two 4TB HD's

[Ultfris101]

Great googly moogly.     

And I thought 8 externals was bad.

What do you use to index them? I found a program called Cathy Lister that is simple and gets the job done.

[Phil Zone]

I keep the original wav and the flacs, I have a bunch of external storage so why not. As long as your organized the storage is limitless unless your short in money or room space, but drives are small.

[page]

I keep the original wav and the flacs, I have a bunch of external storage so why not. As long as your organized the storage is limitless unless your short in money or room space, but drives are small.

There in lays the problem, id rather spend my cash on gear than hdds...

[fatstratcat]

Mine are all in flac; raw 24 bit, and 16 bit edited files on two HDs, and backed up on DVD+Rs just to be on the safe side. after reading the comments on decoding problems, I'm changing to keeping raw 24 bit wav too. I used to just keep everything on DVDs, but I saw a story on TV about the problems they have at the Library of Congress regarding deteriorating CDs and DVDs; some of them you could clearly see through - like the clear plastic disc at the bottom of a spool.

Is anyone using solid state HDs?

[404 Not Found]

I am running two identical and RAID 0 mirrored 12 TB towers that I put together (6 2TB Seagate Barracuda HD's) with USB3 for access and a SD card reader in the computer to move the files from my recorder to the outboard drive. I still have all my CDR's from the days of ripping DAT to CDR using an HHB800 that was done before the day of DVDR as backups. Everything new in the past 4+ years was on CF and now SD cards.  I keep both my 24 and 16 bit files indexed per folder of the show itself, in separate sub folders.  The Outboard towers I made can easily be disconnected and removed from the house in case of emergency.  It's one of the things I would try and save if the house was on fire

[cashandkerouac]

for each show recorded i maintain two file folders: one for the raw master files, which i keep in original WAV format.  in the other folder i keep the processed 24 bit files in flac format.     

[candor]

Is anyone using solid state HDs?

I store copies of my originals on both ssd's and portable hard drives kept in a safe deposit box-which I have outgrown.  I keep copies and edited files, flacs, etc on external HDs at home.  I once heard someone opine a file wasn't important to you if you didn't have more than one copy.

[shownomarcy]

I keep the original wav files and I have the edited final (usually) tracked versions in flac.
Stored on internal and external hdd.
I dont have much space so Im just about to buy more externals, so everything would be on 2x externals.

[tom.lac]

I create an md5 of the wav masters.
I create flac files of the orginal wav files with ffp
I save all on a Raid 5 NAS (synology)

[ArchivalAudio]

I save wavs and all edits ..
Is anyone storing masters on the cloud yet?

[rigpimp]

I have crates of externals like datfly's shelves.  Raw .wav, .flacs, .shns, video, etc

I am now taking my backup DVD spindles (FLAC/SHN) and loading it all onto a 20tb QNAP NAS for FTP, streaming, etc.

[faninor]

I am running two identical and RAID 0 mirrored 12 TB towers that I put together (6 2TB Seagate Barracuda HD's) with USB3 for access and a SD card reader in the computer to move the files from my recorder to the outboard drive. I still have all my CDR's from the days of ripping DAT to CDR using an HHB800 that was done before the day of DVDR as backups. Everything new in the past 4+ years was on CF and now SD cards.  I keep both my 24 and 16 bit files indexed per folder of the show itself, in separate sub folders.  The Outboard towers I made can easily be disconnected and removed from the house in case of emergency.  It's one of the things I would try and save if the house was on fire

If a drive fails are you able to replace and mirror that single drive from your other tower or do you rebuild the entire 6 disk array?

[404 Not Found]

I am running two identical and RAID 0 mirrored 12 TB towers that I put together (6 2TB Seagate Barracuda HD's) with USB3 for access and a SD card reader in the computer to move the files from my recorder to the outboard drive. I still have all my CDR's from the days of ripping DAT to CDR using an HHB800 that was done before the day of DVDR as backups. Everything new in the past 4+ years was on CF and now SD cards.  I keep both my 24 and 16 bit files indexed per folder of the show itself, in separate sub folders.  The Outboard towers I made can easily be disconnected and removed from the house in case of emergency.  It's one of the things I would try and save if the house was on fire

If a drive fails are you able to replace and mirror that single drive from your other tower or do you rebuild the entire 6 disk array?

There are two of these towers and they are mirrored.  If a drive fails, then I can replace it.  If there are more than one drive that could fail, I do have Seagate Outboard 1 and 2 TB drives I also keep as a backup to these as well, but they are only for backup storage.  I am all for backing it up and copying.  The advantage of the mirrored towers I made is that they run USB3 and eSATA, so my transfer speeds are pretty quick especially for streaming purposes between computers on my network at home.  The other drives I have I do not use for anything but storage. 

In about a year I will be looking at possibly updating the drives since they are getting old, but knock on wood, I have never had a prob with these.

[rastasean]

What do you use to index them? I found a program called Cathy Lister that is simple and gets the job done.

looks like an offline method: a sticker numbered with some kind of short descriptor and i guess the key is written on an index card stored near the drives.

[Phil Zone]

im ordering two new segate hard drives 1tb each to add to my collection but i keep the mastered 24 bit waves and the flacs as well. and i use one HD as a master drive and one is a mirror for a back up

[tim in jersey]

Recent experience with optical media: local taper-buddy of mine recorded a couple different sets where he grabbed both SBD and AUD on 2 different M-10's. Wanted me to do a matrix-in-post of the sets. At a show a few days later, he gave me the raw file-sets on DVD+/-R he had burned on quality media the same day he passed them off to me. The discs were handled gently by both he and I. No scratches, or other obvious physical defects.

50% of the files were unreadable on all of my machines. I have regular access to about 6 laptops with internal drives and another 2 or 3 external USB drives, including the venerable Plextor.

I'm not relying on optical ever again.

[rigpimp]

What do you use to index them? I found a program called Cathy Lister that is simple and gets the job done.

looks like an offline method: a sticker numbered with some kind of short descriptor and i guess the key is written on an index card stored near the drives.

I have always used a small app called JR Directory Printer.

http://download.cnet.com/JR-Directory-Printer/3000-2094_4-19648.html

You can make a file (Word/PDF, etc) that is made from printed dirs, subdirs and include cool metrics like # of files and folder/file sizes.  You can do generic dir names or tell it to recurse.  I label drives A-Z (and AA-ZZ) and then put a sticker on the drives so that I know which drives contain what.  I am then left with a master file that is searchable.  It is a really cool app and it works instantaneously.
A sample of the JR Directory Printer output looks like this:

  Directory of I:\Dropbox\Phish Phamily\MP3\ph2013-07-31.kreider.NeumannMidSide.mp3f\
 
   I:\Dropbox\Phish Phamily\MP3\ph2013-07-31.kreider.NeumannMidSide.mp3f\
   =====================================================================
   phish2013-07-31.txt                      1 KB     8/3/2013 10:35:26 PM      a
   phish2013-07-31d1t01.mp3              9506 KB     8/3/2013 10:35:26 PM      a
   phish2013-07-31d1t02.mp3              7627 KB     8/3/2013 10:35:06 PM      a
   phish2013-07-31d1t03.mp3              5612 KB     8/3/2013 10:34:52 PM      a
   phish2013-07-31d1t04.mp3             12123 KB     8/3/2013 10:34:40 PM      a
   phish2013-07-31d1t05.mp3             11602 KB     8/3/2013 10:34:14 PM      a
   phish2013-07-31d1t06.mp3              5299 KB     8/3/2013 10:33:52 PM      a
   phish2013-07-31d1t07.mp3              9128 KB     8/3/2013 10:33:42 PM      a
   phish2013-07-31d1t08.mp3             12351 KB     8/3/2013 10:33:24 PM      a
   phish2013-07-31d1t09.mp3              4098 KB     8/3/2013 10:33:00 PM      a
   phish2013-07-31d1t10.mp3             14620 KB     8/3/2013 10:32:48 PM      a
   phish2013-07-31d1t11.mp3             14951 KB     8/3/2013 10:32:20 PM      a
   phish2013-07-31d2t01.mp3             53178 KB     8/3/2013 10:31:52 PM      a
   phish2013-07-31d2t02.mp3              9015 KB     8/3/2013 10:30:10 PM      a
   phish2013-07-31d2t03.mp3             13043 KB     8/3/2013 10:29:52 PM      a
   phish2013-07-31d2t04.mp3             13001 KB     8/3/2013 10:29:26 PM      a
   phish2013-07-31d2t05.mp3              5213 KB     8/3/2013 10:29:02 PM      a
   phish2013-07-31d2t06.mp3             17803 KB     8/3/2013 10:28:48 PM      a
   phish2013-07-31d2t07.mp3             11225 KB     8/3/2013 10:28:10 PM      a
   phish2013-07-31d2t08.mp3              5323 KB     8/3/2013 10:27:48 PM      a


  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Total 0 folder(s); 20 file(s)

  Total files size: 235 MB; 234729 KB; 240362144 Bytes

  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[Rosettastoned729]

I voted "other" as for me, it's kind of a combination of two of the presented options. I take the raw 16-bit WAVs directly from the recorder via USB (I'm using a Zoom h2n at the moment) and put them on my computer's internal HD (until space becomes an issue - it isn't at the moment for this newbie, though). However, my computer could theoretically crap out at any given moment, and personally, I'm not overly concerned with keeping the raw files if that were to occur as I usually track them, do some EQing, etc. in Audacity. Plus, once space does become an issue, I'll probably just ditch my raw files anyways. I know that may sound sacrilegious to the more experienced tapers or at least to those who are true archivists/completionists, but once I have the touched up/enhanced and tracked WAVs I can put to CD for my listening pleasure, I don't know what purpose my master files are still serving me.

[beatkilla]

As you say your a newbie most likely your EQing touch up skills will become more advanced in the future and you will notice you probably added to much high end etc. to your old masters and without the raw files youu can not do a new EQ version.ALWAYS KEEP THE MASTER FILES.

[page]

As you say your a newbie most likely your EQing touch up skills will become more advanced in the future and you will notice you probably added to much high end etc. to your old masters and without the raw files youu can not do a new EQ version.ALWAYS KEEP THE MASTER FILES.

This.

I dug out a Deer Tick show to upload this past weekend and I realized that I could do so much better now than I did when I taped it years ago.

[tim in jersey]

As you say your a newbie most likely your EQing touch up skills will become more advanced in the future and you will notice you probably added to much high end etc. to your old masters and without the raw files youu can not do a new EQ version.ALWAYS KEEP THE MASTER FILES.

This.

I dug out a Deer Tick show to upload this past weekend and I realized that I could do so much better now than I did when I taped it years ago.

Ditto. I've been listening to a Shockenaw Mountain Boys recording from just a few years back. Acoustic music where the crowd is hear-a-pin-drop-quiet during music but the applause/crowd reaction is thunderous during the remaining strains as the instruments fade out. I've acquired a few new skills (thanks to this forum) and I *know* I can make a more natural-sounding, subtle edit using parallel compression and envelope filters. I can't believe how ham-handed my technique was, and I've improved a lot, but still have a long way to go.

EQ is an entirely different beast. Complete mystery to me. I've been taping for 15 years and anything I've applied EQ to, especially from my early tapes sounds either way too brassy or way too bassy. I attribute that to being over-zealous on either side of the spectrum. Still have yet to train my ear on how to *subtly* apply EQ in the regions that need a little work... But I'm trying, and without the masters from rooms I regularly record in and am familiar with, I have no baseline to compare against.

So yeah, keep those raw masters if you can... They will make you a better taper.