As in Obsessive Compulsive Disorder. It would solve a lot of problems.
As I approach the 1000th post on Daily Photo, I've been gathering the source files for that archive from my various disk drives. It has been an alarming eye opener to see the state of my archive.
One startling fact is that, even though I am moderately compulsive and one of the more organized photographers I know, I lose pictures. As in poof, gone. I had a major hard drive failure a couple of years ago, which took out a swath of data, and I am a big believer in triplicate backup now. Nonetheless, there are gaps in the record over the three years that I've been shooting digital.
A workflow problem I've been struggling to overcome is the issue of using more than one computer to ingest and process images, and managing multiple portable hard drives on location. Keeping the main archive synchronized has been a big headache, and I've royally messed up sometimes. Once I overwrote a half day's shooting on a hard drive, not realizing that two identically dated folders held different images, and I deleted the backup after I thought it was safely on the desktop. I didn't realize my mistake for months afterwards, and after the portable hard drives had been through multiple reformat cycles and the images were long gone from the laptop.
A weak link I didn't know I had is my archiving protocol from cr2 to dng. I just found an entire day's worth of shooting at Williams College, in 2007 no less (by which time I thought I'd worked out the wrinkles in my workflow) that never made it into the conversion queue. I have the original raw files on DVD, and I'm adding them back in now. I only caught this by wondering why there wasn't a link to the original file in my Iview Media Pro archive.
Then there were the three dozen corrupted dng files that Bridge wouldn't open. Again, I restored this from a backup on an optical medium. Another time I couldn't find one raw file, but I knew I had a Photoshop derivative on a DVD. But my Mac wouldn't open up the file, telling me the data on that file (and only that file) was corrupted. My old PC read the file just fine however. It makes me wonder what other disks burned with previous DVD burners are going be troublesome in the future.
There is no end to this problem, so far as I can see. Caring for a digital archive is a little like caring for something that's alive. Ignore it, and your data will die, either because the storage is ephemeral, or, more likely, because technology will abandon the medium (look at the problem we have reading Photo CD files).
I have struggled with digital archives for a long time as well. Mine go back to 1997 though I doubt I have anywhere close to the number of photos you have and I do not have to worry about bringing in photos from multiple computers. When I take pictures on vacation, I will use my laptop and a small external hard drive to hold copies of them so that I can clear off the card, but once I get back home, I import from the laptop onto my main machine. I don't do any editing or processing on the laptop--I'll add metadata there sometimes, but that's about all.
The other thing I have done is abandon optical media altogether. DVDs are too finicky and I found too many corrupt files on them. Luckily I am meticulous and I was able to recover them from the second copy that I keep at work. And, because I'm paranoid about how long the optical media will last, I used to verify the contents of all my optical media at least once a year.
Now, though, I just keep things on hard drives. I have a Windows Home Server that duplicates the pictures across multiple hard drives and, once a week, I hook up an external drive, update it, and take that offsite (to my office at work). And then I have another copy of the RAW files (not the processed TIFFs) on external drives connected to my main computer.
But, as you can see just from what I have written, it takes a lot of care and feeding to keep it alive.
And I have conveniently ignored talking about changing cataloging programs over the years and the effectively lost metadata from the times before the tools write the metadata back into the files (or I chose to do that), or before XMP, and so forth. I have the actual pictures, but about the only way I can find them is to remember when I took them and go browsing for the older stuff. That wouldn't work with the volume of pictures I generate now.
Posted by: Tommy Williams | January 31, 2008 at 12:54 PM
I still like using DVDs as a backup, because I like having more than one kind of medium that the data lives on. I don't think the disks deteriorate, it's more that the writing mechanism is so finicky that it's hard to transition through the hardware upgrades. It's a lot more convenient too to find a set of files on a disk than it is to fire up a backup hard drive.
Posted by: Doug Plummer | January 31, 2008 at 01:40 PM
I actually found that some of my earliest CDs did degrade, or at least things changed from year to year: it wasn't that the whole disk couldn't be read, but that individual files couldn't be read at all had corrupt bits in them. And I saw this on CDs that were fine one year and bad the next, even using the same optical drive that had successfully read them before.
I have had more problems with DVDs and I do attribute that to the finicky nature of writing to them. I never write at faster than 4X anymore and that definitely helped.
Posted by: Tommy Williams | January 31, 2008 at 03:47 PM