Another sad chapter in the storage woe chronicles

You'd think there'd be an easy solution to this by now. I haven't found it. There must be thousands of us with terabyte-sized storage woes.  The latest effort is turning into another debacle.

I hired CreativeTechs to advise me on a storage solution. I don't have the resources to spring for their first choice (an Apple Xserve RAID ($8 to 12K), so I settled on their pick of a RAID box, a Sans Digital 5 bay. I had the Seattle Mac store install an eSata card when I bought the MacPro, so I was good to go.

Nothing in storage, or anything complex with Macs, is ever good to go. First there was a long, complex configuration of the Sata box, with way deep geek knowledge required. It took many hours for the 5 750gb drives to format into a RAID 5 array, but I was told it would be a simple matter to point the Mac at it when it was done, create a volume, and start shoveling data into it.

Well, two more problems arose. One, the eSata connection didn't work. Two, the Firewire connection did, but the box was only visible as an "Unknown Device." After another $180/hr visit and a long consult with the Sans Digital tech support, we learned that there is an undocumented, but hard 2tb ceiling to the capacity of the box when using Firewire. You don't have that limit with an eSata connection, but the configuration switch to lift that lid is also an undocumented setting.

You can see why this is not ready for civilian use.

Then the eSata problem. Turns out the Mac Store never put in a driver for my Sata card. Also turns out it's a cheap card that may not have the capability to handle a RAID box. We got the card working, we created a volume on the box, and my tech guy left, assuming everything was fine and working. We disregarded the ominous issue of one of the monitors failing to boot up a couple of times. Just a fluke, right?

Tech guy's gone, I start transferring. After 5 minutes of dumping data into the box, it goes offline. Or the MacPro freezes and has to be hot rebooted. The box is unusable. My working theory now is that I need a more robust eSata card that is meant to be used with RAID, and it's on order.

So where are we at with storage? On the low end there is the Drobo, the only user friendly one in terms of set-up, but it only connects with a USB. It's slow to respond, it's slow to transfer data, and it often messes up my boot-up, even on a Mac. I've got more than 2 terabytes in it, and the Max OS yells at me every time I start the computer because of that. It's really annoying. I'm trying to get rid of mine, and I would steer anyone away from getting one.

On he high end there is the Apple Xserve, for the busy advertising photographer with the cash flow to support the overhead. I'm a busy editorial photographer. It's different.

There's nothing I see in the middle. Mine is a three thousand dollar doorstop at the moment, with an expensive meter that runs each time I have someone over to try and get it working. This just has to get easier. What we have now doesn't work.

Comments

[Colin Jago]

I'm not sure whether I should feel lucky or worried...

I bought 2x 2tb Raid 5 firewire 800 arrays from LaCie. I plugged them in (daisy chain - they don't even each get their own socket on the computer) and they initialised and have run fine since.

Maybe I should never disturb this arrangement.

[Mike Peters]

First, I do not work for Drobo. The Drobo issue on boot up is an easy fix. USB will only see volumes up to 2 tb, same as FW. What you have to do is initialize the 2nd lun, whatever that is, see below.

This is how it was explained to me from Drobo...

When a user has an available capacity greater than 2 TB, he has two luns. In windows, a lun is basically the same as a drive letter. So, a user that has, like this user, two 1 TB drives and two 750 GB drives, could have drive letters e and f. On a Mac, the manifestation is a bit more confusing. There are 2 'drobo' disks on the desktop.

The GUI doesn't format the 2nd lun. So, the user will, if they choose to format via the Dashboard, have 1 formatted lun and 1 unformatted lun. The user has to go to the disk utility to format the 2nd lun. Each lun has a maximum capacity of 2 TB.

This user didn't format the 2nd lun. That means that each time it boots, the Operating System notices that there is a drive that is unformatted and notifies the user of this fact. It's not doing any harm by not formatting. It's just some space that can't be used until it is formatted, because each lun has a maximum capacity of 2 TB. If they have enough space, they can't use the additional space until they do the formatting of the 2nd lun.

The minimum configuration for a 2nd lun with our Drobo is four 750 GB drives. That and anything more will kick off a 2nd lun.

The Drobo is a bit slow to respond, as are ALL e sata drives, and the USB runs at about half the speed of FW 400. That being said, I only use it as mass storage and would not ever consider it as a drive to work from, for that I have FW 400 drives, one 80 gig just to use for scratch disk with CS3, and a 250 gig to work from. As I go along saving files, I copy them to the Drobo. Works like a charm. Your RAID experience is what I knew I would be in store for, and what I knew I wouldn't be able to handle without technical assistance. The Drobo is very simple and seems pretty well sorted out also. Their customer assistance is quite helpful if you need it.

[Jason Foong]

How about a Windows Home Server, might those be slightly better than the Drobo? They've got gigabit ethernet and USB 2.0, at least.

Link: http://www.shopping.hp.com/store/product/product_detail/GG796AA%2523ABA?

Not sure how it'll work with Macs, but I gather it should support running them as Network Storage.

[Doug Plummer]

What I don't know is if I can format that 2nd "2tb" drive (it's really just the amount of storage available above the 2tb threshold, in my case Drobo says it's 300mb), without destroying the data on the whole array. So I'm not touching it.

I had a computer that functioned as a gigabit server, but I didn't like that I had to have another computer to turn on when I wanted those files. Also, access got a lot less reliable with the Mac--it dropped the connection whenever I tried to move a lot of files.

My goal is to use the Drobo to store older files--pre 2007, say--that I don't need to access as often as this year's data. Right now I have about 4tb of data from 3 years of digital shooting. This is alarming, as I expect at least another 15 as an active photographer.

[RAK]

I set up a second computer as a linux based server (with 2 sets of mirrored 750gb) drives, which was fairly easy. Then I set up a linux program called rsync to automatically run on my main computer, and once a day it backs up all new photos (and any other watched folders, like iTunes) to the linux box. Getting rsync to work took about an afternoon, most of which was parsing through the geekspeak directions on various linux sites. It works great now, and, although it did take an afternoon to get everything working, it sounds like less trouble than you have gone through.

[Mike Peters]

Yes, you can format the 2nd LUN without any effect on the existing data. Actually you can use all of the data available without formatting, it just doen't show up on your desktop, but it's there.

[Bob Tilden]

Sorry to hear about your chronic storage problems! I'm trying to figure out how to best use a new 2Tb RAID5 system myself...

What do you do for offsite backups?

[Doug Plummer]

When I upgrade hard drives the old ones become storage for offsite. They live across the street at my mother-in-law's house.

[Tom]

I've been looking at the buffalo drive station Quattro 2tb model. It runs about $750. Has four drives. Uses e-sata. Dunno how great or expandable it is but at that price I'm not sure I'd care.

[jjj]

I also recommend the Windows Home Server solution. I just got a HP EX470 and intend filling it with 4x1TB drives and adding 4x 1B ext USBs and it also has an ESATA port, plus Gigabit ethernet.
Vista may have been rubbish as they had the clever guys working on WHS.
WHS works with Macs too. Though as I'm not getting my MacPro until after the current range gets revamped, I can't comment on any issues if any as yet, though it's meant to be handy for Time Machine backing up.

[Doug Plummer]

If you look at my most recent post, you'll see I'm getting close to a resolution. I'm not sure what the usefulness of dedicating a computer to function as a storage cabinet is, unless you're serving a bunch of computers on a network. My understanding of Time Machine is that it's a huge storage hog, but then everyone who knows more than I do is telling me not to upgrade to Leopard yet.