250,000 images: a photo studio’s last ten years of work, nearly lost.
I got the panicked phone call at 11am Sunday, my day off, having brunch with my friends and family. “My RAID array is not working!”
At this point I asked if anything has been done, when he got a few sentences in I told him to stop and bring it into the shop as soon as he can. The more you do at home, the more you can damage your data.
The RAID was homemade with two 4TB RAID 5 sets in a generic enclosure with each RAID set going to an eSATA Port Multiplier. The “raid card”, and I use the term loosely, was just an eSATA card that used software drivers to enable the RAID 5 volume. The data was arranged in a RAID 50 configuration. Each RAID 5 set was then striped to create a single large 8TB volume. When I asked where the backup was he said the RAID was the backup. I responded “Ok, that’s good then you still have the primary data on your other drives or computers” A long pause on the phone was followed by: “No, all the data was on the RAID. It backs up to itself!!”
He told me what happened:
1. The RAID was working and in use copying data. It was shared from a PowerMac G5 tower over Ethernet.
2. In the middle of the copy the shared volume disappeared and the RAID was not accessible.
3. They powered the RAID and server off and on.
4. The raid drive came up with the dialog that it was unreadable and needed to be initialized.
5. The raid monitoring webpage indicated a drive failed four months ago! There were no recent failures to the RAID hardware
6. Figuring the volume is just corrupt they used Disk Warrior to rebuild the directory. When it finished five hours later they previewed the volume and all the data was there. A preview of Disk Warrior is really slow, maybe 11 megabytes per second on a good day on a fast system. The G5 they had was really slow.
7. As the system was not in a good location and the copy would take like 3 days they powered off the RAID and server then moved it to a better location so the long copy could be done.
8. When the RAID and server were powered on the RAID volumes were GONE! No errors, no notifications, nothing. All the drives report as being online and working but no data.
9. They then tried Data Rescue, File Salvage, R-Studio and just about every program they could run and got nothing over a week.
10. Then they called and here we are.
I spent a few hours looking at the RAID and determined that the “bad” drive that failed months ago must have had a bad connection. When the RAID was moved, it came online and the software got confused and added the missing drive to the RAID configuration. This caused the loss of all the profiles of the RAID volume. As this was a ten-drive RAID no amount of guessing or prodding could determine how it was configured. All the ports were not labeled and it was impossible to tell if it was a RAID 5 or a RAID 3 pretending to be RAID 5? It was a mess. At this point I left it alone and made a call.
I asked how important the data was and if it were all lost how bad would it be? Shock and desperation were the next emotions that came over the phone.
We do about 90% of the data recoveries at Mike’s, in house. That helps us keep cost down and get your data back to you faster. Sometimes, however, even the best needs help, and when we do, we call the best lab in the business: Drivesavers.
So I called Drivesavers and got a quote. For a ten-drive RAID with what was really an unknown configuration, standard service was quoted at $7,500 to $35,000. That’s right, if they can even get the data it can cost up to $35,000 to get the information back. So you could have your data rebuilt and saved or purchase a new BMW sedan, nicely optioned.
Our customer had no backup. He could have to spend up to $35,000 to fix that mistake. It’s ten years of his photos; he’s a photographer, he needs those photos.
In the end the data was saved. Over thirty hours of work by a three-person team was required to get the data. The final price was $15,000 and all the data was recovered. The data was only 6TB of information and fit on two 4TB $292 LaCie D2 Drives. Everything in life is obvious in retrospect: $600 or $15,000?
Do you have a backup?
So what lessons can we learn from this near tragedy?
First, always have a backup. A backup is an independent copy of your data on a separate drive.
Second, a RAID is not a backup of itself.
Third, if you think you’re in a situation where you may lose data and that data is valuable to you, don’t try anything. If you cannot make a copy of your data, turn off the computer and bring it to us. Running programs like Disk Warrior can further damage your data, not always, but if you try and fix it yourself, you risk making it worse. Even turning on the computer risks damaging the drive further.
Fourth, if you can make a copy of your data do it immediately. Don’t turn off the computer for any reason just make that copy.
Fifth, if your data is valuable to you, don’t put it on a home-grown RAID that you made because it was cheaper. Homemade RAIDs are not cheaper if they fail more often and cost you a data recovery.
I hope all of you can benefit from this experience without having to pay $15,000.
Do you have a backup?