RAID5 repair

[dayzd]

Right, I'm in something of a pickle and could do with some help and advice.

 

I have a 4-disk RAID5 at home, which has had some failures... First, a disk failed through too many SMART errors (reallocated sectors), and as far as I can tell HADN'T finished rebuilding before a second drive failed with I/O errors.

 

A week ago, my array looked like this

Disk 1 - 1500GB Good
Disk 2 - 1500GB Good
Disk 3 - 1500GB Good
Disk 4 - 1500GB FAILED - SMART errors

 

After Disk 4 failed, I replaced it with a larger drive and left it to rebuild...

Disk 1 - 1500GB Good
Disk 2 - 1500GB Good
Disk 3 - 1500GB Good
Disk 4 - 2000GB CRITICAL - Rebuilding

 

What I think what happened next was the I/O failure of Disk 3:

Disk 1 - 1500GB Good
Disk 2 - 1500GB Good
Disk 3 - 1500GB FAILED - I/O errors
Disk 4 - 2000GB CRITICAL - Rebuilding

 

Now I know at this point, anyone who understands RAID5 would say I'm up a small stream without an appropriate implement... ...And they'd probably be right, but I'm desperate to fix this!

 

What I've done is tried to clone the original Disk 4 (SMART errors) to a second new 2000GB drive using dd in Ubuntu. It seemed to work, although it went astonishingly fast (around 110MB/s average). Sadly, I'm not sure I did it right, as the RAID controller does not accept this new disk 4 as part of my RAID. Should this even work?

 

Does anyone have any suggestions what else I could try? I'm sure there must be something I can do to rebuild my array, as I still have two original disks, and a third that simply has SMART errors (it's still readable, just not reliable).

 

Cheers,

Dayzd

 

(And please let's not have too many people saying 'You're screwed'. I know I am. This whole thing stings enough as it is...)

[p3dr0]

mine rebuilds itself you dont have to do anything check what make and model of raid controller you have and check the manual.

anyways you will lose 500GB of that last disk you added as raid will see only size of the smallest disk in array

[Butuz]

If you've lost 2 disks then it's game over for your array I am afraid.

 

Only other option maybe to pay for some expensive data recovery company to have a look.

 

Butuz

[3s-gtech]

Could you turn off SMART on that unit, with the old disk in, just until the RAID has rebuilt? Obviously not a great idea, but may allow the rebuild to go ahead at least.

[dayzd]

mine rebuilds itself you dont have to do anything check what make and model of raid controller you have and check the manual.

anyways you will lose 500GB of that last disk you added as raid will see only size of the smallest disk in array

I know I won't gain the full 2TB of space -I understand how RAID5 works, thanks, and that wasn't part of the question. Also, thanks for the 'read the manual' answer. You think I ask strangers to spare their time reading about my problems if I've not even bothered to read the manual?

 

If you've lost 2 disks then it's game over for your array I am afraid.

Thanks.

[dayzd]

Could you turn off SMART on that unit, with the old disk in, just until the RAID has rebuilt? Obviously not a great idea, but may allow the rebuild to go ahead at least.

I've thought about that, but the RAID card I have (Highpoint RocketRaid 1740) doesn't offer the facility to turn off SMART checks. And to be honest, I'm not surprised by that! ;-)

[p3dr0]

You think I ask strangers to spare their time reading about my problems if I've not even bothered to read the manual?

 

yeah most of people do that..

 

and as far as i can see only one disk failed other is rebuilding so you still might have a chance to get it back to work..

[dayzd]

Sorry for being snappy. The prospect of losing 2.5TB (the raid wasn't full...) of data which constitutes pretty much everything I have made/done/downloaded etc for the last 10 years is making me cranky. (Also not having the best day in the workplace so far, either!)

 

The fourth disk requires all of the first three to rebuild fully, but only two are online, hence my problem.

[keithu]

I think your copy of disk 4 needs to be on a disk with the same geometry as the original for the RAID controller to accept it. The only option I can think of is to get another 1.5TB disk (same as original) and use ddrescue to clone the original disk 4 onto it. Pull disk 3 and try to force the array online using disks 1 and 2 and the cloned disk 4 only. There may be some data loss, but it's better than nothing. If this works you can use the new 2TB disk to rebuild the RAID5.

 

ETA: I'm assuming you haven't been using the array since it first failed...

Edited July 14, 2011 by keithu

[p3dr0]

i know that feeling very well :/ but i still hope it will rebuild itself and you will get at least some of your data back!

how about replacing failed one with the one with smart errors and then let it rebuild?

 

gah sorry i thought that disk will work and just spit errors out but it looks like it wont work...

Edited July 14, 2011 by p3dr0

[dayzd]

I think your copy of disk 4 needs to be on a disk with the same geometry as the original for the RAID controller to accept it. The only option I can think of is to get another 1.5TB disk (same as original) and use ddrescue to clone the original disk 4 onto it. Pull disk 3 and try to force the array online using disks 1 and 2 and the cloned disk 4 only. There may be some data loss, but it's better than nothing. If this works you can use the new 2TB disk to rebuild the RAID5.

 

That's my next plan of action. I've been looking into finding another drive of the same type to try that with, but they're not the newest model and are a little harder to find, it seems!

[dayzd]

I think your copy of disk 4 needs to be on a disk with the same geometry as the original for the RAID controller to accept it. The only option I can think of is to get another 1.5TB disk (same as original) and use ddrescue to clone the original disk 4 onto it.

 

Sadly this did not work. Majorly bummed out now... (Not to mention heavily out of pocket!)

 

The RAID controller accepted the new Disk 4 as the fourth disk of a RAID5 array. But a new, separate RAID5.

The controller software set it out like this:

 

RAID ARRAY 1   <-- Original, broken array
Disk 1 - 1500GB Good
Disk 2 - 1500GB Good
Disk 3 - Offline
Disk 4 - Offline

RAID ARRAY 2   <-- New 'phantom' array
Disk 1 - Offline
Disk 2 - Offline
Disk 3 - Offline
Disk 4 - 1500GB Good <-- newly cloned disk

 

Going to look into software raid recovery utilities over the weekend, as the controller/software does not support telling it 'Actually, that disk belongs HERE'...

 

"Help me, Obi-RAID Recovery-Kenobi". You're my only hope!"

Edited July 15, 2011 by dayzd
Additional detail

[smithson83]

Going to look into software raid recovery utilities over the weekend, as the controller/software does not support telling it 'Actually, that disk belongs HERE'...

 

I've had this in my sites for a while but, touchwood, never had the need to use it. Maybe worth a look

ReclaiMe Free RAID Recovery software

Would be interested to know if it works

[dayzd]

Cheers. I'll be trying anything I can get my hands on, so I'll let you know if I have any luck!

[chazzy2501]

Run spinrite not using the raid controller on either disk 3 or 4.

[SYNACK]

Eep,as it had started rebuilting and may have changed a bit between the first and second failure it could make it much more trick. The best advice is to just turn it off and take it to a DR place now.

 

If you want to have a crack at it then get/borrow four more disks, DD all sectors from each source disk (mounted as read-only) to the working disks so that you can play without digging the hole deeper for yourself.

 

At that point your only real course of action is to use software like above that uses software to attpemt to reconstruct the RAID array in software bit by bit which, so long as the rebuild/fix attempts have not hosed it should be able to recover the data from only three of the disks.