Recovering from RAID fail

[sparkeh]

Ok so I have a server with RAID 5 and the array won't rebuild (1 dead disk replaced but two others with read errors), HP's solution is to replace the disks but of course this means data loss.

 

The machine is Server 2008r2 running Hyper-V with three VMs.

 

Of course I have backups but I have managed to copy the VHDs over to an external hard drive. So wondering if there is any problem with the following:

 

-Replace disks

-Reinstall Hyper-V

-Copy VHDs back to server, attach and start up again.

 

Thoughts?

[alexsanger]

If the array won't repair/rebuild then whatever you do you'll have to replace the disks. Once you've done that and got the server running it should be fine to do exactly what you said, assuming the VHDs weren't corrupted during the RAID failure and copy process. Either that, or restore from backup. Which ever is quicker/easier/you're happiest doing. I guess thats the beauty of using VMs - it gives you options!

 

Personally, I'd probably get the server running, then do a full restore from backup, unless there's a compelling reason to use the copied VHDs instead (which you could always do after the restore).

[plexer]

Sounds about right

 

Will you continue with it in RAID5 or possibly look at RAID6?

 

Ben

[sparkeh]

Thanks for the replies.

 

@plexer: You know I hadn't thought of that until you mentioned it. Well we currently have 4 disks + 1 spare in RAID 5 Array. So we could use RAID 6... just need to get it straight in my head whether this would be better or not...

[sparkeh]

So we could use RAID 6... just need to get it straight in my head whether this would be better or not...

 

Thinking about it the other errors occurred when trying to rebuild the array using the hot spare. I guess with the double parity of RAID 6 this lessens the chance of a failure during a rebuild.

[plexer]

Yup that would be the most dangerous time for a raid5 array the rebuild because if anything else goes wrong you are screwed.

 

Did you run regular consitency checks on your array?

 

Ben

[sparkeh]

Oh the controller doesn't support RAID 6 anyway.

 

And no on the consistency checks, not for while. Will do after this though!

[Chuckster]

One thing you will find a problem with your VHDs is, is that they will need reactivating again. If you've managed to retain the exact MAC address of each VM and input these into each of them as they were before, then it won't be a problem.

 

I'm assuming you simply copied and pasted the VHDs as opposed to doing something fancy.

[sparkeh]

@chuckster: Thanks! I hadn't thought about that, the server is still running so I can note these down and make sure they are the same.

[zag]

I always keep a spare hyper-v host running with nothing on.

 

When something like this happens I'm hoping its just a case of moving the VHD's over.

 

You might also want to look at single SSD drives in the future. I've had great success in running a few servers using them. No more mechanical failures

[plexer]

Oh the controller doesn't support RAID 6 anyway.

 

And no on the consistency checks, not for while. Will do after this though!

 

What controller is it?

 

Ben

[sparkeh]

What controller is it?

 

Ben

 

HP Smart Array... um can't remember the exact model but its an HP ML350 G5.

Also regarding checks, I regularly check the HP Array Diagnostic Utility which I presumed would flag up the errors... hmm but perhaps the drives started having issues during the rebuild...

[plexer]

E200 controller then by the looks of it.

 

Ben

[sparkeh]

E200 controller then by the looks of it.

 

Ben

 

Yeah just nipped over to the school and it is that one. Won't do RAID 6.

[plexer]

Nope, the P400 with the battery backup module would do though.

 

Ben

[zag]

HP Smart Array... um can't remember the exact model but its an HP ML350 G5.

Also regarding checks, I regularly check the HP Array Diagnostic Utility which I presumed would flag up the errors... hmm but perhaps the drives started having issues during the rebuild...

 

Far more likely its a generic RAID 5 error and the other 2 drives are absolutely fine.

 

I had this happen to me a few times and eventually got rid of Raid 5 on all servers other than my file server.

 

Not had a failure since (remember the fact that you have a 3 times more likely chance of a drive failing when using raid 5 simply because of the amount of drives required).

[sparkeh]

Ok to move this on a bit:

 

After a reboot the HP RAID utility that comes up during boot informed me that the rebuild had failed and asked if I wanted to attempt another rebuild. I said yes and the array rebuilt fine.

The server is still intermittently crashing with disk/RAID errors.

 

So I was wondering if it was a good idea to try replacing the disks with read errors with the new ones from HP one at a time , and attempt to rebuild the array each time?

Or is better to just to swap the disks and recreate the array and reattach the VHDs as originally planned?

[Chuckster]

Are the VHDs mission critical, i.e. they're constantly in use? If not, my personal preference would be to offload the VHDs and rebuild the entire array by replacing all faulty disks, and see how it goes with that.

[sparkeh]

I had a to make a decision so am trying the swap one disk at a time and rebuild option (I have the VHDs stored in case I need a full recreation of array).

 

Interestingly, the original set of disks that came with the server were WD which have all failed with the same kind of error, the replacements HP now ship are Seagate, possibly tells me that they were having issues with these disks?

 

edit: quick Google confirms I am not the only one experiencing this.

Edited January 13, 2011 by sparkeh

[Chuckster]

Don't blame you. The current set of spare SAS drives I have are all Seagate. But the older servers (2-3 years) have WD in them. :|

[SYNACK]

Yea the WD drives from a few years back had a definite lifespan issue, replaced 40% of them inside two years, these were desktop drives though.