Failed RAID

[klop]

Hi all, not having the best start to the day!

 

We had a failed disk on a RAID 5 setup on a Dell PowerEdge 2900III

 

Replaced the Disk last night and all looked good, monitored in Dell OpenManage Administrator and the RAID / Disk showed to be Rebuilding.

 

However I have woken up this morning to find the new disk is Offline and a second disk has failed.

 

I've had a string of emails from Sophos complaining it cannot read certain files - otherwise all seems ok - accessing file wise.

 

I know I need to replace the failed Disk asap but not sure what to do about the Offline new replacement Disk - do I wait until the failed disk is replaced too?

In Dell OpenManage Administrator I can click to make the Offline disk Online, however I have this warning;

 

Warning: Making a physical disk online does not verify the virtual disk's parity or data. If data was written to the virtual disk while the physical disk was failed, then making the failed physical disk online may result in data loss. It is strongly suggested that you either allow an automatic rebuild of the physical disk to complete or initiate a check consistency of the virtual disk after the Online task completes. Are you sure you want to make this physical disk online?

 

Will I lose data - I'm a little nervous to try it!

 

Some screenshots from Dell OpenManage Administrator;

 

 

We do have a backup of all data that may have been lost but the thought of needing to recover using that doesn't appeal - but it's good to know it's there!

 

Thanks, Ben

[klop]

Ok, it turns out there are problems with files being accessed - oh bugger!!

 

Thank God we backup!!

[computer_expert]

If you lose two disks in the same RAID5 array, you'll be restoring from backup as RAID5 can tolerate a maximum of 1 failed disk.

 

edit - revised to say that if you lose 2 disks from the same raid5 array, you'll be restoring from backup.

Edited November 10, 2016 by computer_expert

[klop]

Yes thought so - was hoping the replacement one might have rebuilt in time - but seeing as it is offline I'm assuming not?

 

So not sure whether or not to switch it online - which it appears I can at least try to do but am unsure

[pcstru]

It may be worth saying two things that are not directly helpful :

 

1. Avoid RAID 5 with large drives (or just avoid it).

2. In the event of a failure of a disc in a RAID 5 array, before sticking in a new disc and initiating the rebuild, backup the data (if you can). There is an implication here that you should never allocate a hot spare in a RAID5 array.

[SHimmer45]

to me that looks like you have a failed drive in 2 different arrays, which make up you Virtual disk 1 & 2.

if a disk has failed and you pull the failed one and replace it it *should* start an auto rebuild. pulling the correct disks is key (if you have hot swap caddies you should have solid amber lights iirc)

 

on the server can you still see the files?

 

sophos / file access might be because its slow to repsond as the array (if it is still workings is in degraded mode and will be much slower)

[Joanne]

Take a deep breath and make the replaced drive online and see what happens.

 

Then if all is well, replace the other failed drive.

 

I had a dicey moment with a Dell where I had one completely failed and one about to fail. Luckily Dell sent me replacement drives in time!!I was bricking it that the rebuild would cause the failing one to fail, but it held out in the end!

[TwistedHelixis]

I had a dicey moment with a Dell where I had one completely failed and one about to fail. Luckily Dell sent me replacement drives in time!!I was bricking it that the rebuild would cause the failing one to fail, but it held out in the end!

 

I had the same and the other drive died about an hour after the rebuild had completed. phew

 

All my servers are Raid 10 now, even the small primary schools ones.

[Joanne]

@klop how are you getting on?

[klop]

Ok, so having re-read the warning I took the plunge and made the new drive online - amazingly luck was on our side and it worked.

Files were missing beforehand - I thought having lost 2 disks from a RADI5 all access would be gone, but some were still available.

The new drive is now online and I only have one fault - one failed disk - so the RAID is fully operational again.

 

New disk ordered!

 

We backup every night so if we had lost it all we could have recovered. I think the most annoying thing would have been the repeated explanations as to what had happened and why staff couldn't get their work. Email was sent this morning - but still people knocking on my door - even though they'll not experience any problems now we're down to one failed disk.

 

There are two virtual disks on one array - which may be confusing - confused me at 6am this morning for a moment!

 

Pleased to say new server is RAID 10

 

Thanks for everyones help :-)

 

now nervously awaiting delivery!

Edited November 10, 2016 by klop

[Joanne]

Good news! Sometimes you have to be brave!

[klop]

Yes, was feeling tired and stressed though so I know it's not a good time to take decisions that require bravery - but having re-read the warning there seemed nothing to lose really - the data on the disk wasn't currently accessible so if it was lost - no change!

 

Good fortune was on my side though - ignoring the fact that it happened in the first place!!

[Willott]

If your raid controller allows it, and you have space in your case, may be worth grabbing an additional disk and doing a raid level migration to RAID6 (not all controllers will support this), should then make a single disk failure a non-issue.

[klop]

If your raid controller allows it, and you have space in your case, may be worth grabbing an additional disk and doing a raid level migration to RAID6 (not all controllers will support this), should then make a single disk failure a non-issue.

 

Thanks, I would - or at the very least I would buy a spare hard drive to keep in readiness but this server is being replaced asap and my current budget is so small I can't spend on things until I need them.

New server should be up and running soon, RAID 10 so I'll sleep a lot better :-)