Dell PE 2950, PERC 5/i, RAID5, Failed HDD

[Fraser-09]

Hi all,

 

Been a while since I've posted on here (all has been working!)

 

So, I have a Dell PowerEdge 2950 which one of the hard drives has failed. There are 6 SAS disks in a RAID 5 configuration and it's only one disk that's failed. The server is working away quite the thing with only the 5 disks. Tbh, I haven't noticed any preformance decrease since the RAID was degraded. I've got a new disk ready to install but I just wanted to double check how to proceed with it:

 

 

Do I:

 

A) go ahead and swap the failed disk for new disk while the server is powered on and running? Then will the RAID automatically rebuild, or will I need to do this manually kick this off on Dell Open Manage?

 

B) shut the server down, swap the failed disk for new, boot the server and enter Ctrl + R to rebuilt the RAID?

 

 

I've read a few things online but didn't manage to get a definite answer. Read somewhere that said I risk destroying the RAID altogether by turning the server off however read elsewhere that this should always be done offline. (I thought these HDDs were hotswappable?).

 

Anyway, my gut instinct is option A but thought I'd double check as can't be bothered with the hassle of rebuilding the server over Christmas due to making an arse of swapping the disk and completely destroying the RAID. My Mrs already complains that I spend too much time at work and she doesn't get to see me. Blah blah blah. Can't be dealing with the earache especially when Christmas is meant to be about spending time with your family.

 

The RAID controller is a PERC 5/i and the replacement Disk is brand new so should be in the "ready" state.

 

 

Also, to throw a spanner in the works, the RAID controller's battery shows a warning in Open Manage saying it's below the threshold. Been like this for a while and I just never got round to replacing it. From what I understand, the battery is minor and it is only required in the event of a power cut to write the controller's cache to disk. The server has been connected to a UPS all its life so that was another reason replacing the battery was never high in my priority list.

 

I have a new battery now, and I understand this must be changed when the server is off - no big deal but I was thinking to rebuild the RAID 5 first, then replace the battery or would you advise battery first (which would mean powering down the server when it has a degraded RAID)?

 

Look forward to your advice.

 

Thanks,

Fraser.

[Norphy]

You can hot swap the drive and it'll rebuild in the background without any downtime. I have done this numerous times with 2900s and 2950s with various different versions of the PERC controller.

 

It shouldn't make any difference as to whether you replace the battery before or after the rebuild. Strictly speaking, you don't even have to wait until the rebuild has finished before you power off the server.

 

One thing though, make sure that the hard drive is the same model as what's being replaced if that's possible. One time, I got a hard drive from Dell which was advertised as the same size but was something like 4MB smaller than what was being replaced. The array refused to use this disk to rebuild and I had to go back to Dell and shout at them.

[Sagima]

OTOH I have never had any issues with having mixed hdds (diff model or manufacturer) in my dell servers (at one point dell replaced a 10k drive with a 15k one from a different manufacturer and it went along quite happily like that.

[Fraser-09]

Thanks guys.

 

I'll swap the disk out this afternoon and see how it goes. It bought the exact same disk (same part number) so the PERC should accept it without any fuss.

 

Will get the battery in the new year once the array have been rebuilt. Is replacing the battery just as simple as powering down the server, popping the case off, disconnecting the old battery and connecting the new? There's no gotchya's or anything I should be wary about when doing it?

 

Fraser.

Edited December 24, 2015 by Fraser-09

[3s-gtech]

Also, as RAID 5 can be a funny old beast when rebuilding, make sure that your backups are good before you start the rebuild. If all goes pear-shaped at least you won't have lost everything that way.

[Fraser-09]

Also, as RAID 5 can be a funny old beast when rebuilding, make sure that your backups are good before you start the rebuild. If all goes pear-shaped at least you won't have lost everything that way.

 

Awesome!! Just what I want to hear! haha. Thanks for the heads up. Fingers cross all goes ok - Will let you know.

 

In the event it goes belly up, I should be ok. I have about 15 DC's (in multiple sites) so the AD is safe and the majority of the data is replicated to multiple servers too. The morning the disk failed, I spun up a new virtual Server (in the same site) and promoted it to a DC then replicated the all the files over to it too so in the event the server with the degraded RAID fails, I have a second DC pretty much ready to do (I'd just need to setup DHCP, printers and a few other bits and bobs which should get them back up and running. Then I'm free to rebuild the RAID5 and start from scratch!

 

Fraser.

[twin--turbo]

Online Disk rebuilds have been arround since at least the PERC3 shipped in the early 2000's .

 

If the battery fails the controler should switch from Write-Back to Write-Thru mode to prevent the chance of the cache not correctly flushing to disk in a power failure. However if the battery is marginal it may fail during a change. For safety if you can presently switch to Write-Thru or disabled on the cache the do it.

 

Disks need to be the same size or larger for replacement, but additional space is wasted.

 

We have had ~10 Disks fail this year..

 

TT

 

TT

[newpersn]

We got the same server. It gave us the warning for a new battery about a year ago but we had a power cut and the warning was gone. But got a battery off ictdirect and waiting for the right moment to swap it anyway

 

I've been here 5 years and it's gone though 4 750gb 7200rpm in that time.... (Raid 5) and dell have always sent me 1tb drives....

Edited December 24, 2015 by newpersn

[twin--turbo]

If the battery message has gone away without replacing then the Write Cache may be disabled, which will impact performance.

 

TT

[Fraser-09]

Thanks all for the replies and comments. I swapped the disk over this afternoon without a hitch. Old disk out, new disk in and it automatically started to rebuild

 

When the disk failed 10 Days ago, I instantly got an email alert. For some bizarre reason Dell Open Manage does not have email alerts built into the utility, however email alerts can still be achieved using the "execute application" feature. See this link for how to create the email alert scripts. Or, if you are lazy like me, then you can buy them for £3 from this link. If I didn't have this set up then I wouldn't have known the RAID had failed and would probably be working from the degraded RAID for some time until I noticed the server's amber LED on the failed disk so IMO the £3 was worth it!

 

 

Once the RAID has finished rebuilding, I will scheduled some down time to replace the battery. Currently the Write policy is Write through and Read policy is no read ahead.

 

Happy new year y'all

 

Fraser.

Edited January 2, 2016 by Fraser-09

[Fraser-09]

Well that's the RAID 5 finished rebuilding. All seems OK

 

Just the battery left to do now which I'll do one night next week after hours.

 

Thanks again for all your help.

 

Fraser.

[Fraser-09]

Just reporting back with my final update - I replaced the PERC 5/i battery this afternoon. It took about 6 hours or so for the battery to charge before the controller went back to all green ticks.

 

Currently the Write policy is Write through and Read policy is no read ahead.

As soon as the battery was replaced, the Write policy changed from Write Through to Write Back - Even when the battery was still in the 'charging' state!

 

All sorted now. Thanks everyone for their replies and comments.

 

All the best for 2916.

 

Fraser.