Failing server hard drive

[jmak]

In our sole server, we have 4 HDDs in RAID5. The additional complication is that the 4 servers are virtualised on Server 2012R2 Hyper-V (non-GUI). I ran the Dell diagnostics and one of the disks is starting to fail. It is out of warranty. I know what should have been done and told people in plenty of time, but that time is gone....

 

A replacement server is due to be installed on 25th July. I have cloud backups

 

My query is what to do next. I see my options as:

 

1) Pray

2) Take the failing disk out and re-configure the RAID across 3 drives and then restore from backup

3) Buy a replacement HDD and let the RAID rebuild

 

The delay in replacing the server was financial constraints, so spending money will be a hard fight. Any other free options welcome

[Joanne]

if you have sound backups (test them) then let it limp on for another month.

 

A replacement drive wouldn't be too expensive if you are worried.

[bossman]

@jmak

 

Don't trust to luck....buy the drive as soon as possible and get it installed asap....

 

Safe not sorry...

 

Had a SAN a few years back that threw up a degenerating disk I was lucky as I had a replacement...installed it straight away and the RAID5 rebuilt after a few hours...good job as we had another of the drives fail the very next day.....

 

So lucky or good practice.....

 

Don't wait .....do it!!!

[speckytecky]

Replace the failing hard drive for sure I'd say. We had one actually failed over the weekend and when I swapped it out yesterday found the case was hot, hot, hot.

[Steve21]

Would replace it for sure, especially if all the drivers were bought at the same time as logic says (yes I know it's logic not reality) they'll get end of life near the same time and if it does fail it'll only put more pressure on the others

 

Steve

[jmak]

Do you think I could make this a poll and then use it as my business case to put in front of the head [emoji6]

[mukz]

@jmak what server is it? And what is the model of the drive? You can pick up a refurbed drive for next to nothing dependant on server which will ensure that your server will stay alive until the new one comes in

[jmak]

@jmak what server is it? And what is the model of the drive? You can pick up a refurbed drive for next to nothing dependant on server which will ensure that your server will stay alive until the new one comes in

 

It's a Dell T410 - I know they're 500GB Cabled (not hot-swap) SAS drives, but I can't find the original build sheet and I can't see the model details without taking the drives out. Planning to do that tomorrow after school.

[mukz]

Does the server not have omsa installed?not sure if you can install without rebooting

[jmak]

Does the server not have omsa installed?not sure if you can install without rebooting

It's not installed on the host - at the time it didn't seem worth the hassle of working out how to do it on the GUI-less hypervisor. Also the idea was to leave the hypervisor as minimal as possible - maybe this should have been an exception....

 

Installed on the guest only shows the virtual hardware devices...

[3s-gtech]

Remember that with RAID5, it's massively more likely for another drive to pop/hit unrecoverable errors while rebuilding and kill the array. With this timescale, I'd be making sure that my backups are good and holding off on any rebuild. If you can, pull the vhdxs off onto another storage medium, rebuild with RAID 10/storage spaces/something else better than RAID5 (with the new disk) and put it back together - should be faster and more reliable.

[SHimmer45]

if they dont give you the money to replace the drive they are bonkers regardless if a new server is on its way a drive at most will be a couple of hundred and is well spent to avoid down time and the aggro of having to rebuild a setup.

 

install omsa on core/hyper-v server is documented on the Dell site and is pretty straight forward and is soo useful.

 

If a supplier like upgrade options want stupid money for a drive amazon is worth a look for dell branded kit or ebay or just get the Seagate Model number without the dell branding and you should be ok. (OMSA will moan about being a none dell drive)

if your going down the none dell drive route just make sure the controller isnt going to reject a none dell drive (unlikely)

 

If it goes tits up so to speak you will be blamed and the finger pointing will start and they will forget they have restricted you from spending money.

[jmak]

 

If it goes tits up so to speak you will be blamed and the finger pointing will start and they will forget they have restricted you from spending money.

 

I have already made an email trail of both my recommendations from before the time I thought we should have acted (years ago) and of how unhappy I was about not replacing the server before the warranty ran out and how strongly I felt that it was stupid to run without a warranty.

 

I'm now trying to put together in writing what I think should happen given where we already are. My doubt about getting a new drive is the point @3s-gtech made about the risk of the whole array failing during rebuild due to the additional load on drives which are the same age as the one that's failing.

 

I won't have raid 5 again, but at the minute I could afford to lose the failing drive. If another one fails during rebuild, I'm properly stuck.

[chazzy2501]

after anxious moments with RAID 5 on my old array the new SAN was built with RAID 6. I don't care that it's slower, my sphincter thanks me.

[HTCPCP]

I would go for the replacement. If one fails, then the chances are, another one risks failing too, and its then a lot more work for you. If money is a major issue, i'm sure ICT Direct will be able to give you a good price, they are brilliant at stuff like that (and got me out of many holes in the past where out of warranty equipment breaks)

 

Then, once you replace the server, you can then use it as either a test rig, or a failover rig too.

[NetworkServices]

Always replacement. If it doesn't survive until the new server comes it's going to be you that has all of the work to do whilst fielding the inevitable questions of "when is it going to be ready?" and "why was this allowed to happen?".

[jmak]

Does the server not have omsa installed?not sure if you can install without rebooting

 

So, on the positive side, I've learned how to do something new and have OMSA installed and working on the host. The downside is that it isn't seeing the "failing" disk at all! Which now that I think about it makes sense with the message on boot:

 

1 Virutal Drive(s) found on the host adaptor

1 Virtual Drive(s) degraded[/Quote]

 

I'm thinking a new/refurbished disk is the way to go. Is it as simple as: Turn off server (disks are not hot-swap), remove failed disk, insert replacement disk, reboot server, cross fingers and wait?

 

Just sounds too easy

[mukz]

So, on the positive side, I've learned how to do something new and have OMSA installed and working on the host. The downside is that it isn't seeing the "failing" disk at all! Which now that I think about it makes sense with the message on boot:

 

 

 

I'm thinking a new/refurbished disk is the way to go. Is it as simple as: Turn off server (disks are not hot-swap), remove failed disk, insert replacement disk, reboot server, cross fingers and wait?

 

Just sounds too easy

 

Depending on the PERC controller you may need to add the drive back to the array as it would show up as a foreign drive which is done by going to the raid controller on boot.

Do take a FULL BACKUP before incase anything goes @@@@ up

[jmak]

It's all working [emoji41]

 

It all went according to the text book. The disk showed as Ready when the server rebooted and in OMSA I used the task to set it as a global hot spare. It then immediately started rebuilding.

 

I then just crossed my fingers and waited.

[AlanD]

Out of warranty doesn't mean server has to be replaced...does it? I don't replace my car when its out of warranty....

 

although I would pay to replace a component if it failed.....

 

So I'd be buying a hard disk - urgently - next day delivery....to avoid the hassle of any restore from backup procedure. Do a quick google to find out how to take the existing disk off line - and get it to reconstruct the RAID in situ....which should happen without taking the server off line.

[jmak]

Out of warranty doesn't mean server has to be replaced...does it? I don't replace my car when its out of warranty....

 

although I would pay to replace a component if it failed.....

 

So I'd be buying a hard disk - urgently - next day delivery....to avoid the hassle of any restore from backup procedure. Do a quick google to find out how to take the existing disk off line - and get it to reconstruct the RAID in situ....which should happen without taking the server off line.

If it was your only means of transport and you had to use it every day to get drinking water you'd have some kind of backup plan.

 

The issue was that it's the only server in the organisation and there was no warranty and no spares on site. Also the original spec didn't include hot swap drives, so the server had to be taken down to swap the disks.

 

I had advised the relevant people about the risks, but couldn't get any money spent until crisis point was reached.

 

Anyway, all good now. And as I have a new server being installed at the end of July, I'll happily keep running this as a spare without a warranty.

 

I also meant to thank everyone who'd offered advice on my last post - especially @mukz