HP ML350 RAID failure 3 drives out of 5 failed

[hallb15]

To cut a long story short, ML350 G5 server was working fine before the half term break. Powered down properly last Saturday due to electrical work taking place in school. When powered back on, failed to boot as 3 out of 5 drives in the RAID are showing RED LEDs.

Seems too much of a coincidence that drives 1, 2 and 3 could fail at the same time.

 

In the RAID option ROM Configuration utility, only 2 drives are showing as failed (1 and 3).

 

I've got replacement drives (thanks to super efficient delivery via @VeryPC) but my concern is that it may be the RAID controller that's dodgy.

 

Any thoughts?

[difinity]

Sounds possibly like a faulty startup capacitor on the RAID card.

[hallb15]

Having done some more testing it turns out that the drive in bay 2 hasn't failed completely, but has been flagged as due to fail imminently. I've replaced drives 1 and 3 and these are being recognised as new and working so I'm holding out hope that the RAID controller is OK.

So I now have to decide whether to press F2 to attempt to rebuild from the existing working drives onto the 2 failed ones, or just cut my losses and replace the imminently failing drive 2 and build a new array from scratch.

[hallb15]

Well its still attempting recovery of the array - 15 hours later! How long should I leave it?

I'm beginning to panic a bit as I've realised there is some data on there that wasn't backed up - I know, I know, lesson learnt!

Anyone tried a SAS to SATA cable with any positive results?

[Geoff]

The format on the raid drive is proprietary to the raid controller (even down to firmware revisions) you will not be able to read the disks other than attached to an identical raid controller. You have a high chance of losing the whole array during the resilver operation, so just hope and pray it survives without another disk failure.

 

As a general rule of thumb, I am not using hardware raid anywhere. In a server such as this I would use the OS software raid, then at least I could move the disks to other hardware and still read them. My bigger setups either involve a NAS running it's own software based raid (ZFS on FreeNAS for example) or a SAN.

[hallb15]

I was in the process of moving everything off this server anyway as we have pretty much completed our virtualisation process. Just hadn't found a suitable time to finish the job. You know how it is when everything is working OK and you don't want to rock the boat, especially with Ofsted imminent. I can find most things I need of the backups, just some aren't as up to date as they could be.

Just wondering how long I let it rebuild itself before I accept its not going to work...

[edutech4schools]

It can take a very very very long time to re build but thinking back I am sure I could view a % completed or something.

[Geoff]

Yeah, the raid tools should show you.

[hallb15]

Thanks for the replies - we're going down the data recovery route as it turns out there was more on the array than originally first thought...

[Geoff]

Repeat after me: "RAID is not a backup solution".

[hallb15]

And neither is Windows Server Backup to NAS it would seem...

[Geoff]

Well no you only end up with a single copy. I get round the issue by using FreeNAS as my NAS and utilising it's snapshotting feature to snapshot the windows backup.

[3s-gtech]

And neither is Windows Server Backup to NAS it would seem...

 

If you have iSCSI capability on the NAS, you can set up a LUN on the NAS which Windows sees as a disk. Then you can use Windows Server Backup to give you multiple backups, overwriting older ones etc to this dedicated backup disk. Massively more useful and flexible, but you need the NAS capability (which most will have now).

[win]

And neither is Windows Server Backup to NAS it would seem...

 

I use this as an additional backup on the side for whole VMs. Haven't used it to recover yet (touch wood) but the quote above doesn't give me much confidence!

[hallb15]

I use Altaro for backing up VMs and this appears to work well.

For me, the problem was that our old server was being backed up with Windows Server Backup as a temporary measure until all roles were transfered to the new virtual environment. It was reporting the backups as running successfully, however it was only when I tried to recover a file that it decided not to play and that it has been telling lies all this time!