P2000 SAN Degrading

[garethEds]

Hi All,

 

I have a P2000 SAN and drive 1 in the enclosure is showing an orange light. I've checked the management console (never done that before) and it says

 

Health: Fault

Health Reason: Disk failure is imminent

 

It is listed as a VDISK

 

Drive 14 in the enclosure has also now failed and this was a VDISKSP - It was one of three.

 

All of our services are running fine so I assume something is keeping them alive.

 

WHat do I do guys? Do I need to order some new drives asap? and get 1 and 14 replaced? The drives are around 3 years old so are doing well I think

 

Any advice as I didn't sety this up.

 

Gareth

[andydis]

what RAID level is on your VDISK?

is the p2000 covered under support?

is the HDD covered under support?

when is your last successful backup ?

[garethEds]

Hi @andydis

 

Thanks for getting in touch. Here are the answers:

 

Out VDISK is RAID 6

Support - Not sure - how would I know? Where can I find out?

HDD Support - see previous

Backup - All our virtual machines are backed up nightly onto a separate NAS drive (two in fact - at different locations)

 

Gareth

[andydis]

get the serial number of the SAN ASAP and put it in Hp's online warrenty checker ASAP, you may have same day response.

or contact who you purchased it from and ask about the warrenty (above method is quicker)

 

RAID 6 allows 2 drives to fail ( i am assuming the disks are part of a different VDISK and all the HDD's are not setup in one giant VDISK)

 

keep us posted !

[garethEds]

Hi @andydis

 

A quick look on the management and I found this information:

 

So everything looks okay until I get the disks sorted hopefully. It says I still have 2 spares. Would 14 in the enclosure have been a spare? Does VDISKSP mean spare? Do I need to move them around or are they okay where they are?

 

If I order two more disks to replace the broken ones - is it a case of just slotting them in?

 

Trying to find our warranty information.

 

Gareth

[andydis]

Does VDISKSP mean spare?

yep it sure does, how many physical disks do you have in total in the SAN 12?

 

is it a case of just slotting them in?

yep as long as disk1 holds out (it says imminent failure)

[garethEds]

yep it sure does, how many physical disks do you have in total in the SAN 12?

 

14 disks. Picture attached.

 

 

 

yep as long as disk1 holds out (it says imminent failure)

 

It's been like that for a while.

 

Gareth

[psydii]

For what it is worth I had a P2000 go south on me over a weekend. Logged it via HP website, and they called me back instantly, and dug me out of the hole despite not being covered by the warranty.

 

In your case you are very close to a vdisk failure that will result in data loss.

[garethEds]

For what it is worth I had a P2000 go south on me over a weekend. Logged it via HP website, and they called me back instantly, and dug me out of the hole despite not being covered by the warranty.

In your case you are very close to a vdisk failure that will result in data loss.

 

Hi @psydii - I'm slightly confused as to why you say that. I have two spares in there that seem to be holding it together. Everything is running fine - new drives will arrive on Tuesday. Isn't the purpose of the VDISKSP drives to hold it together and kick in when something goes wrong?

 

Gareth

[garethEds]

Right then - two drives on the way to arrive tomorrow. So what do I do?

 

I would assume one drive at a time so as not to overload anything. Being as Drive 14 was a spare, then should I just remove Drive 1 and slot the new one in? Do I need to do anything to it? Prepare it etc etc? Instructions on the web are scarce about this.

 

Once 1 is running again and all is green I'll then remove 14 and put the second new drive in there. Again do I need to configure that as a spare or do anything else?

 

Thanks everyone - your advice here is really appreciated.

 

Gareth

[jcs808]

My P2000 setup is older than yours and is configured for RAID 5 and I have 2 spares. If one fails then I still have a spare while I wait for a new one to be delivered.

 

It's been a while since I replaced a drive but from what I remember, the console will display that there is a new drive once a new one has been fitted. If I remember correctly there is a provisioning wizard that can be run and you basically tell it to use the new drive as a spare.

 

Is the other image posted a screen grab of your enclosure (the one with drives 1 and 14 failed)?

[garethEds]

Hi @jcs808 - Cheers mate. I notice that your spares say GlobalSp whilst mine say VDISKSP - is there a difference? Yes - the picture I put up there was what my enclosure is displaying.

 

On another note - how can you tell which firmware your P2000 is running? I get emails all the time from HP about firmware and P2000 but have always left them as nothing has indicated I need to change firmware and it has been working. It is probably somewhere really obvious.

 

Gareth

[psydii]

https://community.hpe.com/t5/Storage-Area-Networks-SAN-Small/How-to-replace-failed-hard-drive-in-HP-MSA-P2000-G3/td-p/6537822

 

This has a link to videos on how to replace a drive. (I've not been able to view them, as on iOS and the videos are flash)

 

On why I say you are close to a data loss failure: more for posterity - someone in the future following this thread might need a nudge to pick up the phone rather than hang around for hours waiting on a response on the forum, missing a deadline for shipping etc. Two failed drives is not a happy place.

[jcs808]

If you only run one vdisk then it makes no difference whether you have them marked as a global spare or not. If you have multiple vdisks then you can either assign spares to a particular vdisk or set them as global, for all vdisks. I think global is the default.

 

To check your firmware go to the Tools menu and Update Firmware. That will list the versions for all the components of your SAN.

[garethEds]

https://community.hpe.com/t5/Storage-Area-Networks-SAN-Small/How-to-replace-failed-hard-drive-in-HP-MSA-P2000-G3/td-p/6537822

 

Thank you @psydii for the link. Appreciated.

On why I say you are close to a data loss failure: more for posterity - someone in the future following this thread might need a nudge to pick up the phone rather than hang around for hours waiting on a response on the forum, missing a deadline for shipping etc. Two failed drives is not a happy place.

 

I take your point and agree with you.

 

Gareth

[garethEds]

@jcs808 - So I've put the new drive in slot 1 of the enclosure and it said AVAIL. It has now been configured as a Global Spare but I'm not sure how to get it to say VDISK like all the others. Is it part of the VDISK array?

 

When I put the second drive into slot 14 - I assume that is okay as a GLOBALSP even though the others are labeled as VDISKSP. How would I get it labeled as VDISKSP?

 

I feel I'm missing some configuration options. I could run the provisioning wizard but am scared it will re-do the whole array and not just integrate my new disks.

 

A quick look here: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c02254382 tells me that if the blue light is off (it was on) then Offline. The drive has been selected by amanagement application (SMU).

 

Edit: The drive is now sitting there. No lights, but showing up in the console.

 

Edit (15:07): Drive is sometimes blinking green. Enclosure is still showing degraded. I've removed disk 14 as advised by HP so that it can start reporting all things working (but this hasn't happened). Also trying to re-get a warranty. Am going to leave it overnight in case it is doing something. The docs say the SMU will grab the new drive and confogure it accordingly. Fingers crossed.

 

Gareth

Edited December 6, 2016 by garethedmondson

[jcs808]

@jcs808 - So I've put the new drive in slot 1 of the enclosure and it said AVAIL. It has now been configured as a Global Spare but I'm not sure how to get it to say VDISK like all the others. Is it part of the VDISK array?

If you were meaning VDISKSP then leaving it as Global spare is fine. How many disks should be active in your vdisk arrary, and do you have that many in the array?

[garethEds]

If you were meaning VDISKSP then leaving it as Global spare is fine. How many disks should be active in your vdisk arrary, and do you have that many in the array?

 

No - that's not what I meant. Disk 1 was originally part of the vDISK labelling. I've uploaded a picture. You can see that disk 14 has been removed so there are 13 drives in there. Drive 1 was the main failed one and originaly said vDISK (like the one next to it). HP instructions told be to configure this as a global spare and the system would claim it - but it's now not how it was originally.

 

The Enclosure is all showing good again now with no orange or yellow warnings so I assume all is good - so am asking the question for my own knowledge.

 

The extra spare (replacement) is now going in slot 14 as a global spare. yet I cannot see where to set it as a VDisk spare (as opposed to a global spare - which I can do).

 

GJE

Edited December 7, 2016 by garethedmondson

[jcs808]

That's OK. A drive fails in the array so a spare takes it's place. You replace the failed drive and that now becomes the spare which will kick in should another drive fail. When my SAN was installed drives 15 and 16 were spares, now 6 and 11 are the spares.

[garethEds]

@jcs808 - So if every drive fails over the coming years, they could all be called GlobalSP? I now have two Global Spares and two VDISK Spares - how do I make them part of the main array if I start running out of space? I noticed I can stop them being GlobalSP and they simply become 'Available' but how do I add them as drives that are used to store?

 

Gareth

[jcs808]

The spares should automatically become part of the active array when a drive fails, this is when their label changes to VDISK.

[garethEds]

The spares should automatically become part of the active array when a drive fails, this is when their label changes to VDISK.

 

Well that makes sense. Do I need to change my VDISKSP ones to global? Not sure how I change their usage. I shall keep them how they are.

 

Thanks for all your help with this.

 

Gareth