RAID issues

[synaesthesia]

Experienced a drive failure in one of our servers last week however I'm rather disappointed in the resiliency side of things. Wonder if there's something I'm doing wrong - only thing I can think of is that I should try some different "drivers" for ESXi.

 

Single Intel server with an Adaptec 6805 8 port SAS/SATA3 RAID controller with 2 drive arrays - a 2 drive RAID1 and a 4 drive RAID10.

The Raid1 is home to the first domain controller, the Raid10 is home to the first file server. The drive that failed was in the Raid10. This shouldn't be an issue, it's a single failure and would just be degraded until the drive was replaced. Whipped the drive out so I know which one it is, drives on order for replacement on Wednesday however the VM guest only lasted a day before it just died a death. The datastore is "live" albeit extremely slow, and the guest is inaccessible - not even pinging.

 

This will be resolved on Wednesday however I will have this nagging doubt in my mind should another drive fail. I don't want to interrupt my holiday time off again with things that should take care of themselves or at least tick over until we can resolve them! Any ideas on how I can resolve this permanently?

 

(For reference, the failing drives are Seagate. Never, ever again.)

Edited August 24, 2014 by synaesthesia

[SYNACK]

Seagate has gone down hill and are now rubbish, I have never found an Adaptec RAID I am happy with, HP rebrands and LSI rebrands in IBM seem to do an all right job but have never had good luck with adaptec. Perhaps a different and beefier RAID controller with more memory, RAID 10 should not slow down that much with a single dropped drive. Had RAID 5 sets fail one drive and keep going at almost full speed with the HP Smart Array stuff.

[ericdano]

You should always have a drive available to pop in a RAID in case of drive failure. Or you could move everything to a RAID 6 configuration, where you could survive TWO drive failure. I personally don't see much of a point in doing RAID10. I'd rather do RAID5 with a hot spare or a RAID6 configuration.

[DMcCoy]

I'm going to make a guess. SATA drives? Do you still get write caching with a failed array?

[synaesthesia]

Yeah they're SATA drives - a server built on rather a tight budget. And yes, write caching with a failed array apparently.

A little research shows this could be an ESX issue, there's problems relating to this sort of thing since 5.1 - will need to do some more digging, but not until start of term!

[synaesthesia]

Secondarily, @ericdano we don't need lectures that are not relevant to the problem in hand thanks! We should have a spare, but personally I'm glad we didn't. A spare would have meant putting in another Seafail drive. We do keep spares for all arrays, just luck of the draw we didn't for this one. I would not use a RAID6 on a VM host, the write speed drop is far too harsh.

[SYNACK]

Just to be a pedant

 

I would not use a RAID6 on a VM host, the write speed drop is far too harsh.

 

on your available hardware.

 

Again agree with the Seagate assessment, even their Enterprise RAID sata drives are shocking, had three fail with weird SMART errors inside a year and a half.

[synaesthesia]

Indeed, these were enterprise drives. Annoying really. Seatools doesn't even find the drive defective yet every other tool available says it's toast.

[PotNoodleTech]

This is where a hotspare is worth it's weight in gold (literally!).

 

IMHO you'd be better off flattening that server (at some point) and losing the RAID1 array as it's just a waste of drives. Use 5 drives in Raid6 and keep the 6th one online set as a hotspare.

[synaesthesia]

Not on SATA drives I'm not! I would prefer to keep the DC as physically separate from the rest of it as possible hence the separate array. As said re the RAID10 setup, we have a very limited budget and the setup is built around it, so bearing in mind the lower end controller and drive combination we need to keep performance up as much as possible. Not fussed about write performance on a DC but the file server concerns me greatly - that's home drives and shared areas. Maybe in a few years when those shared areas are done away with, but not yet!

[pcstru]

I'd rather do RAID5 with a hot spare

It has been discussed here a number of times but it is probably worth repeating. RAID5 should never be provisioned with a hot spare. A failure of a drive on a RAID5 should trigger copy of the data and only then should you attempt to put in a new drive and rebuild the array. RAID5 is really best avoided in any kind of production environment.

 

(Apologies for the slight hijack, we now return you to your normal viewing).

[synaesthesia]

Funnily enough, I might not really haveany other option but to accept the performance hit. It won't take up the slack of the failed drive on the hotspare >:| Currently tryign to force it online to recover it.

[PotNoodleTech]

To be honest, once your virtualised, keeping resources physically separate is kind of defeating the point. I would suggest it is much better to have one pool of more reliable storage complete with hotspare, rather than multiple pools with less resiliency and no hot spares.

 

Either way - it is a bit worrying that the guest has become inaccessible, that shouldn't happen (hasn't happened when I've had degraded arrays). I could understand it may happen during the rebuild process as most low end raid cards can't cope with rebuilding an array whilst serving data off it, but it shouldn't happen during the initially degraded stage. it may be highlighting a further problem with other drive(s).

 

Hate to be the one to ask this - but do you have backups?

[synaesthesia]

Yeah, backups are good. I'm half way through recovering from the array as a "live" backup recovery, if that fails I'm not too worried (other than losing my holiday to resolve this!) as I can recover from those. Waiting on Parcelfarce to deliver the replacement drive (plus extra spare) that they should have delivered yesterday (GRR!) then it looks like I'll be flattening the arrays and starting from scratch. Performance copying from the drives whilst it's "rebuilding" is oddly fine, how or why it impacted on the running of the server does indeed concern me, especially on the other array (which is what convinced me that it's probably best to do as you suggested as we then gain absolutely nothing keeping them separate).

 

It would be so nice to even believe we'd be listened to if we said we needed a SAN - a couple of years ago yes, but I really don't believe *anyone* in schools should be doing that currently, not with the way things are going. Hence I'm not worrying too much, get this back up and running and it'll tide us over nicely until we basically have machines in school that are only a physical gateway into servers hosted elsewhere!

[PotNoodleTech]

Yeh making do and mending here too. I'm looking at VMware VSAN next year for our server refresh. No san needed, yet all of the benefits of a san (and less drawbacks).

[ericdano]

Don't lecture him on RAID. He's obviously an expert......NOT.

 

 

This is where a hotspare is worth it's weight in gold (literally!).

 

IMHO you'd be better off flattening that server (at some point) and losing the RAID1 array as it's just a waste of drives. Use 5 drives in Raid6 and keep the 6th one online set as a hotspare.

[synaesthesia]

@ericdano - please grow up, post reported, noone is having a pop at anyone on here?

[PotNoodleTech]

Hmmph

 

How did it go? Got new HDD in yet?

[synaesthesia]

Not good - one of the replacements is dead. Moving VM's around like mad now!

[Domino]

Just an FYI, there are vmware ESXi specific drivers for these it appears: Adaptec - Adaptec - Support

 

Also, I know dell and HP both do customized esxi downloads for their boxes with the relevant drivers installed - we also add the hp repositories to our vmware update manager and remidiate newer driver releases along with the esxi patches.

 

Not particularly helpful now, I know - but might be in future. I've been hit with odd driver issues around the flex-10 modules before...

[synaesthesia]

Yeah, I'm running the latest drivers for this. Spent a bit more time researching, some people have noticed issues with Barracuda drives and those controllers.

After waiting for the delivery of the replacement WD drives and finding one dead, I decided to rid of all the drives entirely and use something we had lying around - some WD Red 4TB drives in RAID6. Not ideal for a server but damn slight better than consumer level, and will certainly last until we can replace all the drives with WD RE drives in October. Decided to test the speed on R6 and it wasn't as bad as expected - and let's face it, reliability really needs to be the name of the game. Backup is still restoring but at least it has all weekend!

[3s-gtech]

I've been impreesed with WD Reds in a Synology NAS, pretty solid. Also have some RE in a server, been ace. Not many Seagates in service now, most failed or retired.

[synaesthesia]

Yeah, we had the Red's in waiting for our second Synology NAS - we have 8 of them in a DS2413+ at the minute and very impressed.

[Arthur]

For reference, the failing drives are Seagate. Never, ever again.

Not many Seagates in service now, most failed or retired.

Out of interest, which models were yours? I haven't had any issues (so far) with the Seagate drives designed specially for NASs.

 

Also have some Re in a server, been ace.

Same here (in addition to an 'Se' too). We purchased the Re models mainly because they designed to handle upto 550 TB/year. The 'Se' can only manage 180 TB/year.

Edited August 29, 2014 by Arthur

[synaesthesia]

No idea on the exact model number, Barracuda ES 2TB is all I can remember.