SAN set up

[andyturpie]

Afternoon,

 

We have just not long virtualized our CC4 network and all was fine for a while until a couple of our VM servers decided to BSoD on us and we need to perform a disaster recovery. What had happened was that on our 2nd array of disks, there was a heck of a lot of scrub errors and a degrading HDD on that array. Logging into the SAN web interface, there was no errors that the SAN thought was important enough to highlight in the GUI, but would continue to run corrupting data as it went along.

 

My question is this : Our site now consists of 2x Physical Hosts 1x 16TB SAN 1x SATA 16TB RM Diskbox (first stage backup) and a LTO5 Tandberg tape loader (D2D2T in essence). Is what we experienced quite rare (should I start migrating the VM's back onto that RAID group) or should we look to use further redundancy/resilience to make data recovery quicker and easier?

 

It took our 3rd line around 3 days to get all the VMs and the data back on line, it was squeaky bum time and don't wish it to happen again too quickly!

 

Any suggestions welcome

[Sylv3r]

It sounds exactly the same setup as us and our SAN died the other month after getting corrupt with no visible warnings.

 

Is it an EMC VNXe3100 by any chance?

[andyturpie]

Hi Sylv3r - No we opted for the HP Storageworks P2000 - RM broke the borked RAID array and we replaced a faulty HDD and now the scrub is completing without any errors, the corrupt VM's were rebuilt onto the good array, but I was expecting there to be a bit more resilience behind the SAN & virtual servers. Maybe most schools write to two SANs concurrently and we have factored this in?

[Sylv3r]

Would love to write to two SAN's I just coudldnt justify the costs.

[pcstru]

Maybe most schools write to two SANs concurrently and we have factored this in?

Multi node sans fail too! We've had a Datacore dual node system go down, because a sataboy disc array didn't quite fail a disc - which caused huge I/O problems. Our current Open-E dual node hasn't yet lost any data, but we have run on one node for as long as we have run on both.

 

Ultimately for us I hope (99% assured, 1% hope) even if it does fail, we have a good backup strategy which works. I am starting to think that there must be better ways (in terms of VM hosting + SAN - not backup).

[PotNoodleTech]

As someone who did SANs in 2009 and was dissapointed with speed and reliability we've gone back to local storage here over the last couple of years and am happy again.

 

Who ever invented the SAN is a very clever person - take a £3k server with lots of drive slots, put some custom web interface on it and then charge 10x more than before. People will lap it up!!