Failover cluster with one SAN

[zag]

Are you using Hyperv and veeam? I just had a look at it and looks promising, however I'd like first hand opinion about it.

 

Yep Veeam is the greatest thing to happen in backup in a long time. Its a game changer in terms of ease of backup and instant restore.

 

You can restore entire VMs very quickly or individual file restores.

 

We set ours to backup every server, every night for 30 days in a row so we can instantly restore any server, even to another host in about 20mins.

Edited October 22, 2014 by zag

[tmcd35]

I'd ask the question why bother with replication and load balancing?

 

Load balancing - I suppose it's about getting optimal use out of the hardware and decreasing potential bottlenecks for end users. I could run all my VM's on one host maxed out, or I can split them between the three hosts with plenty of failover head room and present the increased bandwidth from the extra processors and NICs to the end user.

 

Replication - Will hopefully allow me to get machines back up if a server fails quicker than it'd take the head to get to the top of the stairs and ask me what's going on. Hyper-V Replica is slightly quicker than veem in that you don't have to restore any backups and a preconfigured VM is already waiting to be manually started. Down time is measured in how long it takes to press the virtual power button. CSV's allow for instant failover - so no pressing of the virtual power button needed, the software will do it for you. The down side of CSV is it doesn't take into account storage failure, which Hyper-V Replica does (you replicate to different storage), so for full failover with CSV you need to look at mirroring SANs.

 

At the end of the day is a Cost vs Risk analysis. How much risk is the school willing to take? How much are they willing to pay to mitigate that risk? For total paranoia go iSCSI mirrored SANs with dual NICs, redundant switching and CSV (or the VMWare equivalent) but don't expect much change out of high five figure sums.

 

 

That said - I do want to re-look at VEEM in my solution, I'm sure it has a part to play in improving up times.

Edited October 22, 2014 by tmcd35

[stevenlong1985]

Hi,

 

We are in the process of fishing out and studying possible virtualization options. We have been given a quote for two new servers and two new SANs. Is it a must to have two SANs? One for each node? I mean, it gives you the ultimate redundancy but the SAN has got redundancy of its own like RAID 6 in two different arrays. Any suggestions? Getting one SAN less will help our budget a LOT. I have attatched a picture of what I have in mind.

 

[ATTACH=CONFIG]27169[/ATTACH]

 

Top switch is for client communication with the nodes, middle one is for heartbeats and the bottom one is for node storage access. The SAN at the bottom will be a CSV.

 

Thanks

 

Pretty much the same setup I have here

[VeryPC_Ed]

There's no necessity to have 2 SAN's, it all comes down to whether you want (or can afford) a full belt and braces solution.

 

We have a wide range of customers who we have provided "active - active" SAN units (full HA with replicated RAM cache etc.), as they required as much performance and uptime as humanly possible and wanted to cover all bases.

 

On the other hand, we also have a lot of users who have single SAN and multiple header nodes (including some users here) as they don't need the additional redundancy as they plan for this in other ways such as using Veeam to replicate to a standby server or just maintain very tight backup schedules.

 

Ultimately, it comes down to things such as your preferred strategy moving forwards (do you need this level of redundancy? - can it be addressed some other way) and your actual requirements such as the required performance (what level of iops do you need to deliver etc.), disaster recovery plan etc.

 

I think with Server 2012 R2, SMB 3.0 is a valid alternative to traditional ISCSI set-ups and depending on how complex a solution you are looking for, it can offer a flexible, low cost alternative.

 

Also, I wouldn't put yourself down about only being 21 and the governors not listening to you, just make a detailed plan and make your case !

 

if you want to discuss anything or want any pricing on any kit, give me a shout (PM).

 

Ed

[localzuk]

You don't need a SAN at all. We didn't buy one, as it makes getting "tiered storage" expensive. We wanted some SSDs and some SAS drives for our storage. The SSDs are for use with high IOPS VMs such as RDS and SQL, the rest we use for low IOPS VMs and general data.

 

We did this by buying a normal HP DL380 Gen8, with lots of drive bays and slapped lots of disks in it. We're running Server 2012 at the moment.

 

We also have a second one set up identically, which has a nightly VM backup to it, and the general data copies via DFS-R. So, if the first were to go down, we could switch the VMs and shares over in a very short period of time.

 

Windows Server 10 will have a built in capability to replicate disks on the fly at block level, so the same functionality as you'd get from a SAN - and when it comes out, we'll be upgrading to it as soon as we can (within reason).

 

A SAN for our use would've cost us, what, £20k+ as it had SSDs involved. Our storage servers cost half of that.

[blu4]

Thanks for all the input- I need to write up all the options and decide with my manager. But so far our budget is very tight and by eliminating the SANS we save a metric ton of money. I will be definately looking at local storage options since they look "cheap" and promising at the same time. Also thanks tmcd35 for the links and tutorials. Localzuk, can you give me a link to a example of a server such as the one you suggested? How many bays would you reccomend? Also dual controller?

[localzuk]

This is the sort of thing - http://www8.hp.com/uk/en/products/proliant-servers/product-detail.html?oid=6458536

[tmcd35]

This is the sort of thing - http://www8.hp.com/uk/en/products/proliant-servers/product-detail.html?oid=6458536

 

Seeing this (IT porn!) made me think of one reason why you might want to have your storage separate from your host servers. Depending on how many and the type of VM's you are planning on running you're likely to need a number of processor cores and RAM for them. (our top spec'd host is 24 cores and 128Gb RAM). It may be easier spec'ing host servers in 1U racks to suit and then have a couple of these monsters for storage, than try to spec out servers like this HP with a lot of cores and a lot of RAM.

[VeryPC_Ed]

Seeing this (IT porn!) made me think of one reason why you might want to have your storage separate from your host servers. Depending on how many and the type of VM's you are planning on running you're likely to need a number of processor cores and RAM for them. (our top spec'd host is 24 cores and 128Gb RAM). It may be easier spec'ing host servers in 1U racks to suit and then have a couple of these monsters for storage, than try to spec out servers like this HP with a lot of cores and a lot of RAM.

 

One thing to mention with regards to HP / Dell is that they have limited options available whereas a lot of the independant guys (like us) have a wider array of lego to play with.

 

72 x 2.5" in 4U - Super Micro Computer, Inc. - Products | Chassis | 4U | SC417E26-R1400LPB

 

36 x 3.5" in 4U - Supermicro | Products | Chassis | 4U | SC847E16-R1K28LPB

 

Both are available in single / dual SAS expander variants.

 

Hows that for some storage porn ?

 

Depending on how dense you want your storage / compute, it's possible to get quite a lot in a small amount of space !

 

Ed

[tmcd35]

72 x 2.5" in 4U - Super Micro Computer, Inc. - Products | Chassis | 4U | SC417E26-R1400LPB

 

Hows that for some storage porn ?

 

Ed

 

I'll be in the gents and may not come out for...awhile

[blu4]

One thing to mention with regards to HP / Dell is that they have limited options available whereas a lot of the independant guys (like us) have a wider array of lego to play with.

 

72 x 2.5" in 4U - Super Micro Computer, Inc. - Products | Chassis | 4U | SC417E26-R1400LPB

 

36 x 3.5" in 4U - Supermicro | Products | Chassis | 4U | SC847E16-R1K28LPB

 

Both are available in single / dual SAS expander variants.

 

Hows that for some storage porn ?

 

Depending on how dense you want your storage / compute, it's possible to get quite a lot in a small amount of space !

 

Ed

 

Get that off here! You'll get banned for this porn! That 72 bay server is waaaay too overkill for our scenario.

[zag]

72 bays haha wow!

 

Our file servers have 4 bays filled with 1tb SSD's and thats too much

[nicholab]

Dell vrtx has the option of 48tb

[localzuk]

One thing to mention with regards to HP / Dell is that they have limited options available whereas a lot of the independant guys (like us) have a wider array of lego to play with.

 

72 x 2.5" in 4U - Super Micro Computer, Inc. - Products | Chassis | 4U | SC417E26-R1400LPB

 

36 x 3.5" in 4U - Supermicro | Products | Chassis | 4U | SC847E16-R1K28LPB

 

Both are available in single / dual SAS expander variants.

 

Hows that for some storage porn ?

 

Depending on how dense you want your storage / compute, it's possible to get quite a lot in a small amount of space !

 

Ed

 

Absolutely. I didn't make the original decision to buy this specific set of servers, but if I had, I'd've gone for a Super Micro/Intel based server instead as I could've probably trimmed a couple of thousand off the price of the storage (using commodity disks rather than HP specific disks).

 

Not to mention, if I were actually doing it all again, I'd probably be going for local storage with Hyper-V replication instead of "centralised storage". In 2 years time when I start looking at replacement servers I will more than likely look at that very option.

Edited October 22, 2014 by localzuk

[blu4]

Absolutely. I didn't make the original decision to buy this specific set of servers, but if I had, I'd've gone for a Super Micro/Intel based server instead as I could've probably trimmed a couple of thousand off the price of the storage (using commodity disks rather than HP specific disks).

 

Not to mention, if I were actually doing it all again, I'd probably be going for local storage with Hyper-V replication instead of "centralised storage". In 2 years time when I start looking at replacement servers I will more than likely look at that very option.

 

I had looked up this solution but I stumbled accross a problem with replication- problem that is devastating. Split brain? Is there a way to prevent it? I'd be going for local storage therefore replication will be in place.

[VeryPC_Ed]

I had looked up this solution but I stumbled accross a problem with replication- problem that is devastating. Split brain? Is there a way to prevent it? I'd be going for local storage therefore replication will be in place.

 

When doing Active-Active, we have used Starwind ISCSI SAN Software, if both servers go offline, it will not allow connections in until you specify which node should be marked as synchronized and then you can set a manual resync.

 

If a node goes off (reboot / BSOD etc.) and the other is online, it will simply do an auto sync.

 

Not sure how other vendors say Open-E handle this but we found Starwind to be the easiest for people to manage with it being windows based.

 

Ed

[VeryPC_Ed]

Absolutely. I didn't make the original decision to buy this specific set of servers, but if I had, I'd've gone for a Super Micro/Intel based server instead as I could've probably trimmed a couple of thousand off the price of the storage (using commodity disks rather than HP specific disks).

 

Not to mention, if I were actually doing it all again, I'd probably be going for local storage with Hyper-V replication instead of "centralised storage". In 2 years time when I start looking at replacement servers I will more than likely look at that very option.

 

Also agree with this regarding drive prices from HP / Dell are just extortion and this is where we win a lot of server deals (storage), as we are massively cheaper on the disks - they would win more if they didn't pwn everyone on the storage prices.

 

I think centralised storage can be a good thing - again dependant on your situation, we're doing an 2 Node HA implementation at the moment where it's active-active as they need the uptime and a unit in each building for DR purposes but it's got the storage local to the compute nodes to save costs and keep performance high - it's all about working with people and implementing what fits best !!

 

Ed

Edited October 22, 2014 by VeryPC_Ed
missing 'one' in everyone

[sonofsanta]

This has been a thoroughly fascinating thread to catch up on.

 

Our servers will get renewed in a couple of years, and I've long been planning to do it differently. I had still planned on getting a SAN and sticking with the CSV HA cluster, as it does work well, but I'm intrigued by much of the talk here.

 

Question for those of you with local, replicated storage: how do you handle failure of a database server? e.g. cashless catering server, finance server, where restoring from a backup even an hour old would cause grief (not to mention domain controllers!). Can replication be set up to be live?

[tmcd35]

Question for those of you with local, replicated storage: how do you handle failure of a database server? e.g. cashless catering server, finance server, where restoring from a backup even an hour old would cause grief (not to mention domain controllers!). Can replication be set up to be live?

 

So far, only tested Hyper-V Replica on 1 unimportant server to see how it functions. Until I did that I was thinking of getting a full blown iSCSI SAN! Replica continually monitors the live running VM and sychronises the changes with another VM host. The synch is not 100% live, more within x number of minutes depending on how much as changed and needs copying, and how many machines are set to replicate.

 

The second host is set up to store the copied VHD in a different location to the first, in my case it will be a second storage server, and has a replica VM waiting to be switched on. It won't let you switch on the replicated VM unless the primary VM is turned off either by failure or forced failover. This means I can survive a storage server failing or a host sever failing and at a flip of a switch can get everything working again through the replication system.

 

It's not an instant failover solution, from what I can see (unless I missed a setting), you seem to have to manually turn the VM's on after failover. Also, there is the possibility of some data loss if replication hasn't occurred in say 5 minutes or so.

 

Either way it's better than nothing and gives us the level of security, continuity and failover we require. The cost of going iSCSI and CSV seems to prohibitive and frankly a little more complex to set up for what little gains instant failover would give us. Also with a SAN solution it's be more work working out a mirroring system to allow for storage failover. Here I'll have both storage and host failover through a system that is pretty close to identical to what we are running at the moment - straight SMB shares.

[VeryPC_Ed]

This has been a thoroughly fascinating thread to catch up on.

 

Our servers will get renewed in a couple of years, and I've long been planning to do it differently. I had still planned on getting a SAN and sticking with the CSV HA cluster, as it does work well, but I'm intrigued by much of the talk here.

 

Question for those of you with local, replicated storage: how do you handle failure of a database server? e.g. cashless catering server, finance server, where restoring from a backup even an hour old would cause grief (not to mention domain controllers!). Can replication be set up to be live?

 

Well our Starwind HA setup's which are active-active so everything is replicated in real time so yes you can lose a storage node and it should keep on ticking but HA isn't a substitute for protecting against data corruption.

 

Depending on the backend for your workloads, say SQL can be clustered so you could run 2 VM's in a cluster (Guest Cluster) with one on separate nodes and then if one goes down, then the cluster remains up.

 

Domain Controllers should be replicating with other DC's in the domain so losing a single DC shouldn't be the end of the world.

 

As to other workloads, it's a case of looking at what you can do to protect them such as whether they support clustering or backing up SQL/File data as often as is possible/feasible - say Veeam or DPM on a very tight synch schedule.

 

Ed

[zag]

Question for those of you with local, replicated storage: how do you handle failure of a database server? e.g. cashless catering server, finance server, where restoring from a backup even an hour old would cause grief (not to mention domain controllers!). Can replication be set up to be live?

 

I run all those types of servers and 1 day backups are acceptable to our business manager and detailed in our disaster recovery plan.

 

It was only a couple of years ago we backed up servers once a week.

 

You can actually setup Veeam to backup once an hour if you want. Its all about cost to benefit really.

 

I used to argue in the old days that "over backing up" was also a risk to uptime due to the fact your continually thrashing disks. Its less of a problem these days with new storage systems.

[sonofsanta]

Cheers all. Most servers can stand to lose the data - I happily restore VMs from last night's snapshot when something goes horribly wrong - but if, worst case, a storage server went down in the middle of lunch we would lose a portion of sales. I suspect that "five minutes of free meals once a year if you're unlucky" is less of a total cost to business than the differential between local storage and an iSCSI SAN, though

 

@VeryPC_Ed - my concern with DC was a DC coming back up with outdated data and corrupting the AD environment, in much the same way as restoring a DC from a snapshot would do. I suppose if you're manually powering on, though, you can be paying attention to boot into non-auth restore mode - still a bit of a faff, but again, hopefully it's a solution on the better side of the cost-benefit analysis

 

Certainly be interesting to see what Windows 10 Server (Server 2015?) will bring to the table, and what options there will be come 2016 when the refresh is scheduled. We have a dual controller SAN at the moment, and whilst HA and CSV is absolutely bloody magical, dealing with a SAN has its own set of quirks (hardware VSS providers are still a PImyA).

[blu4]

Cheers all. Most servers can stand to lose the data - I happily restore VMs from last night's snapshot when something goes horribly wrong - but if, worst case, a storage server went down in the middle of lunch we would lose a portion of sales. I suspect that "five minutes of free meals once a year if you're unlucky" is less of a total cost to business than the differential between local storage and an iSCSI SAN, though

 

@VeryPC_Ed - my concern with DC was a DC coming back up with outdated data and corrupting the AD environment, in much the same way as restoring a DC from a snapshot would do. I suppose if you're manually powering on, though, you can be paying attention to boot into non-auth restore mode - still a bit of a faff, but again, hopefully it's a solution on the better side of the cost-benefit analysis

 

Certainly be interesting to see what Windows 10 Server (Server 2015?) will bring to the table, and what options there will be come 2016 when the refresh is scheduled. We have a dual controller SAN at the moment, and whilst HA and CSV is absolutely bloody magical, dealing with a SAN has its own set of quirks (hardware VSS providers are still a PImyA).

 

I read about this problem and it kind of put me off doing a local storage because if you have two active DC's in the network you are screwed so much. But as Ed mentioned, there are security measures that can prevent this. I guess the main thing when doing this is to have proper research and testing done. I guess you could set up things such as cashless catering and sims and whatnot on a failover cluster with lets say, a 500GB added storage and have them running constanty, even if a VM goes down so there is no downtime. I might be doing this sort of hybrid for my environment. Seems like a good idea!

[VeryPC_Ed]

Cheers all. Most servers can stand to lose the data - I happily restore VMs from last night's snapshot when something goes horribly wrong - but if, worst case, a storage server went down in the middle of lunch we would lose a portion of sales. I suspect that "five minutes of free meals once a year if you're unlucky" is less of a total cost to business than the differential between local storage and an iSCSI SAN, though

 

@VeryPC_Ed - my concern with DC was a DC coming back up with outdated data and corrupting the AD environment, in much the same way as restoring a DC from a snapshot would do. I suppose if you're manually powering on, though, you can be paying attention to boot into non-auth restore mode - still a bit of a faff, but again, hopefully it's a solution on the better side of the cost-benefit analysis

 

Certainly be interesting to see what Windows 10 Server (Server 2015?) will bring to the table, and what options there will be come 2016 when the refresh is scheduled. We have a dual controller SAN at the moment, and whilst HA and CSV is absolutely bloody magical, dealing with a SAN has its own set of quirks (hardware VSS providers are still a PImyA).

 

No worries

 

If the DC is offline for a period of time and then comes back on-line, the internal counter for AD (Cant remember the name off the top of my head) will have rolled and it shouldn't start bashing your environment as all other DC's should reject sync requests from it to them so it shouldn't corrupt.

 

But yeah - DC snapshots - that's a paddlin'

 

 

Ed

Edited October 23, 2014 by VeryPC_Ed

[tmcd35]

I know Server 2012 is supposed to behave nicely as a DC in a VM environment. Haven't looked at it myself and am a little old school on this matter. I have a physical DC and two virtual. If I thought one of the DC's would be off for long enough to cause replication errors, I'd flatten it and build a new DC. DCPromo doesn't take that long on a new server. Might have to adjust some DNS setting in DHCP but I'd rather do that than deal with an AD with replication issues.

 

As far as DC's are concerned I don't think there's a problem unless all three disappear then I suppose it's a bear metal recovery of the window system back up.