Big VMWare crash - Sorted - What can I do now to stop it happening again ?

[mikkydoos]

Hey all,

 

Last weekend my VMWare host crashed out big time - whole network was out on Monday morning - DC's, print servers, everything.

 

Could have been something wrong with the power in the room.

Nothing in the UPS logs though!

 

In addition to that, Smoothwall was hung, CCTV and a couple of switches hung. RAID needed rebuilding on a backup NAS.

I have no idea what happened and logs don't point me in any specific direction!

 

Nothing consistent except I suspect power problems.

 

A full reboot sorted it, and a reboot/restore of all the guest machines.

 

 

What I'm getting at is where my VMWare host is concerned is it possible to configure a 2nd host to 'hot' migrate.

 

e.g. if Host 1 goes down, guests are in a 'hot' up to date state to take over ?

 

I know this won't help some of the issues above but I'm planning on implementing a 2nd host at Easter and would like to do this rather than load balance.

 

Is it possible ? We're running Esxi 5.5 Essentials plus (will be upgrading to v6 at Easter) and also using Veeam.

 

Will vMotion do this for me ?

 

 

 

Cheers in advance

[twin--turbo]

you can do it two ways .

 

HA - will start failed guests on the second machine. takes a boot time.

Failover - keeps a stand by copy up to date which is instant swithcover. ( does not even drop a ping usualy )

 

TT

[mikkydoos]

In this case it looks like it was the host that failed.

 

Whats HA ?

 

I presume that's with vMotion ?

[Geoff]

VMWares page on it covers both options. I think you get HA in standard and you need Enterprise for Failover (I have SRM so I'm not sure).

 

VMware vSphere with Operations Management: High Availability | VMware United Kingdom

[pantscat]

If you're running essentials plus, then HA will do the job nicely.

 

You'll need a vcentre server, an additional host and shared storage that both hosts can access.

 

Let me know if you need any pointers... I'm a VCP.

[mikkydoos]

Cheers @Geoff and @pantscat

 

VM products are so confusing

[mikkydoos]

If you're running essentials plus, then HA will do the job nicely.

 

You'll need a vcentre server, an additional host and shared storage that both hosts can access.

 

Let me know if you need any pointers... I'm a VCP.

 

 

So I cant do this with a replicated datastore on the second host?

 

I'm using the hosts storage rather than e.g. a SAN at present.

[pantscat]

No, it really needs to be shared storage - it could be a NAS or a SAN.

[mikkydoos]

No, it really needs to be shared storage - it could be a NAS or a SAN.

 

Hhhm. Not happy with the whole network serving from a NAS.

 

Rethink required....

 

 

Cheers @pantscat

[Geoff]

So I cant do this with a replicated datastore on the second host?

 

I'm using the hosts storage rather than e.g. a SAN at present.

 

 

You should look at VSAN if you are using host storage.

 

Virtual SAN Software-Defined Shared Storage: VMware | VMware United Kingdom

[mikkydoos]

You should look at VSAN if you are using host storage.

 

Virtual SAN Software-Defined Shared Storage: VMware | VMware United Kingdom

 

But if my host goes down my datastore is unavailable

[PotNoodleTech]

I think I am right in saying you can't do what you want to do with the hardware you currently have.

 

For HA or Failover you need shared storage (DAS/NAS/SAN) and for VSAN you need 3 VMware hosts minimum and it is excruciatingly expensive to licence.

 

So to get this functionality with VMware you're going to have to spends the dollars.

 

Or you could consider moving to Hyper-V in which case you can use Replica to keep a spare copy of the VMs on a separate host which can be spun up manually if anything catastrophic happens to the main server. If you already have server on EES licencing that would be a no brainer (apart from the time taken to migrate everything)

[pantscat]

Or you could consider moving to Hyper-V in which case you can use Replica to keep a spare copy of the VMs on a separate host which can be spun up manually if anything catastrophic happens to the main server. If you already have server on EES licencing that would be a no brainer (apart from the time taken to migrate everything)

 

Yep, that's an idea.

 

Alternatively you could stick with VMware if you're already familiar and happy with it, and just use Veeam to replicate to another host, which you can then fire up manually.

 

VSAN is a nice idea... in practice though it's expensive and probably not a great deal cheaper than a dedicated SAN depending upon your workload.

[mikkydoos]

My set up is this thus far...

 

1 x Esxi host - (x) VM's - VM'd vSphere Web Client

 

1 x Physical 2nd DC

1 x Physical SQL server for MIS - going virtual at Easter - This machine will be Host 2

1 x Physical 12TB file server - standard tower full of raided disks

1 x QNAP 16TB NAS - for all backups

1 x 1TB Buffalo NAS for 2nd backups of critical data

1 x Smoothwall

 

What I'm aiming for is zero downtime if either host fails. - I understand shared storage is needed for hot redundancy- not happy with NAS 1 holding all the servers though - not reliable enough.

 

What other options do I have with what I've got?

Edited March 1, 2016 by mikkydoos

[pantscat]

Even with HA you won't get zero downtime - there'll still be a delay while the VMs are booted up on the 'spare' host.

 

With what you've got currently, you have the following options:

 

1. Go to Hyper-V and use its built-in features to replicate the VMs to a second host.

 

2. Stay with VMware and use Veeam to replicate the VMs to a second host.

 

3. Use one of the physical servers and turn it in to a SAN - Perhaps use 2012 R2 server and windows storage spaces to present the storage to VMware via iscsi.

[mikkydoos]

3. Use one of the physical servers and turn it in to a SAN - Perhaps use 2012 R2 server and windows storage spaces to present the storage to VMware via iscsi.

 

That's a very very good idea !

 

Not sure I'd be happy using the current file server but its more reliable than my NAS.

Or a cheap Fujitsu server with a couple of TB would do it.

 

Still though, if any 1 physical machine that holds data fails, the servers and data are still unavailable. Thats a pain of centralised data.

 

Its a tricky one without spending many thousands ££££

Edited March 1, 2016 by mikkydoos

[Geoff]

I was unaware of the limitations and cost of VSAN so thanks for point that out @AButters

[pantscat]

It is tricky... but it's a case of weighing up likelihood of failure, and what the most important thing is to you/your school.

 

SANs will generally have built-in redundancy - network, power, controllers... which mitigates some of the risk. Using a server as a SAN doesn't have quite the same level of redundancy... but a decent server should be pretty reliable.

 

Don't forget that with Veeam you can use instant restore which will start up the VMs from their last good backup directly from the backup device.

[mikkydoos]

True. And when you have a VM storing data (like my SQL server) it won't get lost when its migrated back to the host.

[Asad_Symitry]

Have you had a virtualisation expert take a proper look at it. It's funny how many times an issue can be fixed just by changing the configuration.

[jtoland]

We have 2 esxi host and we clone vm backups from 1 server to another every week and we have a nas that backs every thing up daily we do this incase of some kind of failure so when one dies we just boot the other up

[bart21]

Sounds like you want fault tolerance (ft) rather than ha.

 

Explained here

http://searchvmware.techtarget.com/guides/Know-your-uptime-Differences-between-VMware-FT-and-HA

 

Still requires shared storage though.

 

Nick

Vcp vcap

[mikkydoos]

Have you had a virtualisation expert take a proper look at it. It's funny how many times an issue can be fixed just by changing the configuration.

 

No, but that is an option. TBH I'd rather put it together myself but advice is always welcome.