Shutdown ESXi Server

[Hightower]

I've utilised a script to shutdown my ESXi server when the APCUPSD program tells it the power has gone. I've got it working pretty neatly on a Nagios box (VM on the ESXi server).

 

Basically, the nagios box monitors the UPS using APCUPSD. When the power goes it runs a local shell script that connects and runs a script via SSH to the ESXi box (unsupported mode) - the script it runs is stored on the Datastore of the ESXi box that then proceeds to shutdown all VM's and then finally the ESXi server.

 

The problem I have though, is by running the script via SSH - the script runs in the nagios box (for all it is hosted on the ESXi server) so therefore it might shutdown 3 out of 5 VM's and then the nagios box (as it is a VM). This leaves a VM online as well as the physical ESXi server (as the nagios box controlling the shutdowns has turned off and therefore cannot carry out any more commands).

 

Hope I've explained this clear enough. Anyhow, long story short is, how do I tell ESXi to run a script instead of having the nagios VM run the script that is hosted on the ESXi server?

 

 

ARGHHHHHHHHHH

[Theblacksheep]

I've utilised a script to shutdown my ESXi server when the APCUPSD program tells it the power has gone. I've got it working pretty neatly on a Nagios box (VM on the ESXi server).

 

Basically, the nagios box monitors the UPS using APCUPSD. When the power goes it runs a local shell script that connects and runs a script via SSH to the ESXi box (unsupported mode) - the script it runs is stored on the Datastore of the ESXi box that then proceeds to shutdown all VM's and then finally the ESXi server.

 

The problem I have though, is by running the script via SSH - the script runs in the nagios box (for all it is hosted on the ESXi server) so therefore it might shutdown 3 out of 5 VM's and then the nagios box (as it is a VM). This leaves a VM online as well as the physical ESXi server (as the nagios box controlling the shutdowns has turned off and therefore cannot carry out any more commands).

 

Hope I've explained this clear enough. Anyhow, long story short is, how do I tell ESXi to run a script instead of having the nagios VM run the script that is hosted on the ESXi server?

 

 

ARGHHHHHHHHHH

 

Using the free version of ESXi?

vSphere - ESX 4u1?

 

(you can see where this is going- few features removed inc rcli from ESX 3.5)

Edited February 22, 2010 by Theblacksheep

[glennda]

The way we had it set up was, the UPS sent the shutdown signal to all the VM's including the linux box. once the linux box received the signal it told the esxi box to shut down in like 10 mins (plent of time for vms to shutdown). but when it told the esxi box to shutdown it set itself to shutdown in 7 mins so the linux box was shutdown before the esxi box went.

 

Does that make sense?

 

it does require the APC addon ethernet card.

[Hightower]

The way we had it set up was, the UPS sent the shutdown signal to all the VM's including the linux box. once the linux box received the signal it told the esxi box to shut down in like 10 mins (plent of time for vms to shutdown). but when it told the esxi box to shutdown it set itself to shutdown in 7 mins so the linux box was shutdown before the esxi box went.

 

it does require the APC addon ethernet card.

 

This is how our script works - but in order to shutdown the VM monitoring the UPS it needs to be run from the ESXi host. Our script also checks to make sure all VMs are powered off before turning off the ESXi host, instead of your more basic timer. We want it this way because we are hosting a mail server on one VM which can take some time to shutdown. We need to make sure it's off before the host goes down (not just guessing with a timer) because if it doesn't get shutdown properly it can be a bugger to recover.

 

BTW - we have the network card for the UPS. All thought of when bought

 

In my old vmware system, I wrote a little bash script that ran on the vmware host. It checked for a flag file (which was placed on a machine somewhere over NFS - the detail escapes me) and if it saw the file it would start a shutdown.

 

Hmmmmm, that could be one solution if I can't do it my way.

 

Using the free version of ESXi?

vSphere - ESX 4u1?

 

(you can see where this is going- few features removed inc rcli from ESX 3.5)

 

I don't think this affects it. I know the free version has features removed, but my shutdown script works perfectly when run from the ESXi host. It shuts down all VMs and then finally the host.

 

All I need to do is find a way to:

 

1) UPS loses power. SNMP tells Ubuntu VM that power has gone (WORKING)

2) ESXi host has script to shutdown VMs and host. This works when run physically logged into host via SSH

3) I need the Ubuntu VM to say "power has gone, please run this script"

 

At the minute Ubuntu VM is actually running the script itself, which is not what I want. I need Ubuntu VM to tell ESXi host to run the script.

 

Sorry if this isn't clear - finding it hard to get my thoughts into words.

[glennda]

i no its not ideal but old pc attached to the ups that doesn't matter if it shutdowns via power loss, you could set it to remotely execute the script on the esxi host. not quite the way you want it, but if all else fails!

[Duke]

Anyone else think VMware should have better UPS support built in? It's kind of an important area, and what I'd really like is the ability to Vmotion my VMs to a different server room if the ESX host notices that it's going to lose power from the UPS. The ability to detect UPS status and write action rules accordingly would be very useful. I emailed VMware about this a couple of weeks ago an didn't even get a reply.

[Hightower]

i no its not ideal but old pc attached to the ups that doesn't matter if it shutdowns via power loss, you could set it to remotely execute the script on the esxi host. not quite the way you want it, but if all else fails!

 

That's kind of what I want, but using a VM instead of an actual physical machine. I just need to know how to "SSH into the ESXi and run the script from the ESXi host"

[glennda]

but would the ssh session running the script not fail if the vm has shut down?

[Hightower]

but would the ssh session running the script not fail if the vm has shut down?

 

That's my dilemna

 

I need to get the VM that monitors the UPS status to tell the ESXi host to run the script, rather than having the VM run it via an SSH session.

[glennda]

It wouldn't matter - the ESX has started a shutdown - this can't be stopped. It does a graceful shutdown of all VMWare services first, then shuts down the hosting OS.

 

yes, but if you run a script that shutsdown the vm's including the one that is running the script, while it is waiting to see if all the vm's have shutdown then the esxi host will not shutdown properly

[Hightower]

It wouldn't matter - the ESX has started a shutdown - this can't be stopped. It does a graceful shutdown of all VMWare services first, then shuts down the hosting OS.

 

It does matter. The ESX hasn't initiated a shutdown of itself yet.

 

The script starts a shutdown (via VMWare Tools) of each VM, and each minute checks to see if any are online. Once they have all shutdown the ESXi server then proceeds to shut itself down (still reading from the script here).

 

Now, when the script is run from the ESXi host (which is what I'm trying to achieve) the shutdowns work perfectly.

 

However, when run from a VM on the ESXi host the VM actually gets shutdown, thus stopping the last part of the script from running which shuts down the ESXi host.

[glennda]

It does matter. The ESX hasn't initiated a shutdown of itself yet.

 

The script starts a shutdown (via VMWare Tools) of each VM, and each minute checks to see if any are online. Once they have all shutdown the ESXi server then proceeds to shut itself down (still reading from the script here).

 

Now, when the script is run from the ESXi host (which is what I'm trying to achieve) the shutdowns work perfectly.

 

However, when run from a VM on the ESXi host the VM actually gets shutdown, thus stopping the last part of the script from running which shuts down the ESXi host.

 

maybe just sent up another linux box with no resources but enough to send the shutdown signal and the just let it get powered off. as vmware don't support the ups until that happens we could be coming up with all sorts of querky ideas to try!!

[glennda]

OK, I'm not too familiar with the the free ESXi, but I do know in vsphere you can configure the machines to shutdown their OS when they get a power down signal. Can you do this? If so, all you need to to is execute an init 0 command on the ESX host.

 

problem is the vm will have shut itself down, so the vm cannot send the shutdown signal!

[Hightower]

I think the way to go is a flag file, so:

 

1) VM monitoring UPS gets power loss signal

2) VM creates files called 'shutdown' on NFS share

3) ESXi monitors NFS share for 'shutdown' file and initiates shutdown sequence if it finds it

4) All machines gracefully shutdown, including ESXi host. When VM that monitors shutdown goes down it deletes the 'shutdown' file (as shutdown is already started on all VM's) so that when they get turned back on they don't get shutdown again and again etc.

 

This looks like the way forward. Just got to get ESXi to keep checking for the file....

[Hightower]

How do I go about mounting an NFS share in ESXi?

[Hightower]

Scratch that last one! School boy error doh!

[Hightower]

Ok - could use one last bit of help....

 

I need to get ESXi to monitor /vmfs/volumes/ups-shutdown for a file called shutdown

 

When it sees this file it needs to firstly remove the shutdown file and then call the VM shutdown script.

 

Should this do it?

 

if [ -e /vmfs/volumes/ups-shutdown/shutdown ];
then
   rm /vmfs/volumes/ups-shutdown/shutdown
   "/vmfs/volumes/Local Disk/powerdown-esxi4.sh"
fi

 

I was planning to add this to run from cron every minute. Would this be ok? It needs to be run regurlary but is every minute overkill and will it take much performance away from the server?

[Hightower]

Time to throw a curveball...

 

What brand of UPS are you using? If it's APC, I can wholeheartedly recommend their VMware agent. I sent away for it on CD, and despite the instructions being completely wrong, it works splendidly! No more messing about with windows agents and VM monitoring for me anymore!

 

From what I read online their agent wasn't available yet! How does this work then?

 

Got it all working now anyways, but for curiosities sake....

[Hightower]

Odd. I bought it in December 2008! It works fine (we've had some real power cuts).

 

APC - PowerChute Network Shutdown - Product Information

 

Oh, I forgot, your UPS will need a management card, too. ££££.

 

Glad you got it working, anyway.

 

Ahh I see - it's all money at the end of the day though by buying each agent needed.

 

I have a network management card in the UPS, but used APCUPSD to do our shutdowns. It hasn't cost us a penny (other than hardware) to setup and even though it was a headache in places we're finally there.

 

Everything seems to be working, but will give it a real test tonight after hours.

 

Over the moon I got there with it though with use of a few tweaked scripts and a little hard work

 

 

Thanks for all your help

[Hightower]

I forgot to mention - you need to be able to reboot your signalling machine for maintenance.

 

If ever you need to reboot the machine hosting the NFS (e.g. patch upgrade) the ESX will see it as a signal to shutdown.

 

To get around this, set a flag file on the host to signal that it shouldn't shutdown.

 

Just before maintenance, you could do something like

 

touch /tmp/noshutdown

Add a routine in your script so that if this file exists then it shouldn't shutdown.

 

Delete the flag file after maintenance, of course!

 

I did it the other way so it creates a flag file when the power goes - so this shouldn't be a problem for my individual setup.

 

Can't wait for 3 o'clock to test it properly!

[Duke]

Time to throw a curveball...

 

What brand of UPS are you using? If it's APC, I can wholeheartedly recommend their VMware agent. I sent away for it on CD, and despite the instructions being completely wrong, it works splendidly! No more messing about with windows agents and VM monitoring for me anymore!

 

That's exactly what I meant about better UPS support in VMware - cheers!