Shutdown ESXi and VMs During a Power Failure

Shutting Down ESXI and VMs During a Power Failure

In a previous article I outlined configuring a UPS on a standalone Ubuntu Server to shut it down in the event of a power failure. Those instructions are fine for a standalone system.
What if we want to shutdown a virtual machine, other VMs running on the same physical system, and ESXi itself?

Since I have converted all my server system to be VMs, running under ESXi, my next step was to figure out how to do just that..

After MANY hours of Googling this topic, and reading about the myriad of ways others have come up with to shutdown VMs and ESXi itself, including writing scripts and installing them in ESXi, I found this article:(http://www.nojokeit.com/2013/02/shut-down-esxi-51-guest-vms-and-host.html) which prescribes how to do this in a simple way, the key ingredient being 2 simple command(s) we need to send to ESXi via SSH to
a) shutdown the VMs gracefully, and
b) shutdown ESXi itself
namely: /sbin/shutdown.sh && /sbin/poweroff

Based on the above, I modified my system(s) according to the following game plan.

- First, I made sure VMWare Tools are installed on all the VMs.
Note: This is fairly easy to do with a Windows VM, while in Linux it requires a few commands executed in a terminal window.

Next we need to configure the VMs to automatically start, and shutdown, along with ESXi:

- In vSphere Client, click on the host name in the left column, Configuration Tab.

- In the section labeled Software, click "Virtual Machine Startup/Shutdown".
- Click Properties in the upper right corner.
- Click each VM name, one at a time (they are set to Manual Startup by default) and
- Click the "Move Up" button to reposition the VM name up into the section labeled "Automatic Startup"
- Make sure the checkbox at the top of the screen is set to "Allow virtual machines to start and stop automatically with the system".
- In my case I set both the Default Startup Delay and Default Shutdown Delay to "0" seconds.
- Click "OK" to close the window.

Enable SSH Server in the ESXI Host:

In order to be able to send our desired commands to ESXi via SSH, the SSH Server component of ESXi needs to be enabled.

- Click "Security Profile", also located under the "Software" section
- Click "Properties" in the upper right corner
- Click "SSH" in the list
- Click the "Options" button, lower right corner of the popup window
- Under Startup Policy click the radio button labeled "Start and stop with host"
- Click the button labeled "Start" (this will start the SH server on ESXi)
- Click the "OK" button, and also on the underlying screen. SSH will now be running on the ESXi host.

Connect the USB port for the UPS to Ubuntu Server

In order for an Ubuntu Server to be able to communicate with the UPS over USB, the USB port needs to attached to the particular VM in vSphere Client.

- Right click the VM name in the left column of vSphere.
- Left click "Edit Settings"
- Click the "Add" button near the top of the window
- Add a USB Controller to the VM
- Add a USB Device to the VM. If the UPS is detected it will appear as a device which can be attached/added to the VM.

Install Plink Utility Program

The above article refers to a little utility called plink, which allows sending a command via SSH, such as from a batch file in this case, from one machine to another. In this situation we want to send the above 2 commands over to ESXi.

Back in Ubuntu Server, I typed in "plink" at a command prompt, and it said that program was not found, although it told me that utility can be installed as part of a package called "putty-tools". The command to install it became:

# apt-get install putty-tools

In order to test plink, and the SSH server we enabled in ESXi above, I issued the following command:

# plink -ssh -2 -pw password root@192.168.0.4 "ls"

where "password" is the root password for my ESXi host, and the ip address is the LAN IP address for the ESXi host.
The first time I ran this command, it told me it didn't recognize the machine, and asked if I wanted to store the key in the local machine's cache. I typed "y". After that, the above command returned a directory of the root folder of my ESXi machine, so that confirmed I was able to send a command via SSH to the ESXi host, and it returned the reponse, which in this case was a simple directory listing ("ls").

The next step was to establish communications between Ubuntu Server (running as a VM) and the UPS. The following are setup instructions we need to perform in vSphere Client, the Windows based application which lets us control ESXi:

The next thing I did was to modify the file called apccontrol, located in the /etc/apcupsd directory.
This batch file is called by the apcupsd software when certain events occur having to do with the UPS, such as if it loses communication with the UPS, or a power failure occurs.

# cd /etc/apcupsd
# nano apccontrol

There are several sections in this file. The main ones in which we are currently interested are "killpower" and "doshutdown":

My killpower section now looks like this:

    killpower)
        echo "Apccontrol doing: ${APCUPSD} --killpower on UPS ${2}" | ${WALL}
        sleep 30
        ${APCUPSD} --killpower
        echo "Apccontrol has done: ${APCUPSD} --killpower on UPS ${2}" | ${WALL}
    ;;

And my doshutdown section looks like this:
    doshutdown)
        echo "UPS ${2} initiated Shutdown Sequence" | ${WALL}
        ${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"
        echo "****** Executing ESXi Shutdown Command ******" | ${WALL}
        plink -ssh -2 -pw password root@192.168.0.4 "/sbin/shutdown.sh && /sbin/poweroff"
    ;;

where password is the root password for my ESXi system.

The Moment of Truth

Now that we have all the above tweaked and configured according to our grand plan, it's time to test things out.

Note: I tested the above on the latest system that I recently constructed, which is my new email/calendar/contacts server. On this ESXi system I am running a copy of Windows Server 2003, and Ubuntu Server 12.04.

I logged into both VM systems with Putty, took a deep breath, and pulled the power plug of the UPS from the power strip.

The UPS started beeping, which was telling me "Hey, the power's out!"
I started seeing messages on the Ubuntu Server Putty window, notifying me about the fact the power was out, and the UPS had switched to using its battery.
After roughly 60 seconds (according to the TIMEOUT parameter I set above), the Ubuntu Server starting shutting down.
In the other Putty window, Windows Server was logging me out, and it began to shutdown.
I was also running vSphere Client at the time, and it began issuing messages to the little status section at the bottom of its window, letting me know the VMs were shutting down.

After a few minutes (Zimbra takes a while to shutdown), the front panel of the UPS went dark, and the computer shut down.

Note: One thing I forgot to mention.. I have the BIOS of all my servers set to automatically start up when power is resumed to the system.

So the final test was to plug the UPS back into the power strip, and made sure the computer started back up, ESXi rebooted, and a few minutes later I logged back into vSphere to make sure the VMs were both back up and running, which they were.

To recap, my writeup might make it sound like this was all easy to figure out, and setup. It was NOT! It literally took me HOURS of Googling, testing different commands/configurations, having the UPS power down (before the VMS powered down, and before ESXi shutdown), powering the system back up, and waiting for ESXi and the VMs to boot so I was able to do another test.

Feel free to send me an email through this web site and let me know if this article, or any of my other Tech Articles, are of use to you.