The bug:
Starting this morning, we could not power on or VMotion any of our Virtual Machines. The VI Client threw the error “A general system error occurred: Internal Error”.
Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:
Code:
Aug 12 10:40:10.792: vmx| http://msg.License.product.expired This product has expired.
Aug 12 10:40:10.792: vmx| Be sure that your host machine’s date and time are set correctly.
Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: “http://www.vmware.com/info?id=4″. A call to tech support confirmed this as a known problem with a temporary workaround.
The work-around:
Turn off NTP (if you’re using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s “08/10/2008″ at the Service Console command line on the ESX hosts.
As soon as the date was reset to the 10th - problem solved.
So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.