ESXi - VMs taking an age to power on/boot after host reboot

[Obzen]

I've been looking after a site for a while now and discovered that the vmware hosts haven't been rebooted in 600ish days. So naturally I decided to give them a reboot today. After doing so vcenter didn't start up automatically, the vms are taking a long time to power on and boot. The vcenter client seems to be very laggy also.

 

So far 2 hours after rebooting I still can't get on any vms yet, is there anything I can do to diagnose this issue?

 

All vms were gracefully shutdown and the hosts placed into maintenance mode before reboot. the SAN was not turned off. I can see the iscsi connections.

 

We are running VSphere Client 6.0.0 (Build 5112508) ESXi 6.0.0 (Build 3380124) on 2 DELL R260s (E5-2640, 128gb ram, OS on 8gb enterprise sdcards)

 

Another thing I would like to add is that if I use the host from the console and use F2 to view logs, keyboard input will stop when i reach the logs item in the F2 menu (on both hosts)

The hosts also took a mighty 48 minutes to reboot, they were booting fine until vflash started to load and from there it goes down hill.

 

Any help is much appreciated.

[Bruce123]

We had almost exactly the same problem a couple a weeks ago, it's uncanny.

 

VMs wouldnt come up, esxi taking ages to load, pressing F2 caused mouse and keyboard to stop working (so could not view the logs), iSCSI connections (multipath) were showing as up, vCentre had trouble connecting to one or two servers in the farm, like it was having trouble connecting to the management interface.

 

But the problem was because one on the two switches the iSCSI traffic traversed, had failed. But it wasn't obvious as all of the indicator LEDs were on, blinking etc. It was only after the Network Manager tried turning off all of the equipment that he noticed this switch didn't come up.

 

Part of reason why we didn't notice an issue with the switch was that they are both on a private network between the ESX hosts and the SAN, so our network monitoring software can't see them.

 

I'd suggest your next step should be to check the status of the switches between the ESX hosts and your SAN, if it's setup that way.

 

If you have HA enabled, I would also consider whether it has attempted to migrate VMs from one host to another, and check that all hosts can access all data stores.

 

I know it's not exactly the same problem, but does have a lot of similarities.

 

Good luck...

 

Thanks,

 

Bruce.

Edited October 27, 2018 by Bruce123

[kmount]

Slow boot is common if the hosts cannot talk to DNS or storage.

 

Check if they were configured to talk to DNS servers they host (so slower boot and timeout) - wouldn't explain sluggishness once those dns servers were up though.

 

More commonly though is trouble accessing network storage like iscsi.

 

I'd forget vcentre for now it's a victim too, look on the actual hosts for the storage pathing. If you're using iscsi and jumbo frames make sure they're still reaching with vmkping, and work your way up from there.

 

Both hosts having the problem suggests something shared, switching or storage maybe.

 

Log into the host client on each server and ssh/console and test the nics.

 

Have a look in the vmkernel log too.

[Obzen]

We had almost exactly the same problem a couple a weeks ago, it's uncanny.

 

VMs wouldnt come up, esxi taking ages to load, pressing F2 caused mouse and keyboard to stop working (so could not view the logs), iSCSI connections (multipath) were showing as up, vCentre had trouble connecting to one or two servers in the farm, like it was having trouble connecting to the management interface.

 

But the problem was because one on the two switches the iSCSI traffic traversed, had failed. But it wasn't obvious as all of the indicator LEDs were on, blinking etc. It was only after the Network Manager tried turning off all of the equipment that he noticed this switch didn't come up.

 

Part of reason why we didn't notice an issue with the switch was that they are both on a private network between the ESX hosts and the SAN, so our network monitoring software can't see them.

 

I'd suggest your next step should be to check the status of the switches between the ESX hosts and your SAN, if it's setup that way.

 

If you have HA enabled, I would also consider whether it has attempted to migrate VMs from one host to another, and check that all hosts can access all data stores.

 

I know it's not exactly the same problem, but does have a lot of similarities.

 

Good luck...

 

Thanks,

 

Bruce.

 

Honestly speaking, the switches were next on my list to troubleshoot at the time of reading your post, so I jumped straight to it. I had one host cold boot and when it was hanging while loading vmk_iscsi I took power from the switches and the host loaded the rest of the files immediately and finished the boot.

 

I plugged the switch back in then I was able to start vcenter, connect to it via web/client, this is when the slow performance came back and vms took a good 10 to 15 minutes to power on. I thought I'd try one more thing before ripping the switches out, I powered on another vm and whilst it was trying to do this I removed the fiber uplink form the switch and the vm powered on without issue, then I powered on all of the vms and it worked as it did before.

 

I'm using ubiquiti unifi switches so I might give their forum a go as I'm not too sure how to figure this one out

Edited October 27, 2018 by Obzen

[kmount]

Hanging on that means it's most likely storage connectivity. Check vmkping on each storage nic to each iscsi target ip and check if you're using jumbo frames or not.

[Obzen]

Hanging on that means it's most likely storage connectivity. Check vmkping on each storage nic to each iscsi target ip and check if you're using jumbo frames or not.

 

vmpkings all returned normal and jumbo frames are enabled at 9000 MTU on the vswitches and physical switches. As soon as I unplug the fiber uplink on the first switch then it all grinds to a halt again, I'll have another play when I get in tomorrow. Another thing I checked was the esxtop on DAVG which were just sitting at 0.01 - 1.10.

[kmount]

Did you run vmkping with the do not fragment option and force it to use each of your storage vmkernelports? (e.g. vmkping -I vmk1 -d -s 8972 10.10.10.10)

 

I have a feeling you might be routing your iSCSI packets through the uplink (the working one) and when you unplug it bad things happen.

 

I'd be very interested in hearing more about the setup and helping you get to the bottom of it

[Davit2005]

What Unifi Switches are you using for iSCSI. Personally I use the Ubiquiti ES-16-XG because I've had issues with spanning tree on the Unifi range i.e. US-** . The one production environment I did have with iSCSI I had completly seperate switches. I don't really see how DNS can effect the boot time of ESXi but I stand to be corrected.

 

As @kmount mentioned, specifically how the host side is configured and the network connections.

[Obzen]

Did you run vmkping with the do not fragment option and force it to use each of your storage vmkernelports? (e.g. vmkping -I vmk1 -d -s 8972 10.10.10.10)

 

I have a feeling you might be routing your iSCSI packets through the uplink (the working one) and when you unplug it bad things happen.

 

I'd be very interested in hearing more about the setup and helping you get to the bottom of it

 

I have just run the vmkping with the DF options to each iscsi target on the san controllers from both hosts, 4 ports per controller, the first two targets return with 0 packet loss while changing vmk1 to vmk2 on the respective targets however the last two targets don't come back with any response. (hope this make sense!)

 

I'd be more than happy to show you the setup, I have sent you a PM.

[Obzen]

What Unifi Switches are you using for iSCSI. Personally I use the Ubiquiti ES-16-XG because I've had issues with spanning tree on the Unifi range i.e. US-** . The one production environment I did have with iSCSI I had completly seperate switches. I don't really see how DNS can effect the boot time of ESXi but I stand to be corrected.

 

As @kmount mentioned, specifically how the host side is configured and the network connections.

 

We are using US-48s for the iSCSI switches, originally I bought a couple to test before doing a full site deploylment and they worked fine. From there all of the swithes were upgraded to Unifi in August and to be honest I haven't had any issues. iSCSI, V Motion VLANs and Jumbo Frame settings were replicated and they haven't missed a beat since install. The only thing I can say is that this is the first time that the hosts were rebooted with these switches, this possibly could be the reason?