Storage vMotion problem

[chris11256]

Hi All,

 

I'm having a problem with storage vMotion. No matter which host is involved or vm I try, a combines compute & storage vMotion fails. The message is 'failed to read stream keepalive: Connection closed by remote host, possibly due to timeout'

 

For context the SAN is SAS, so both hosts are connectedto both controllers. vMotion also has a dedicted nic on each host and Standard compute vMotions work perfectly, it's just storage & compute that doesn't. I can't do a standalone live storage vMotion as we don't have the enterprise license. Also a migration with the vm powered off doesn't give the same error, but progress never moves beyond 30% and I end up cancelling after several hours. I'm also able to move files between datastores in the browser(such as the folder with server ISO files in).

 

Thanks in advance!

Edited July 21, 2020 by chris11256

[kmount]

Check your vmotion nic tcp/ip stack is consistent, check for mtu as well.

 

Which version of esxi?

 

What’s in the vpxa/d and hostd logs?

[chris11256]

Thank you.

 

mtu is default(1500) on both hosts and tcp/ip settings are the same along with security settings being default(reject Promiscuous mode and accept mac changes & forged transmits) and both hosts are running 6.7 u3 patch release 16316930. Very sorry for the messy formatting below, not q

 

Here's vpxa extract at the time the failure message popped up:

2020-07-22T08:47:42.478Z info vpxa[2145156] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-59596@2263866-20e97a8b-61] [VpxLRO] -- BEGIN lro-185368 -- vpxa -- vpxapi.VpxaService.setConfig -- 5297c0d4-e365-5f48-e955-3fea5da97b822020-07-22T08:47:42.483Z info vpxa[2145156] [Originator@6876 sub=vpxLro opID=HB-SpecSync-host-59596@2263866-20e97a8b-61] [VpxLRO] -- FINISH lro-1853682020-07-22T08:48:35.695Z info vpxa[2145167] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-606df6c8-0] [VpxLRO] -- BEGIN lro-185369 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 5297c0d4-e365-5f48-e955-3fea5da97b822020-07-22T08:48:35.696Z info vpxa[2145167] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-606df6c8-0] [VpxLRO] -- FINISH lro-1853692020-07-22T08:48:40.669Z info vpxa[2145163] [Originator@6876 sub=vpxLro opID=sps-Main-89316-757-d6-e8] [VpxLRO] -- BEGIN task-42236 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange -- 5297c0d4-e365-5f48-e955-3fea5da97b822020-07-22T08:48:40.700Z info vpxa[2145154] [Originator@6876 sub=vpxLro opID=sps-Main-89316-757-d6-e8] [VpxLRO] -- FINISH task-422362020-07-22T08:48:40.700Z info vpxa[2145154] [Originator@6876 sub=Default opID=sps-Main-89316-757-d6-e8] [VpxLRO] -- ERROR task-42236 -- catalogSyncManager -- vim.vslm.host.CatalogSyncManager.queryCatalogChange: vim.fault.NotFound:--> Result:--> (vim.fault.NotFound) {--> faultCause = (vmodl.MethodFault) null,--> faultMessage = --> msg = "The object or item referred to could not be found."--> }--> Args:-->--> Arg catalogChangeSpec:--> (vim.vslm.CatalogChangeSpec) {--> datastore = 'vim.Datastore:ds:///vmfs/volumes/5db2be18-940a2f34-9d10-8030e043201c/',--> startVClockTime = (vim.vslm.VClockInfo) {--> vClockTime = 1--> },--> fullSync = false--> }2020-07-22T08:48:41.011Z info vpxa[2099819] [Originator@6876 sub=vpxLro opID=2243c45a] [VpxLRO] -- FINISH lro-1853532020-07-22T08:48:41.011Z info vpxa[2099819] [Originator@6876 sub=Default opID=2243c45a] [VpxLRO] -- ERROR lro-185353 -- vpxa -- vpxapi.VpxaService.GetVmScreenshot: vim.fault.NotFound:--> Result:--> (vim.fault.NotFound) {--> faultCause = (vmodl.MethodFault) null,--> faultMessage = --> msg = ""--> }--> Args:-->--> Arg vmid:--> 50--> Arg snapshot:-->--> Arg options:--> "&h=120&w=160"2020-07-22T08:48:41.265Z info vpxa[2099836] [Originator@6876 sub=vpxLro opID=kcruy6gv-158106-auto-3dzv-h5:70014996-b9-01-b3] [VpxLRO] -- FINISH task-422272020-07-22T08:48:41.266Z info vpxa[2099836] [Originator@6876 sub=Default opID=kcruy6gv-158106-auto-3dzv-h5:70014996-b9-01-b3] [VpxLRO] -- ERROR task-42227 -- vmotionManager -- vim.host.VMotionManager.initiateSourceEx: vim.fault.GenericVmConfigFault:--> Result:--> (vim.fault.GenericVmConfigFault) {--> faultCause = (vmodl.MethodFault) null,--> faultMessage = (vmodl.LocalizableMessage) [--> (vmodl.LocalizableMessage) {--> key = "msg.checkpoint.precopyfailure",--> arg = (vmodl.KeyAnyValue) [--> (vmodl.KeyAnyValue) {--> key = "1",--> value = "10.100.0.172"--> },--> (vmodl.KeyAnyValue) {--> key = "2",--> value = "msg.vmk.status.VMK_MIG_CONN_CLOSED"--> },--> (vmodl.KeyAnyValue) {--> key = "3",--> value = "195887167"--> }--> ],--> message = "Migration to host <10.100.0.172> failed with error Connection closed by remote host, possibly due to timeout (195887167). "--> },--> (vmodl.LocalizableMessage) {--> key = "vob.vmotion.stream.keepalive.read.fail",--> arg = (vmodl.KeyAnyValue) [--> (vmodl.KeyAnyValue) {--> key = "1",--> value = "174325928"--> },--> (vmodl.KeyAnyValue) {--> key = "2",--> value = "4702273180757176283"--> },--> (vmodl.KeyAnyValue) {--> key = "3",--> value = "Connection closed by remote host, possibly due to timeout"--> }--> ],--> message = "vMotion migration [174325928:4702273180757176283] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout"

 

hostd keeps mentioning vim.fault.NotFound, but this is interesting as the vmware KB article talks about this problem in relation to distributed switches. I only have standard switches.

2020-07-22T08:48:41.167Z info hostd[2100047] [Originator@6876 sub=Vcsvc.VMotionSrc.4702273180757176283] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION2020-07-22T08:48:41.167Z info hostd[2100047] [Originator@6876 sub=Vcsvc.VMotionSrc.4702273180757176283] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {--> faultCause = (vmodl.MethodFault) null,--> faultMessage = (vmodl.LocalizableMessage) [--> (vmodl.LocalizableMessage) {--> key = "msg.checkpoint.precopyfailure",--> arg = (vmodl.KeyAnyValue) [--> (vmodl.KeyAnyValue) {--> key = "1",--> value = "10.100.0.172"--> },--> (vmodl.KeyAnyValue) {--> key = "2",--> value = "msg.vmk.status.VMK_MIG_CONN_CLOSED"--> },--> (vmodl.KeyAnyValue) {--> key = "3",--> value = "195887167"--> }--> ],--> message = "Migration to host <10.100.0.172> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).--> "--> },--> (vmodl.LocalizableMessage) {--> key = "vob.vmotion.stream.keepalive.read.fail",--> arg = (vmodl.KeyAnyValue) [--> (vmodl.KeyAnyValue) {--> key = "1",--> value = "174325928"--> },--> (vmodl.KeyAnyValue) {--> key = "2",--> value = "4702273180757176283"--> },--> (vmodl.KeyAnyValue) {--> key = "3",--> value = "Connection closed by remote host, possibly due to timeout"--> }--> ],--> message = "vMotion migration [a6400a8:4702273180757176283] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout--> "--> }--> ],--> reason = "Migration to host <10.100.0.172> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).--> "--> msg = "Migration to host <10.100.0.172> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).--> vMotion migration [a6400a8:4702273180757176283] failed to read stream keepalive: Connection closed by remote host, possibly due to timeout--> "

Edited July 22, 2020 by chris11256