Help Please :( Server NIC issue

[abillybob]

Hi All

 

We have a Hyper-V Server which has two nics. The one nic hosts 3 servers and the other nic hosts 3 servers.

 

All servers on NIC 1 are working perfectly fine, I've checked the physical host and it's logs and can't find anything wrong

 

All servers on NIC 2 keep intermittently dropping connection every minute or so they'll drop for a couple of seconds and then come back online. This has only recently started to happen. Both NIC's are the same make and model and both are on the same driver version.

 

Not sure what would be causing this, the server also seems slower to respond when remotely controlling it but I'm not sure if that's because it's dropping in and out it's connection.

 

I've worked all day trying to work out what this is, I've looked through Wireshark and there doesn't seem to be a massive amount of traffic going to/from the server either?

 

If anyone can help it would be most appreciated!!

[strawberry]

Tried recreating the virtual adapters? . Tried switching the 3 vm's over to see if a VM is causing it?

[ZeroHour]

Does NIC 2 work when sending requests to the HOST server or does it drop as well?

Have you elimated the server port/cable?

[abillybob]

Tried recreating the virtual adapters? . Tried switching the 3 vm's over to see if a VM is causing it?

Yes tried re-creating them but it didn't help. Haven't tried that but I can't just yet as our factory is running and taking them offline for a period of time whilst I move them across and test would cause mayhem I did a restart but that was only offline then for about a minute otherwise this would take too long to test.

Does NIC 2 work when sending requests to the HOST server or does it drop as well?

Have you elimated the server port/cable?

I actually didn't check that so I'm running a ping now to all the IP's under that NIC including it's physical IP and will let you know. Had it running for a bit of time now and it's been solid but this is what it looked like before when I was pinging a VM. It says "no route to host" which is strange as normally it just says timeout so I'm not sure if that's something?

Yes cable all working fine, we are using a new cable now just in case but still has same problem, port hasn't been changed and has worked fine up until now? How would I test the port is in working order?

Thanks both for your help!!

 

I see this quite often also? Nothing to worry about?

 

Edited November 25, 2019 by abillybob

[DJ-1701]

As you work in what you have describe previously as a busy industry now, do you have another HyperV to load the affected VMs on while you diagnose the network issue? If not, might be something to think of going forward for both sanity and DR reasons.

[abillybob]

As you work in what you have describe previously as a busy industry now, do you have another HyperV to load the affected VMs on while you diagnose the network issue? If not, might be something to think of going forward for both sanity and DR reasons.

Yeah I have a second identical Hyper-V server that mirrors our first just in case but I try not to use it unless a real tragedy happens like it's offline indefinitely or corruption etc... I don't think it's the shortest, quickest process in the world have to perform a planned failover. A unplanned failover is quicker but then it messes up the sync so I don't want to do that either if I can help it.

 

My sanity is long gone ....

[abillybob]

By the way I've just witnessed the physical NIC goes down when the VM's go off also so it's got to be a problem I presume with the physical NIC?

 

It doesn't give me much information in Event Viewer

Edited November 25, 2019 by abillybob

[FishCustard]

Are these onboard NICs or PCI-E cards. If the latter, I'd swap the card out and see what happens.

[mavhc]

Best to actually test your failover now, when everyone's not panicking as much.

 

Why are the VMs using different NICs?

[abillybob]

Are these onboard NICs or PCI-E cards. If the latter, I'd swap the card out and see what happens.

 

Swapped all servers to the NIC 1 and it's working with no issues, unfortunately it's an on-board NIC so can't swap it out, I was thinking maybe to test with a USB on one server under the same IP address and see if it still does the same but I don't have a USB NIC to hand!

Best to actually test your failover now, when everyone's not panicking as much.

Why are the VMs using different NICs?

I wouldn't be able to, they'd sack me if I said I'd have to take down the Factory for 15 minutes and I'm not exaggerating

 

Load balancing mainly, I don't rate Teaming NIC's because if 1 goes down suddenly they both play up so by keeping them separate with separate IP's etc it means you have a last resort to swap all over to 1 like I just have pretty quickly.

[3s-gtech]

Slightly unusual NIC setup. My hosts have two onboard NICs and 4 on a PCI-e card. The onboard is used for host only, while the others are teamed and connected to the v-switch with no host access. Fairly common setup, especially where only 1Gb NICs are installed. It’s not necessarily better than what you have done, but teaming should allow seamless failover if a port goes down.

[HPlum78]

Just out of interest what does the switch say? You are thinking host but it may not be as clear cut as that.

[abillybob]

Slightly unusual NIC setup. My hosts have two onboard NICs and 4 on a PCI-e card. The onboard is used for host only, while the others are teamed and connected to the v-switch with no host access. Fairly common setup, especially where only 1Gb NICs are installed. It’s not necessarily better than what you have done, but teaming should allow seamless failover if a port goes down.

I've always had problems with teaming I'm not sure why, every time I've ever got it to work I've had packet drops and things. Maybe I'm doing something wrong but that's why I've always steered clear of it. We are upgrading out Core switch to a 10Gb variety soon so I think I'll probably do similar to you but with 2x 10Gbit ports on a PCI-e card and make it a bit more current as we do have some heavy going servers on this host.

 

Though if teaming allows for a seamless failover how do you know there's anything wrong at all when one of the NIC's do go in the future? Surely eventually you'll just notice the servers responding slower and it will take longer to work out it's the NIC as to the naked eye the teamed NIC will look as though it's functioning normally?

 

Just out of interest what does the switch say? You are thinking host but it may not be as clear cut as that.

 

Yeah did think about this, I did check the port status on the switch and nothing was un-toward so not sure and another device on the same VLAN in the same ports works fine. I have even changed the server to use a different port on our stacked switches but to no avail. Is there anything else I can do in order to see whether it's the switch and not the NIC?

 

I've figured out that even with all VM's now using NIC1 and nothing going through NIC2. NIC 2 still has problems dropping it's connection.

[ZeroHour]

I actually didn't check that so I'm running a ping now to all the IP's under that NIC including it's physical IP and will let you know. Had it running for a bit of time now and it's been solid but this is what it looked like before when I was pinging a VM. It says "no route to host" which is strange as normally it just says timeout so I'm not sure if that's something?

Yes cable all working fine, we are using a new cable now just in case but still has same problem, port hasn't been changed and has worked fine up until now? How would I test the port is in working order?

Aha so it does seem like a NIC or port issue, simplest way to test would be to change the port to another on the switch. If the 2 NICS are on different switches or you are out of ports you could switch the cables around and see if the issue occurs on NIC 1.

DO NOT USE A USB NIC! they have a huge heat problem (generally) with a lot of traffic unless you get a proper one.

[FishCustard]

DO NOT USE A USB NIC! they have a huge heat problem (generally) with a lot of traffic unless you get a proper one.

They have an "overheat and die" problem to be specific.

[bald_pig]

With proper monitoring in place, you can get alerts for hardware issues.

 

Also, get some proper redundancy in place if everything is as mission critical as you claim it is!

[abillybob]

Aha so it does seem like a NIC or port issue, simplest way to test would be to change the port to another on the switch. If the 2 NICS are on different switches or you are out of ports you could switch the cables around and see if the issue occurs on NIC 1.

DO NOT USE A USB NIC! they have a huge heat problem (generally) with a lot of traffic unless you get a proper one.

Changed NIC2 to another switch entirely but still the same issue so I suppose all thing point to NIC?

[abillybob]

With proper monitoring in place, you can get alerts for hardware issues.

 

Also, get some proper redundancy in place if everything is as mission critical as you claim it is!

I have proper monitoring and alerts set up? I have got 4 types of redundancy.. but uh... thanks?

[Katy]

I have proper monitoring and alerts set up? I have got 4 types of redundancy.. but uh... thanks?

 

In which case you'd get a notification if half a NIC team fell over, which you said you wouldn't know about?

 

Can't really say without seeing the setup but I'd have had at least 2 hosts in a cluster and been able to live migrate the VMs off the dodgy host in order to investigate it without causing any downtime?

[abillybob]

In which case you'd get a notification if half a NIC team fell over, which you said you wouldn't know about?

Can't really say without seeing the setup but I'd have had at least 2 hosts in a cluster and been able to live migrate the VMs off the dodgy host in order to investigate it without causing any downtime?

 

Not really the case, the issue from my POV is that every time I've tried to get Teaming working properly somethings gone wrong... could be the rubbish network cards I've used "Realtek & Killer" or the fact I'm just bad at doing it but I've always had one problem or another so I tend to try and avoid if I can. I would probably but wouldn't know how to set this up or what I'm doing as I haven't played with it much.

 

It's ok I have them all going through the 1 NIC at the moment and it's coping quite well so I'm able to test but I'm pretty sure it's bugg*red!

[Boredguy]

Does sound like you really need to get Hyper-V cluster setup quickly depending on how your data is being stored in case you primary NIC starts to suffer, as then you could do a live migration without downtime while fixing the host.

[Katy]

It's ok I have them all going through the 1 NIC at the moment

This is why everyone is mentioning redundancy, as you previously had 3 VMs per NIC apparently set manually this way, so it appeared to the casual bystander that there is none.

 

Anyway it does sound like the NIC is knackered but good luck hope you get it to behave itself.

[bald_pig]

This is why everyone is mentioning redundancy, as you previously had 3 VMs per NIC apparently set manually this way, so it appeared to the casual bystander that there is none.

 

Anyway it does sound like the NIC is knackered but good luck hope you get it to behave itself.

 

That and you was complaining about downtime, which there wouldn't be with a proper failover setup in place.

[Fazza]

The only time I've seen this issue is when we had an IP conflict on the server (with itself). Hyper-V was installed (which creates it's own bridge to your NIC) and had a static IP address of the server installed, later on the 4 onboard NICs (rather than just one being used) were teamed together which had a static IP address set and the Hyper-V one should have been blanked by the OS during this process but wasnt so for a short period of time the server had the same IP address on two NICs so PINGs were erratic at times - so it might be worth checking each adapter in the network settings and just double check the IP addresses.