Network copy speeds

[mjk]

Nah, 8MBps would be 64Mbps, which if the transfer is to a 100Mbit connected device, would be a reasonable speed.

 

The original post did say "gig ports"

[localzuk]

The original post did say "gig ports"

Ah, I missed that it was the OP. Yeah, definitely something wrong then!

[PotNoodleTech]

Yeh. OP needs to fill us in on the setup, server spec, switches etc maybe we can figure something out!

[howartp]

Yeh. OP needs to fill us in on the setup, server spec, switches etc maybe we can figure something out!

 

Destination is 2008 R2, quad core Intel Xeon 64bit with 10GB RAM, 2 Broadcom NetXtreme II GigE ports.

Origin is 2008 R2, dual core Intel Xeon (vSphere 5.5 vm) with 6Gb RAM, 1 VMXNET3 virtual NIC based on Intel 82580 Gigabit NICs.

 

VMware estate is all on a Cisco 3750 which has four 1GB copper ports trunked together to the HP Core 5406zl.

 

File transfer was running on the destination server.

 

We have same problem with speeds between server VMs and the Veeam Backup server which is in another building, on two 1GB fibre ports trunked between HP Core and a HP 2530-24G POE switch.

 

Peter

[howartp]

Typical - I'm now getting 24 MB/s so I presume I was getting 8MB/s yesterday.

[PotNoodleTech]

My gut is telling me this is likely a disk issue rather than a network one - what kind of disk/raid config you got on both the VMware setup and the destination server? Do you have battery backup'd cache on the raid controllers?

[Oaktech]

It could be a disk issue, but if it's also a problem between VMs I wonder if the OP needs a nic driver update. He's already said there is no option to turn off VMQ, so I wonder if a nic driver update will give that option (or fix the issue completely like my last Lenovo server update did)

[Sagima]

It could be a disk issue, but if it's also a problem between VMs I wonder if the OP needs a nic driver update. He's already said there is no option to turn off VMQ, so I wonder if a nic driver update will give that option (or fix the issue completely like my last Lenovo server update did)

 

I'd still run the powershell command to turn it off even if it wasn't exposed in the gui. It does seem a strong possibility.

 

---

 

Get-NetAdapterVmq

 

then for all listed as true

 

Set-NetAdapterVmq -Name "NIC Name" -Enabled $False

 

---

 

There's probably a command line option to do the same.

 

I believe VMQ should be off for 1Gig and under connections and on for 10Gb and over iirc (I don't know about 2.5Gb and 5Gb as I have never needed to look into it as I don't have switches or NICs that support those speeds)

[howartp]

My gut is telling me this is likely a disk issue rather than a network one - what kind of disk/raid config you got on both the VMware setup and the destination server? Do you have battery backup'd cache on the raid controllers?

VMware is on an IBM SAN with everything RAID5.

 

Some datastores are fast, small capacity SAS disks and some are slower, high capacity SATA disks; SQL and DC's are on the SAS disks, general data is on the SATA disks.

 

This data is from the SATA disks.

 

It's highly likely both the NAS and the Backup server are RAID5 if I've set them up. (I think the NAS was my manager before I took his job)

 

It could be a disk issue, but if it's also a problem between VMs I wonder if the OP needs a nic driver update. He's already said there is no option to turn off VMQ, so I wonder if a nic driver update will give that option (or fix the issue completely like my last Lenovo server update did)

 

I'd still run the powershell command to turn it off even if it wasn't exposed in the gui. It does seem a strong possibility.

 

---

 

Get-NetAdapterVmq

 

then for all listed as true

 

Set-NetAdapterVmq -Name "NIC Name" -Enabled $False

 

---

 

There's probably a command line option to do the same.

 

I believe VMQ should be off for 1Gig and under connections and on for 10Gb and over iirc (I don't know about 2.5Gb and 5Gb as I have never needed to look into it as I don't have switches or NICs that support those speeds)

 

I'll try between VMs and see what speeds we're getting - don't think I've tried that recently.

 

I tried the powershell last week but the server didn't have high enough version - i'll have a look further.

[Sagima]

You can upgrade powershell by updatinging the windows management framework. This will update it to version 5 for Server 2008 R2 or later https://www.microsoft.com/en-us/download/details.aspx?id=50395

[PotNoodleTech]

VMware is on an IBM SAN with everything RAID5.

 

Some datastores are fast, small capacity SAS disks and some are slower, high capacity SATA disks; SQL and DC's are on the SAS disks, general data is on the SATA disks.

 

This data is from the SATA disks.[/Quote]

 

It would be useful if you used a disk benchmarking something along the lines of crystaldiskmark works well as it's small, straightforward and can immediately identify disk issues e.g. that write cache is not enabled due to not having battery backup cache on your raid card?

[howartp]

SAS Disk, SATA Disk, NAS server and Backup server images in that order.

[howartp]

You can upgrade powershell by updatinging the windows management framework.

Done - but needs reboot before I can try it

[mjk]

At risk of repeating myself, I did give an example of exactly how you should measure your network speeds before you go looking for other issues.

Ignore me by all means but VMware are offering exactly the same advice : https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052119

[PotNoodleTech]

The disks speeds don't look to be an issue so I would go with mjk idea.

 

hey mjk it only takes 5 minutes to run crystal disk mark, well worth doing to identify any obvious disk issues.

 

I've seen people looking at network issues for days and when they eventually run disk benchmarks their disks were incapable of writing at greater than 10MB/s (crap controller, no cache, no writeback cache, SATA disks)!!

[howartp]

I did give an example of exactly how you should measure your network speeds before you go looking for other issues.

No you didn't - you suggested I "use a network diagnostic tool to get the number" which means nothing to me.

 

Now that you've given me a set of testable commands, I'm more than happy to try them and report back.

 

The nc command test on the VMware link you gave me is giving ~20 MbTX/s and ~950 MbRx/s on both the vmk2 kernel port and the vmnic3 vSwitch port.

 

The vSwitch that all the VMs are connected to has three physical NICs on it - vmnic2, vmnic3 and vmnic4 - but there's generally only traffic showing on vmnic3 - shouldn't I expect traffic on all three vmnics most of the time? (There is some traffic on the other 2 but not much)

 

Peter

[ITGURU]

All depends on the type of files being copied. one large file will always have a better throughput.

[mjk]

No you didn't - you suggested I "use a network diagnostic tool to get the number" which means nothing to me.

 

Now that you've given me a set of testable commands, I'm more than happy to try them and report back.

 

The nc command test on the VMware link you gave me is giving ~20 MbTX/s and ~950 MbRx/s on both the vmk2 kernel port and the vmnic3 vSwitch port.

 

The vSwitch that all the VMs are connected to has three physical NICs on it - vmnic2, vmnic3 and vmnic4 - but there's generally only traffic showing on vmnic3 - shouldn't I expect traffic on all three vmnics most of the time? (There is some traffic on the other 2 but not much)

 

Peter

Sorry if it wasn't clear : my very first post had a netcat command posted into it.

Things are pointing to a network issue if the transmit speeds are down. I would try plugging in another machine (a laptop or something) into the port the vmware sever is in and trying nc between two machines ont eh same switch

https://sourceforge.net/projects/nc110/

[mavhc]

I'd use iperf to test network speeds.

[localzuk]

I'd use iperf to test network speeds.

I second this one. Used it a lot with our VPNs to locate problems.

[mjk]

... it's why I didn't bang on about netcat too much and tried to talk about a 'generic network tool'. iperf is good too.

[howartp]

Sorry if it wasn't clear : my very first post had a netcat command posted into it.

Sorry - I did see that post originally but I'm not a Linux person so those commands went completely over my head - I hadn't realised it was you that posted it when I was looking back to your subsequent post.

 

I seem to think I've used iPerf before so I'll have a play with that.

 

Peter

[howartp]

iPerf is giving me 800 Mbits/sec both ways from VM to NAS.

 

NAS to Backup is 940 Mbits/sec both ways.

 

VM to Backup is 650 Mbits/sec

Backup to VM is 880 Mbits/sec

Edited March 8, 2017 by howartp