Hyper-V Slowness - Connecting and Using the VMs

rich_tech · January 29, 2016

Hello All,

I am posting here as I have been seeing issues with our Hyper-V setup here for a while now and I have noticed it seems to be getting progressively slower for us consoling in and using the servers themselves on certain VMs.

We are seeing a slowness to connect via the console to a VM or to an RDP session with our Hyper-V setup, It seems horribly laggy and slow.

Our setup is a VLAN based network, servers and nodes are on their own VLAN, we are sat on another vlan accessing the servers and interfaces fine, no issues with other servers when we connect to them here.

We use 1 Ethernet port on each node server as a virtual switch, the other IP remains for management and their rigged for failover along with a standalone network specifically for that purpose using high quality crossover cables and bypassing the VLAN setup, connections to iSCSI SAN are also direct cabled and do not go through Vlans as they are all in the same rack. This setup has worked fine until the last few months but we have noticed issues with it slowing down or being slow to access and use the VMs. I have had the networking side of it looked at and monitored by our LEA networking team and there doesnt seem to be anything horrible going on with regards usage and utilisation unless we are moving the VMs between boxes which is understandable.

I have done a fair bit of research and turned off VMQ's as a trial, but this hasnt made any impovement that is noticeable and possibly made it a little worse so I am going to revert that change today.

The setup is Hyper-V based cluster - both nodes run Server 2012 R2 and their configurations mimic one another, having gone through all the same setups, such as NIC settings/Hyper-V server settings and all the associated setup configurations, the cluster passed validation no problems at all before going live also.

The server hardware consists of 2 x DL380e Gen8 servers and 1 HP MSA P2000 SAN, the HP has the up to date NIC drivers for both the dual and quad port NICs installed also.

Has anyone else experience slowness like this on their clusters ? or know of a fix for this issue ?

Thanks,

Rich

stevenlong1985 · February 1, 2016

This sounds familiar, I'm sure this has happened to me.

I'm checking my old emails.

and what i could find was problems were:

1. is your P2000 running the latest Firmware?

2. have you check the Hard drives - I had a faulty drive which was slowing everything down.

Steve

rich_tech · February 4, 2016

I didnt consider it could be the SAN funnily enough, HP made us update it when a flash card failed in a controller last year.

We are running the following at the moment and it appears there is a newer version which I have just downloaded, just need to schedule a late night installation now as I will have to take the attached VMs down.

[TABLE=class: objectTable, width: 0, align: left]

[TR]

[TD=class: objectTitle]Component[/TD]

[TD=class: objectTitle]Controller A[/TD]

[TD=class: objectTitle]Controller B[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Bundle Version[/TD]

[TD=class: objectAltValue]TS251P006-02[/TD]

[/TR]

[TR]

[TD=class: objectValue]Storage Controller Code Version[/TD]

[TD=class: objectValue]T251P08-01[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Storage Controller Loader Code Version[/TD]

[TD=class: objectAltValue]23.008[/TD]

[/TR]

[TR]

[TD=class: objectValue]Memory Controller FPGA Code Version[/TD]

[TD=class: objectValue]F400R02[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Management Controller Code Version[/TD]

[TD=class: objectAltValue]L251R005-11[/TD]

[/TR]

[TR]

[TD=class: objectValue]Management Controller Loader Code Version[/TD]

[TD=class: objectValue]2.5[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Expander Controller Code Version[/TD]

[TD=class: objectAltValue]2028[/TD]

[/TR]

[TR]

[TD=class: objectValue]CPLD Code Version[/TD]

[TD=class: objectValue]22[/TD]

[/TR]

[/TABLE]

rich_tech · February 8, 2016

Had an interesting time of it trying to update this on the weekend, Controller B seemingly blew a gasket, I made sure Partner update was off in line with HP recommendations and it basically never came properly out of its reboot, when it did come back up, it was clearly on the old firmware, multiple attempts at updating the firmware went awry too, A is fine and hold it all up now, B - I had to remove and reinsert the controller module today as it has got itself into a state, where I could not shut down the controller. Everything I tried with it has caused issues, even FTP updating of this hasnt worked, though now its back alive, but now I have a firmware mismatch after trying to update it when it came back alive this morning following the pull and reseat I did.

HP dont make it easier by allowing you to target a specific controller for updating either imo.

Thats it current state of play at the moment below, waiting on HP to get back to me now about it and how I remedy this problem.

[TABLE=class: objectTable, width: 0, align: left]

[TR]

[TD=class: objectTitle]Component[/TD]

[TD=class: objectTitle]Controller A[/TD]

[TD=class: objectTitle]Controller B[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Bundle Version[/TD]

[TD=class: objectAltValue]TS252R007[/TD]

[/TR]

[TR]

[TD=class: objectValue]Storage Controller Code Version[/TD]

[TD=class: objectValue]T252R06-01[/TD]

[TD=class: objectValue]T251P08-01[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Storage Controller Loader Code Version[/TD]

[TD=class: objectAltValue]23.008[/TD]

[/TR]

[TR]

[TD=class: objectValue]Memory Controller FPGA Code Version[/TD]

[TD=class: objectValue]F400R02[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Management Controller Code Version[/TD]

[TD=class: objectAltValue]L252R008-01[/TD]

[/TR]

[TR]

[TD=class: objectValue]Management Controller Loader Code Version[/TD]

[TD=class: objectValue]2.5[/TD]

[/TR]

[TR]

[TD=class: objectAltValue]Expander Controller Code Version[/TD]

[TD=class: objectAltValue]2028[/TD]

[/TR]

[TR]

[TD=class: objectValue]CPLD Code Version[/TD]

[TD=class: objectValue]22[/TD]

[/TR]

[/TABLE]

Oaktech · February 8, 2016

I've been quoted a very similar system and reading issues like this make me unkeen to go down the route, when our current system is aging, but hardware wise very reliable.

My only thought on your problem - check the settings for the network cards as I've previously had problems with the checksum offload settings. They were always set to TX & RX and suddenly one day that made everything take a really long time to respond - ping was fine, but actually doing anything was really slow. Turning all of them to disabled restored performance. This was also DL380s

rich_tech · February 9, 2016

Thanks for the suggestions so far,

I did some reading last week and turned off checksum offload on everything as it was set to Tx and Rx and another thread I was reading said to turn them off when I was looking for solutions to this. At the moment though, we have a serious issue with hardware and I cant be sure if there has been any fix of that until its fixed on our SAN.

It seems we have a much more major issue on our SAN at the moment, I have HP coming out today with a brand new controller module and compact flash, as yesterday afternoon, we had a failure on controller B in our unit of the compact flash, added to which the storage controller seems broken on the module and refuses to upgrade its firmware, so HP are sending someone out to change that for us today as we are still under support to their credit their support is quite excellent from my point of view.

I don't know where I stand with HP SANs though after this myself, from my singular point of reference.

Its our first unit and its impressive in terms of technology and how failover happens and multipathing with this HP unit, but as I said to @garethedmondson yesterday, we have had issues with our SAN over the entire time its been in operation on and off(about 30 months now), the 2 HP server nodes attached to them are absolutely rock solid (*touches wood*), though the SAN is something else.

HP seems to be terribly designed in terms of upgrading it too, you cannot target a specific controller for upgrading, its either an all or nothing scenario it seems and they tell you to turn off their partner controller firmware updating feature as standard as its known to cause issues on their devices.

The compact flash that's being replaced is already a replacement card, as it went before about 15 months ago.
We have also had an issue where this same failing controller module played up about 9 months into service and failed to register on the unit, where the phone support had to talk me through removing and reinserting the module to remedy.
We have also had a single disc failure where HP sent a replacement out for us.

That has quite a dodgy history in my mind especially as it was a brand new device, compared to the servers attached to it, which are solid.

Edited February 9, 2016 by rich_tech

Achandler · February 9, 2016

I know it doesn't help you, and I ahve used a P2000 in the past and know they are a nightmare to update.

We have a P3500 here and it is easy to update, we can jsut use the controller software and tell it to update, live in the middle of the day, its not fussed, it does each half of the SAN one by one. Rebooting it all etc, however, they are much more expensive!

garethEds · February 10, 2016

@rich_tech and @Achandler - would the drives from our P2000 just slot into a P3500? Should we be looking at upgrading our hardware?

Do you have a link to a P3500? A Google search brings up desktop machines.

Gareth

Achandler · February 10, 2016

I think they are all sold as P3000 but you get different versions of it. The drives, I guess it depends on the versions of the machines, ours here are LFF, and you can buy both SANs in both SFF and LFF I believe.

Also turns out we bought the P4500G2 in the end not P3000, apologies. The P4000 when you google it is again a range, where there is lots of versions as always with HP.

Personally, it depends where you are looking to go in the future, onsite storage offsite cloud storage etc. Don't get me wrong I like the SAN we have, it seems quicker then the old P2000 I have used, and it is split in two locations which is a massive help should you ever need to do maintenance to the SAN itself or the power has to be turned off in areas of the school. It also helps with disaster planning etc. With that said, splitting a SAN means you proably need to run fibre if your putting it in two locations in a school and you have to consider if its over the top for the school requirements.

I am currently looking at the possibility of creating a hybrid file server with OneDrive (waiting to see what MS do), and if successful we might move more to better leased lines and away from onsite file servers for, reducing the onsite footprint of servers so making the SAN less necessary. Not fully sure personally, that I would be wanting to spend probably £15k-£20k (list prices) on a new SAN setup unless i knew I would need it for 5 years+ guaranteed.

Hyper-V Slowness - Connecting and Using the VMs

Recommended Posts

rich_tech

stevenlong1985

rich_tech

rich_tech

Oaktech

rich_tech

Achandler

garethEds

Achandler

Create an account or sign in to comment

Create an account

Sign in

Recent Threads

Bromcom Support

My Fluke LinkRunner has a fault...

iOS 27 BYOD and Smoothwall web filtering - Connectivity Assist causing issues?

Home

Forums

What's New