RDS Servers Incredibly Slow at Times

[AB_IT]

Hi All,

 

This is hurting my brain and I don't know why they do this sometimes.

 

We have two RDS Servers (VM's) on seperate Physical Hosts. Both of which have about 40 users connected at any one time doing standard office tasks. Everytime this happens I check task manager which shows the CPU and RAM usage at 20% and 50%. I've checked our Backups, they've stopped and complete at night and I've checked our AV isn't running a scan.

 

It seems to occur when the RDS Servers have recently been restarted, they then tend to go very slow. I'll faff with them for about 2-3 hours trying to work out what's going wrong and then after a couple of restarts again they speed back up.

 

I've never been able to find out the cause but I'm hoping maybe one of you guys can point me in the right direction.

 

Thank you

[slugshead]

What's the full spec or your hosts?

 

Is that the task manager for each session host or the physical host?

 

Are the other roles (gateway, wa, connection brokers) all on separate VMs?

[AB_IT]

What's the full spec or your hosts?

I took a screenshot

Is that the task manager for each session host or the physical host?

I've checked both, and they both seem to be within normal ranges. But what I was speaking about before was Task Manager on the VM's

Are the other roles (gateway, wa, connection brokers) all on separate VMs?

Yes, they're all on seperate VM's

 

We do use User Profile Disks and I wondered if it had anything to do with these? The File Server they sit on doesn't seem to be having any problems with speed etc though. I'm a bit of a novice with UPD as I've never used them in previous companies I've worked in.

Edited March 21, 2022 by AB_IT

[AB_IT]

Sorry guys but I'm really stuck. Been working non-stop since Thursday, actually did a 24 hour day on Thursday so I'm exhausted, worked every day since and today was supposed to be my time off in lieu but now this has happened so I'm at work again. I know.. *Small Violin*

 

Any help would be most appreciated. They're doing some weird things, I've looked through all the logs on event viewer also and couldn't find anything out.

[win]

Check your Windows firewall rules. There has been issues with new rules being created each time a user logs in which slows it down. You can delete these rules or make a script to auto-delete them.

[JackCPickup]

Check your Windows firewall rules. There has been issues with new rules being created each time a user logs in which slows it down. You can delete these rules or make a script to auto-delete them.

 

It's something like Xbox firewall rules that get created every time someone logs in, ends up absolutely rammed full.

[3s-gtech]

Are your host disks being hammered, with long wait times? That'll take an RDS VM to a crawl.

[AB_IT]

Check your Windows firewall rules. There has been issues with new rules being created each time a user logs in which slows it down. You can delete these rules or make a script to auto-delete them.

 

Thanks Both, I checked our firewall rules though and can't see anything untoward?

Are your host disks being hammered, with long wait times? That'll take an RDS VM to a crawl.

 

 

The RDS (CCRDS1) server seems to be writing and reading more than the others as does our File Server (CCFP) which is also where our UPD reside. We have a SAN but I've checked the SAN also and everything seems fine.

 

I'm so confused, I've had two seperate support companies as well on this to try and help but they're just as stumped.

[Boredguy]

Have you got Sophos setup not to scan the UPD files just to make sure it's not that's hogging a lot of resources.

 

When we used to run Sophos, we always tried to ensure it was not doing on access file scans on the servers but just scheduled to ensure they didn't get slowed down all the time.

[3s-gtech]

Profiles on one VM and lots of concurrent logged on users on another on the same host could well be giving you poor performance - the speeds there don't look that high but wait times may well be. Possible to move the VHDX to another drive, or bang a couple of SSDs in the host/SAN for it?

[Warwick_Tech]

Not that it's any help, but we have the same problem with several nodes on a VM - after a restart they are like treacle, but a few days later they perform as expected -

 

We figured it was a dodgy RAM stick and I'm replacing the host soon anyway - I'll let you know if that helps

[AB_IT]

Have you got Sophos setup not to scan the UPD files just to make sure it's not that's hogging a lot of resources.

When we used to run Sophos, we always tried to ensure it was not doing on access file scans on the servers but just scheduled to ensure they didn't get slowed down all the time.

 

Just checked, definetley have that path excluded. I've also checked and it's not currently running any full scans.

 

Profiles on one VM and lots of concurrent logged on users on another on the same host could well be giving you poor performance - the speeds there don't look that high but wait times may well be. Possible to move the VHDX to another drive, or bang a couple of SSDs in the host/SAN for it?

It's been this same configuration though apparently for the last 8 years and never had a problem. The one server (RDS2) is on a seperate array and that's having the same trouble if that helps.

[Boredguy]

Which version of Windows Server are you running on them currently out of interest in-case it was something in an update if it only started occurring recently?

[AB_IT]

Which version of Windows Server are you running on them currently out of interest in-case it was something in an update if it only started occurring recently?

 

Windows Server 2012 R2. It's fully patched and up to date.

 

I've just noticed though if we only turn on CCRDS2 and tell people to connect to this using the servers exact domain name it's as quick as normal. But if I turn on CCRDS1 which DNS seems to point at for the name 'CCDESKTOP' (this is the way we load balance between RDS1 and 2) it goes incredibly slow again.

 

So I think it's a problem with just RDS1 somewhere.

[Boredguy]

Is it the same if you turn off CCRDS2. If so might be a load balance issue.

 

Depending on how your licences are, could always look at reinstalling them to 2019, but that might not be a viable option for you.. wasn't for us until recently

[AB_IT]

So even weirder now. Had both RDS1 and 2 turned on yesterday and running perfectly fine. Woke up this morning (nothing has changed) no shut downs or updates etc... and it's slow again.

 

I'm really pulling my hair out again, I can't work out what this is.

[3s-gtech]

Massive backups running from the host or via the SAN?

[AB_IT]

Massive backups running from the host or via the SAN?

No backups where running, SAN uses RAID 10 but all seems fine.

[slugshead]

I forgot to reply.

 

I had the exact same servers as our old setup. Hosts to SAN (v3700, hybrid array) disk times were stupid and VMs were sluggish

 

There's an option in the physical adapter advanced options which when un-ticked brought everything to life. It was about 4 years ago I stumbled across it, can't for the life of me remember which option it was though

[meakjoe]

Virtual Machine Queues (VMQ) on Broadcom adaptors?

[slugshead]

Virtual Machine Queues (VMQ) on Broadcom adaptors?

 

That's the badger

[Cw2k]

Check your Windows firewall rules. There has been issues with new rules being created each time a user logs in which slows it down. You can delete these rules or make a script to auto-delete them.

 

They fixed that back in the March 2019 CU but you have to apply a reg key to enable the fix

 

Addresses an issue that slows server performance or causes the server to stop responding because of numerous Windows firewall rules. To enable this solution, use regedit to modify the following and set it to 1:

 

Type: “DeleteUserAppContainersOnLogoff” (DWORD)

 

Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy

[mullet_man]

Check your Windows firewall rules. There has been issues with new rules being created each time a user logs in which slows it down. You can delete these rules or make a script to auto-delete them.

This will be the issue, if you read this thread it has information on it and there was a fix released but you need to apply a reg change for the fix to work.

 

November 27, 2018—KB4467684

 

Addresses an issue that slows server performance or causes the server to stop responding because of numerous Windows firewall rules. To enable the changes, add a new registry key “DeleteUserAppContainersOnLogoff” (DWORD) on “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy” using Regedit, and set it to 1.

 

This is the text from the KB fix.

 

https://social.technet.microsoft.com/Forums/en-US/992e86c8-2bee-4951-9461-e3d7710288e9/windows-servr-2016-rdsh-firewall-rules-created-at-every-login?forum=winserverTS

[AB_IT]

Hi All, Sorry been away for a bit as we thought we'd found a fix. It seemed that turning off all Remote Desktop Servers and turning them back on along with only 1 RDS server meant they came back at full speed again. As soon as we ran the 2 RDS Servers together they would perform incredibly slow.

 

This has been fine for the past 2 weeks but then this morning all hell has broke loose and it's back to it's old ways even if I only have 1 RDS Server powered on at a time.

 

I've disabled the VMQ's as per this thread, we didn't have Broadcomm adapters, they where Intel? I've done it anyway and it hasn't seem to have helped. I've looked into the firewall thing as well but we don't seem to have loads of weird entries on the firewall so I'm now very confused again as to what is causing it.

 

EDIT: Tried the Regedit just in case anyway... still same issue

Edited April 6, 2022 by AB_IT

[georgeescott]

Just coming up with some ideas for further troubleshooting:

 

It seems to occur when the RDS Servers have recently been restarted, they then tend to go very slow. I'll faff with them for about 2-3 hours trying to work out what's going wrong and then after a couple of restarts again they speed back up.

 

When you say they go very slow, what do you mean?

Connecting/logging in takes a while? Once connected in do apps take a long time to open? Do they become unresponsive while using them? Opening or browsing for files in File Explorer slow? All or some users affected? Is this happening using the RDS from home, does this happen connecting internally?

 

We do use User Profile Disks and I wondered if it had anything to do with these? The File Server they sit on doesn't seem to be having any problems with speed etc though. I'm a bit of a novice with UPD as I've never used them in previous companies I've worked in.

 

Use/create a local user on the RDS servers. Is this affected by slowness too? Or an AD user without UPD if possible. See if we can rule this out as an issue.

 

The RDS (CCRDS1) server seems to be writing and reading more than the others as does our File Server (CCFP) which is also where our UPD reside. We have a SAN but I've checked the SAN also and everything seems fine.

Try running https://iperf.fr/ between your computer and the RDS servers, maybe the File server where the UPD's are stored too, when it's running fine and again when RDS is slow. See if there's any difference in speed.

 

So even weirder now. Had both RDS1 and 2 turned on yesterday and running perfectly fine. Woke up this morning (nothing has changed) no shut downs or updates etc... and it's slow again.

 

Any similarities in the time of day this is happening? Any fall/rise in connected sessions? Scheduled task on the RDS/Host maybe?

 

I've disabled the VMQ's as per this thread, we didn't have Broadcom adapters, they where Intel? I've done it anyway and it hasn't seem to have helped.

 

I've heard of Intel adapters affected by slow performance that was caused by IPv4 Checksum Offload, IPv4 Large Send Offload, and Large Send Offload v2 (IPV4) before. Worth a shot at testing disabling one/all.

 

 

 

Do the RDS VM's have Dynamic Memory enabled, if so, disable it maybe?

 

Is the number of vCPU's used across all VM's <= to the total on the hosts?

 

Is the power plan set to High Performance? https://docs.microsoft.com/en-us/troubleshoot/windows-server/performance/slow-performance-when-using-power-plan.

 

Any errors in the event viewer on the gateway and RDS servers of note?

Edited April 6, 2022 by georgeescott