Strange Behaviour - Insufficient system resources?

[FN-GM]

Hello,

 

I have a strange issue that is impacting 2 2016 servers. I was hoping someone could advise.

 

Issues we are seeing

• Very slow boot time, around 30 minutes being stuck on the Windows splash screen - The SAN it lives on is full of SSD's.
  
• Search doesn't work.
  
• Windows Updates do not install and fail.
  
• After being booted for a few days you cannot login with an error that the user profile service not running (although it is). A reboot clears it for a few days.
  

 

Any command issued using PSexec from a remote computer results in "Insufficient system resources exist to complete the requested service". Also install an industry specific application fails and throws the same error in it's installation log.

 

LAPS won't install "The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2908."

 

The server doesn't have any resource issues, plenty of disk space, CPU and RAM.

 

The event log doesn't really show anything to note.

 

I was wondering if anyone had any ideas please?

 

Luckily the server is in a test environment so it's not causing too much disruption (only to testing). But it's going to a pain to simply rebuild it.

 

Thanks

Edited September 27, 2022 by FN-GM

[robyholmes]

If it's a VM can you move it to local storage on a host and see if it's the same?

[dmj]

dodgy HBA perhaps? I'd run a disk speed analysis and start looking there.

[FN-GM]

Thanks for the replies.

 

I have shifted the VM onto a completely different cluster on a completely different SAN at the other end of the city. Still no change.

 

We have over 200 servers, I should image we would see bigger issues if there were issues with the storage.

 

The 2 servers were built at the same time, it makes me steer more towards a software issue.

Edited September 27, 2022 by FN-GM

[dmj]

If it's a software issue, blitz it and start again - it's IasC right

[psydii]

A Windows update/ Anti Virus update gone bad? Damaged WMI Repository? Malware giving you a merry chase?

 

Yeah, if you are running 200 servers, they're animals on a farm, not individual pets. It's in the test environment, blow it away and start again.

 

You can keep the vm around in a sandbox environment for when you have time to go deep with troubleshooting, but you might find there are better ways of spending your time. Just be mindful there is a chance it has malware/RAT/persistence so be careful.

[FN-GM]

blow it away and start again.

 

It's not quite so simple. The server is running very specialist software, we are 1 of 5 in the country using it. Then the server will have to built to replicate (as much as possible) the live environment.

 

I hadn't looked at it today, but I will look further using some of these suggestions.

 

Thanks

[psydii]

It's not quite so simple. The server is running very specialist software, we are 1 of 5 in the country using it. Then the server will have to built to replicate (as much as possible) the live environment.

 

I hadn't looked at it today, but I will look further using some of these suggestions.

 

Thanks

 

Fair enough. These things can be finicky. Perhaps (if you haven't already) it might help to reframe the analysis: How you would handle the same problem if it were in production?

[arwen]

If you are connecting via RDP then there was a thing a while back about firewall rules being duplicated each time you tried to log on that slowed log in times drastically. I'm sure if you google you can find the script that fixes it. Might also be worth checking there isn't confliced in any group polices being applied.

 

ALso, how's it's resource use? Does the CPU and/or RAM creep up over time untill it is maxed out ? Possibly dodgy or corrupt proccesses running if that's the case.

[FN-GM]

Thanks for the replies. I haven't had too much time to look into this. However, I ran the below to reset the WMI repository. It fixed the search but everything else remains present.

 

Winmgmt /resetrepository

 

I will pick it up again next week.

 

Thanks

[FN-GM]

If you are connecting via RDP then there was a thing a while back about firewall rules being duplicated each time you tried to log on that slowed log in times drastically. I'm sure if you google you can find the script that fixes it. Might also be worth checking there isn't confliced in any group polices being applied.

 

ALso, how's it's resource use? Does the CPU and/or RAM creep up over time untill it is maxed out ? Possibly dodgy or corrupt proccesses running if that's the case.

 

The resources appear fine in the task manager, in addition our monitoring application doesn’t pickup any issues. It isn’t tanking the CPU or RAM. The server is slow booting on the Windows logo splash screen, when it has actually booted its fine and doesn’t feel sluggish.

 

 

Does anyone else have any thoughts please?

 

Thanks.

[FN-GM]

A Windows update/ Anti Virus update gone bad? Damaged WMI Repository? Malware giving you a merry chase?

 

 

The security team are confident there isn’t any malware etc on the server.

 

I rebuilt the WMI repository but it didn’t help.

 

Thanks.

[Oaktech]

can you snapshot and try an in place upgrade to 19?

 

I've had a similar issue with a 16 print server running Uniflow after a few weeks uptime, certain things (like releasing jobs) just grind to a halt, reboots take ages, windows updates not working, slow to log in - sits preparing windows for far longer than you'd expect for a server with the resources it has... Did an in place upgrade to 19 and touch wood we've had no issues since July.

[psydii]

If an OS upgrade is off the table, and you have to get to the bottom of it, make friends with this: https://learn.microsoft.com/en-us/windows-hardware/test/wpt/

It will collect enough telemetry to allow you to see exactly where the system is slowing down, and what it is likely waiting for. This might then point to what is causing the underlying problem.

 

Also... if you don't have the problem in production, but do in the test environment check to see what the difference between settings are in both environments. This trick may help in that regard:

[tosh74]

clutching as straws here - but is your .Net healthy?

[Duke5A]

It was recommended earlier, but I don't think you did it. Move the VM to local storage and give it another run. Preferably on an isolated hypervisor with local storage. I know you moved it to another SAN, but this might have something to do with the underlying storage tech. Is it iSCSI?

[FN-GM]

It was recommended earlier, but I don't think you did it. Move the VM to local storage and give it another run. Preferably on an isolated hypervisor with local storage. I know you moved it to another SAN, but this might have something to do with the underlying storage tech. Is it iSCSI?

 

We don’t have any local storage. If there was a storage issue wouldn’t we see it on more virtual machines? We have hundreds. It would be very apparent.

[FN-GM]

can you snapshot and try an in place upgrade to 19?

 

I've had a similar issue with a 16 print server running Uniflow after a few weeks uptime, certain things (like releasing jobs) just grind to a halt, reboots take ages, windows updates not working, slow to log in - sits preparing windows for far longer than you'd expect for a server with the resources it has... Did an in place upgrade to 19 and touch wood we've had no issues since July.

 

I’m haven’t done an in place upgrade for many years. Can you mount the ISO and boot from it or do you have to run the .exe in windows please?

[bknaggs]

I’m haven’t done an in place upgrade for many years. Can you mount the ISO and boot from it or do you have to run the .exe in windows please?

 

Run from within Windows.

[FN-GM]

Run from within Windows.

 

I doubt it will work then. I can’t run other installers and executables.