Slooooooow logons

[howartp]

Hello,

 

We've had this issue for at least three years, but never found the root cause so I'm seeing if anyone happens to have a magic wand lying around.

 

Basically sometime in June, usually earlier rather than later, we will find that our two secondary DCs start hitting 100% CPU and everyone on those servers struggles to login at the start of lessons. 15 minutes into the lesson and everything will generally settle down until the next lesson.

 

This might go on for a few days, despite rebooting all DCs, then disappear again.

 

We only ever get this in June time, possibly into July, then when kids come back in September it is never a problem.

 

It usually affects our PC-based language exams, so we moved their user accounts onto the FRDC which isn't usually affected, but the problem only started today this year.

 

I can't think of any date-based / season-based reason for this? I had a theory at one point that it was Art using loads of large PSDs in whichever module they teach at this time of the year, but I've discounted that.

 

Any suggestions, however daft they sound, welcomed!

 

Peter

[SYNACK]

A long winded approach but getting a wireshark packet capture and slowly working through it would show you what was happening at the network level and may shed some light.

[themightymrp]

I assume theres no scheduled task hidden away on those servers? Or a strange full anti-virus system scan with a schedule?

[FN-GM]

If your on 2003 switch to a newer version that has more resources etc. I bet a few quid you will notice its fine then. Thanks

[Grey-gear]

Does this problem only happen to students or is it for all users?

 

I do think @SYNACK idea is were you will need to start, but as a guess as this has happens last few years around about this time do you have a number of the staff stream Wimbledon?

[howartp]

A long winded approach but getting a wireshark packet capture and slowly working through it would show you what was happening at the network level and may shed some light.

Yeah, I wish I knew how to use it - I can get the capture, but reading what it tells me is another question.

 

I assume theres no scheduled task hidden away on those servers? Or a strange full anti-virus system scan with a schedule?

AV scans are all configured by Sophos Enterprise Console. The only task I didn't know about on there is the RM Server Backup which hasn't run since 2006 cos we don't use it any more

 

If your on 2003 switch to a newer version that has more resources etc. I bet a few quid you will notice its fine then. Thanks

As it happens, we've got a brand new domain and servers lined up on our VMs ready to switch at summer - to hopefully get rid of ancient gremlins which have upgraded from NT thru 2003 - but that doesn't help my students at this moment, and I'd like to solve the problem anyway!

 

Does this problem only happen to students or is it for all users?

Anyone on those servers.

 

I do think @SYNACK idea is were you will need to start, but as a guess as this has happens last few years around about this time do you have a number of the staff stream Wimbledon?

Hehe, good point - but no, the internet is fine.

 

Peter

[mikeyd101]

Group Policies? I noticed our login times are extending as we add more GPOs, nothing that hammers our DCs tho.

 

I'd break out prefmon as well as wireshark and find out what is consuming the resources. Also event log is worth trawling during those times.

[Grey-gear]

What are the tasks that are making the DC CPU hit 100%?

[psydii]

Or give MS PSS a call and for £200 they will find it for you.

[SYNACK]

The idea with wireshark would be to run a summary on the connections to see if there are a lot of a certain type of connection from a certain client or clients at the time of the issue. You would probably want to set up a mirror port on your switch to mirror the traffic from your server to a capture pc so as not to put more load on the server and run performance counter logging on the server to give you a graph of when it was maxing.

 

This is another bit of software that has a bunch of summarizing features that may help that takes the place of wireshark Packetyzer | Free software downloads at SourceForge.net

 

You could also look for other TCP type issues by following this:

 

The packet dump will be HUGE over the course of a day so will take some serious time to run summaries on and go through so make sure your capture machine is beefy with a stack of free space to store all of the inbound and outbound traffic your server deals with during the capture.

 

 

I usually use Wireshark/packetyzer/microsoft network monitor as a last resort because it is such a huge amount of data but it does shine a light on exactly what is happening on the network (assuming the issue is on the network).