How to troubleshoot non-responsive server 2003...

[kennysarmy]

Our main DC running server 2003 locked up on Monday and we rebooted it.

 

Today I've had to reboot it twice and it's gone very slowly for periods of up to 4/5 minutes on three other occasions.

 

I've put some performance counters in place to log CPU usage but nothing showing like the culprit...

 

During one lock-up I pulled the network cables twice to see if it was a networking issue and this made no difference.

 

The problem is when it locks up / runs slowly it's impossible to login and find out what's going on....

 

Any suggestions anyone?

[CHR1S]

Event Viewer -> Connect to another computer, during the slow period.. worth a try.

 

Whats the spec of the server?

[cheesypete]

Would suggest to start with

 

(a) see whats running in task manager (even better if you can do it at times of slowdown) - see whats hogging the CPU / RAM

(b) check your event viewer logs

 

Also recommend getting hold of Sysinternals Process Explorer (from M$) which gives more detail as to what may be running

 

Apart from it being a DC, is it running anything else? Do you run apps? Does it handle print queues?

[kennysarmy]

when it goes slow you cannot remote log in to look at event viewer.

have checked event viewer and certainly nothing obvious in terms of error events prior to lockups.

 

server is quite a top spec HP model - not sure of exact - but it's only 3 months old....

 

it does nt run any apps or print queues.

 

it does contain the users desktops and staff data...

it runs DHCP, DNS, WINS

 

I just rebooted it and am watching task manager now:

 

If I suspect a hard disk issue what are the best performance counters to put on?

[kmount]

Run a chkdsk overnight (outside of windows one) and see what it throws up, or check whether it needs defragged.

 

It's amazing how much performance can be lost from delayed writes.

[CHR1S]

Server 2k3 auto defrags on idle right?

 

Are the slowdowns linked with classes logging on and off? Could be a bottleneck. What Drives are they and are they in RAID?

[kmount]

Auto Defrag is a touchy subject for me, I just find it's not as effective as kicking everyone off the server and running a full defrag repeatedly to "fix" things up properly.

[kennysarmy]

Server 2k3 auto defrags on idle right?

 

Are the slowdowns linked with classes logging on and off? Could be a bottleneck. What Drives are they and are they in RAID?

 

No they don't coincide with classes ending.

 

Monday incident was 11.45 - 20 mins in to lesson.

 

Incidents today were also at arbitary times in to lesson times.

 

C:\ mirrored

D:\ raid 5

 

all been OK since classes ended an hour ago so it may only happen again under load tomorrow...

 

Will try the Sysinternals Process Explorer tool tomorrow.

[steve]

Can you ping the server when it locks up?

 

ping \\servername

ping IP

 

It maybe the name isn't resolving (DNS issue)

 

Have you got anything that does LDAP lookups against the server?

 

I had a problem with smoothwall pounding our server for user authentication, caused similar issues.

[CHR1S]

To add to Steve's post if you find a ping issue, try packet sniffing and look for any unusual or excessive traffic to the server IP.

[kennysarmy]

Can you ping the server when it locks up?

 

ping \\servername

ping IP

 

It maybe the name isn't resolving (DNS issue)

 

Have you got anything that does LDAP lookups against the server?

 

I had a problem with smoothwall pounding our server for user authentication, caused similar issues.

 

one of the first things I tried

 

pings are fine and good even when server locks up.

 

just had to reboot it again - was watching CPU useage via remote desktop session and was 0-1% even though phone went red hot with users unable to logon save work access network shares etc.

 

must be disk or memory issue...

 

but how to prove troubleshoot i am not sure...

with nothing in the event logs....

[DMcCoy]

Is it x64? What antivirus (if any)? Any quota management on user folders? Any classes using Movie maker?

[kennysarmy]

It's an HP ProLiant DL380 running Server 2003 4gb RAM...

 

AntiVirus is CA's etrust - which is running on all the other servers fine. Another server is the AV master if you like that sends updates to the other servers/clients.

 

This server runs realtime monitoring but the SYSVOL folder has been taken out of the that....

 

no quota management.

 

don't think movie maker is heavily used by the clients and anyway students store their work on another file server not this one.

 

[CHR1S]

Which RAID card is it, the top spec 512Mb version? and are they SATA or SAS Drives?

Check how big the page file is, is it too small?

Try turning off AV scanning for a while and see what happens.

 

Can you list the servers roles aswell please?

[Geoff]

Also try using sysinternals process explorer.

 

Process Explorer

 

Make sure you add the IO Total Delta and Working Set Size columns though, so you can see memory and i/o performance per process.

[kennysarmy]

RAID is - Smart Array P400 Controller in Slot 1

 

Drives are all SAS and HP DG146BB976

 

damn thing just had to be switched off again...

 

reset page file to be managed by system to see if that helps..

[SYNACK]

How about using the performance monitor application and setting it up to monitor, memory, cpu usage, hard drive and network queues along with other metrics that would be helpful and then leaving it running. This will log what the system is doing continuously in the background and give you a nice little graph that you can use to diagnose.

 

You can evern schedule the logging start and stop times if you need to.

[kennysarmy]

GPO was re-directing users desktops to folders on this server, I've moved this over to the other DC to try and mitigate the effect of it locking up.

 

Will consider in half-term also moving off the staff and other data temporarily to another server too.

 

Then as you say it's set-up some monitoring to see what is going on...

 

anyone used poolmon.exe before?