Server freezing

[indie]

Got a really odd problem that's only started to happen in the last 2 weeks, our domain controller freezes and locks all the clients up on the network. I can still ping it, but can't access any shares on it.

 

There's nothing at all in the error logs, I can't seem to track it down! Anyone seen anything like this before? The only way I can fix it is do a dirty power down and turn it back on.

 

Our DNS is set up OK and there's no physical network problems.

 

The server is a Dell Poweredge 1750 running Server 2003 Standard SP1, Dual Xeon 3.4 Ghz and 4GB RAM. It only serves as the domain controller and file server. DNS, DHCP etc are run on other machines.

[Netman]

Run some hardware checks on the machine...

Memory, CPU, but mostly check the drives as they're the most common culprit...

[indie]

Already had that planned for lunch time, it's the only time I can take the server offline for a while!

 

Anyone else?

[Geoff]

File replication? Memory leak? Bad drivers?

[wesleyw]

I'd say the CPU is overheating not sure why the clients would lock up unless it's a thin client network or RM

 

Whats changed in the last couple of weeks?

 

Wes

[pooley]

Check the size of your event logs - I had a server with the same probs and id forgotton to set max size to event log - security had reached over 1 gig.

[Geoff]

Anything in the event log?

[indie]

Hmm, I doubt it's overheating, the server's in an air conditioned room kept at around 18C.

 

Nothing on the hardware side has been changed, the only things that have been installed are Windows Updates, i've got two other identical (hardware) servers that have the same updates installed and they're OK.

 

I've run a few basic tests using Dells OpenManage and it's come up with nothing wrong on the hardware side. Gonne get the UBCD out at lunch time and give it a going over.

[indie]

The only error in the event log is the following from the 3rd:

 

Event Type: Warning

Event Source: NtFrs

Event Category: None

Event ID: 13562

Date: 03/01/2006

Time: 16:55:14

User: N/A

Computer: MCLC1

Description:

Following is the summary of warnings and errors encountered by File Replication Service while polling the Domain Controller mclc1.clc.mclc.org.uk for FRS replica set configuration information.

 

But then it doesn't actually list any errors Nice

[Netman]

Run some performance counters to see what the network utilisation is like and also pagefile usage...

Check anti-virus and spyware protection is ok on both server and clients... I've seen infections in the past have similar results...

[indie]

I've got Symantec Antivirus 10 running on all clients and servers with automatic updates with scheduled virus scans.

 

The gigabit interface on the server rarely rises above 3%!

[catbert]

you've got antivirus software on your dc? i was told that was bad...

 

If you disable the NAV, does it lock up then?

 

When you say freeze, can you still control the GUI? can you remote control into it?

 

hardware would seem logical place to look, like you said, get the UBCD and do a memtest.

 

any dust on the motherboard or RAM, that has locked a few workstations in my past job.

 

also when you say automatic updates, do you also mean you update via the MS site? as updating the graphic drives has also locked up a few pc's in the past.

 

don't rule out recent fixes as causing your problem. what got installed in the past 2 weeks via windows update? despite the need to update, you should wait a few days to see if anyone reports problems with the hotfixes in the MS forums. installing untested stuff on a live server is never a good thing.

 

do you auto download and manual install, i have known of problems where a hotfix didn't install right because the server was doing something else at the time when an auto install went. servers should really be maually updated.

 

how many times have you "safe shutdown" the server since 2 weeks ago? the hotfixes might of needed a proper reboot to initilise properly.

 

Alan

[indie]

Yes, we run SAV Corporate on our DC, always have done and it's been fine for years!

 

When it freezes, I can't do anything with the start bar, I can still move icons around the desktop, but trying to bring up the task manager took about 10 minutes. There's no major disk activity when it freezes.

 

Not tried to remote into the machine when it goes, I will next time though!

 

I'm keeping the task manager open for the next time it crashes to see if there's anything eating up memory etc.

 

I say automatic updates, what i mean is WSUS I make sure WSUS only downloads OS and Application updates, never any drivers. I alos install the updates manually on the servers.

 

I did a proper reboot this morning, then a couple of hours later it froze.

 

There's no pattern to the problem either!

[catbert]

Whats the update pattern for SAV?

 

try running the server with sav disabled.

 

Alan

[paul]

Its not by any chance a print server to?

[indie]

Updates are applies at 12PM, Monday to Friday, if any clients miss that slot, it retries for 2 hours, then waits till the next day.

 

Yes, it is a print server, the spools are stored on a different drive to the OS.

[paul]

I had this same problem last week, found a new print driver installed by WSUS was causing a memory leak - reverted to old 1 now no problem.

[indie]

I don't let WSUS download drivers, not changed anything to do with Printers in ages either!

 

I'm keeping an eye out for mem leaks, gonna take the server offline in a few minutes and kick it about a bit!

[indie]

Bingo, right i think i've found it!

 

Memtest has come up with a shed load of errors, gonna get onto Dell now

[indie]

Mind, saying that, it's coming up with errors on another 1750 server that i'm not having problems with!

 

Does Memtest give you false positives?

[Geoff]

No. You get false negatives sometimes because of cacheline interations hiding memory faults. Its best to disable your L1 and L2 cache's because of that when doing memtests (unless your specifically testing your L1/L2 cache memory).

 

As this is a server I suggest a restore from a known good backup once the faulty memory has been replaced. You have no idea which files have been corrupted by the memory errors and written to disk with errors.

 

Regarding the second server that also has memory faults, was it bought at the same time? Maybe Dell had a bad batch of ram? Do you have any other Dell machines that were purchased at the same time?

[indie]

Hmm, it looks like it's a problem with Memtest not knowing the server chipsets, it's coming up with errors on all 4 of our 1750's, 2 of which were bought at different times.

 

It can't detect the chipset and our servers use ECC RAM, that's why I think it's coming up with the errors.

[Geoff]

It might be the memory is faulty and the ECC ram is correcting the fault most of the time (that is what ECC RAM is for afterall).

 

What chipset is it?

[indie]

It's a ServerWorks® GC LE chipset, apparently!

[Geoff]

Suffers from some interesting MTRR bugs. Memtest wont be reliable.