Serious Issue with unresponsive servers!

[Zoom7000]

We have a serious issue with 3 servers (2 DCs and 1 Exchange Server) where the server becomes totally unresponsive. Users can't log on and generally everything starts to fail. When we visit the server, the login box has disappeared and the only thing we can do is a hard reset, this issue has been going on for nearly 6 months now, it started with 1 server and has now affected 3!

 

It happens at almost random intervals and SLT are starting to get seriously annoyed and we have no idea what the issue is. In addition to this, Ofsted is due soon, probably next month, so... I think that explains that!

 

The 2 domain controllers are 4 year old IBM eServer MT-M 8670-L1X. Specs:

Dual Intel Xeon 2.80GHz

4Gb RAM (3.81Gb Windows)

Windows Server 2003 R2 x86

 

The Exchange Server - HP Proliant DL380 G5 (Will have to confirm this, but it's something similar!)

Dual Intel Xeon E5320 1.86GHz

6Gb RAM

Windows Server 2003 R2 x64

 

At first we thought the 2 DCs would need replacing, however, the mail server is brand new and giving the exact same issue.

 

Does anyone have any ideas what could be the problem? We are desperate!

Edited September 24, 2008 by Zoom7000

[Michael]

It definitely sounds like some sort of corruption to me, affecting critical services DNS/DHCP possibly even AD itself. I would highly recommend you check the event logs for any clues. I would also (if possible) run Scandisk (especially on the system drive) to check for errors.

 

How long has this problem been going on for? What Service Pack are you running on your servers? What AV software do you use? Have you scheduled an over night scan?

[Zoom7000]

(Updated the first post!)

 

Service Packs: SP2 on all servers, latest updates through WSUS.

Anti-Virus: Sophos (LGFL Agreement) Scans every morning at 5am.

 

I can't make sense of anything really bad in Event Viewer. Can I export the logs somehow?

[Jamman960]

The event logs should give clues as to whats going on, also check for out of place processes within taskman or possibly even use process explorer from sysinternals to take a look.

 

You can export the logs, within even viewer select the log then right click and select save log file as.

[bizzel]

Sounds like something bad is being replicated around AD if the new mail server started doing it.

[Newton]

I've had servers do the exact same, when they have run out of space on the system drive, but you normally get items in event logs for this,

 

Could be a bad Active Directory? ~Beaten to the post by Bizzel

Edited September 24, 2008 by Newton
type to slowly

[FN-GM]

How many global catalogues do you have?

[Zoom7000]

I forgot to mention the Exchange server is on a totally different domain from the 2 DCs. (2 DCs is Curriculum and Mail Server is Admin - Both separate Forests)

[cookie_monster]

When the problem occurs does it affect all boxes at the same time?

 

Have you tried running DCDIAG and NETDIAG on all of the boxes?

[FN-GM]

is there a trust between them?

 

thanks

[Geoff]

Here's a KB about the login box disappearing.

 

No Windows or Network Logon Dialog Box at Startup

[Zoom7000]

When the problem occurs does it affect all boxes at the same time?

 

Have you tried running DCDIAG and NETDIAG on all of the boxes?

 

Not tried those, but no, it doesn't affect all 3 at the same time.

 

is there a trust between them?

 

thanks

 

Yes there is and I've asked the NM, he thinks there are 2 Global Catalogues.

[Zoom7000]

I've got the Event Viewer logs from 1 of the DCs if anyone is interested in having a look, but zipped it's 11Mb so might have to be a email or FTP job!

[cookie_monster]

I'd try running them and viewing the output then. I'd also run rootkit revealer and concider running a scan with different AV if you can or even trying temporarily disabling your current product.

[Zoom7000]

Thanks for your help guys. I'll give these things a try tomorrow, Right now I've gotta run home. Daughter has thrown up nearly 10 times today. Just hope she gets better before the evening. I was supposed to bring her in to school tomorrow for an event and everyone is so looking forward to seeing her! Moral dilemmas! Professional dilemmas! Argh!

[cookie_monster]

Not sure if you're familiar with the utils so here's a couple of good pages.

 

DCDiag Tutorial Examples. Check your Windows Active Directory (+ Free Download)

 

NetDiag Tutorial. Check Server Network Connections. Free Download Support tool

[robknowles]

Hello,

 

Do the servers use Intel Network Cards? Did you update the drivers before it started?

 

Rob

[DMcCoy]

Apart from something common (like sophos) an idea did occur to me.

 

You don't have any of the event logs set to manual clear do you? BAD things happen when an event log fills up.

[Teth]

It will really come down to even logs but I'd pay special attention to DNS and AD replication service message.

 

We had a similar issue a while ago because one of your DCs which is also the print server was running out of non-paged RAM due to a memory leak in a HP printer driver. When the system got low on non-paged RAM the DNS service was the first to fall over which in turn took down logins and eventually the whole network.

[Zoom7000]

Got an external support contracter looking at the problem today, he thinks it's DNS but I will update the thread when we know more.