6 DNS Servers Crash! WTF???
I'm looking for input from the community if possible. My situation is: Monday morning I walk in to work to find that 6 of my Windows 2003 R2 servers have crashed. It's now Tuesday, lunch time and we've been able to recover all user accounts (~1500 of them) and shortly all of our user data will be back online. (I'm taking a lunch break now...)
This is where I'm hoping someone can provide guidance, cause what I don't understand is why this happened. I've been able to identify what time it all started, but not much beyond that. The common factor is that it seemed to affect all my servers that were running DNS services. (I have a slight over kill situation in that the network I just took over had 6 DNS servers on it.)
Do any of you out there have any experience with a series of servers crashing almost simultaneously? It seems to have originated with my primary Domain Controller. Is it possible that this communicated "crash" instructions to the other servers and they ran like lemmings ran off the edge of the cliff???
Given that I had to rebuild network access ASAP, I didn't get to save many logs. Does anyone have a suggestion on how I might be able to go through and identify how this happened?
Thanks for your help!
6 DNS Server Over Kill and More Tech Details
First and most importantly, thanks to all of you for your input so far. It really has been appreciated!!!
Yeah, 6 DNS servers is overkill. The guy I replaced was also running hourly, 6 hourly and daily back ups as well...I've been trying to untangle the configuration he had set up and had every intention of reducing the number of DNSssss we had. This crash revealed that our ADserver1 was not actually the primary AD serve but was the BackupServer. And, yep, they are all in the same location. There are...wait for it...21 servers in my school. Not school district, but in my school! I gleefully shut down 3 last week and have plans to shut down 7 more over the next month or two.
As for the state of the servers when we walked in Monday morning. They were all on and seemingly working. But when we went to the servers all tools listed under the Administrator Tools folder were gone. My Control Panel only had a few icons shown (no discernible pattern as to which ones were there or missing). The AD list of users was empty. Users could not logon. No IP addresses were being servered as our DHCP server was affected (it was also running DNSs). But, any server not running DNS wasn't affected which is why I was asking if it could have been something that targeted the DNS.