6 DNS Servers Crash! WTF???

[byron67]

Hey Community,
I'm looking for input from the community if possible. My situation is: Monday morning I walk in to work to find that 6 of my Windows 2003 R2 servers have crashed. It's now Tuesday, lunch time and we've been able to recover all user accounts (~1500 of them) and shortly all of our user data will be back online. (I'm taking a lunch break now...)

This is where I'm hoping someone can provide guidance, cause what I don't understand is why this happened. I've been able to identify what time it all started, but not much beyond that. The common factor is that it seemed to affect all my servers that were running DNS services. (I have a slight over kill situation in that the network I just took over had 6 DNS servers on it.)

Do any of you out there have any experience with a series of servers crashing almost simultaneously? It seems to have originated with my primary Domain Controller. Is it possible that this communicated "crash" instructions to the other servers and they ran like lemmings ran off the edge of the cliff???

Given that I had to rebuild network access ASAP, I didn't get to save many logs. Does anyone have a suggestion on how I might be able to go through and identify how this happened?

Thanks for your help!
Sean

[Dos_Box]

The only thing that I know would do that if is the DNS settings themselves were incorrectly setup. This would populate problems across many servers.
Ensure that the DNS settings on each server point only to themselves and that the forwarders are correctly setup to point to external DNS servers as a starting point.

[eejit]

Ensure that the DNS settings on each server point only to themselves and that the forwarders are correctly setup to point to external DNS servers as a starting point.

Are you sure about the DNS servers only pointing at themselves? I have two DNS servers and in the TCP/IP properties of the LAN NIC on both of them I point to itself as the "preferred DNS server" and to the other DNS server as the "Alternate DNSserver". Is that not right?

[DMcCoy]

Are you sure about the DNS servers only pointing at themselves? I have two DNS servers and in the TCP/IP properties of the LAN NIC on both of them I point to itself as the "preferred DNS server" and to the other DNS server as the "Alternate DNSserver". Is that not right?

That should be fine, I do the same here. It's a great help when you restart the DCs individually as they can look at the other one for dns while their own is still starting up.

[DMcCoy]

As to the crashing, it could be automatic updates, but Windows 2003 also has some denial of service vulnerabilities with its dns service.

[Dos_Box]

The alternates (as long as they are part of the same domain) should be picked up from the name servers tab in domain controllers DNS settings. There isn't really a requirement for the domain controller to have them listed in the NIC properties.
I've just remembered another thing that will stuff a domains DNS servers is that if a domain controller has had its date and time adjusted (for whatever reason) then when replication occurs it kills AD due to the date change. I last saw this when some bright spark decided to roll back their DCs clock in an attempt to get some trial software to continue working! Their network was killed stone dead. A complete reinstall was required.

[eejit]

The alternates (as long as they are part of the same domain) should be picked up from the name servers tab in domain controllers DNS settings. There isn't really a requirement for the domain controller to have them listed in the NIC properties.

Ah! You're assuming load balancing - I'm assuming crash

I have the alternate in for the time when the local DNS is not available.

[byron67]

As to the crashing, it could be automatic updates, but Windows 2003 also has some denial of service vulnerabilities with its dns service.

Thanks for your input. Maybe it's paranoia on my part, but I am worried that my network was "intruded" over the weekend. Do you know where might I look for evidence of a DOS attack?
Thanks,
S

[Ric_]

Thanks for your input. Maybe it's paranoia on my part, but I am worried that my network was "intruded" over the weekend. Do you know where might I look for evidence of a DOS attack?
Thanks,
S

A good place to start would be the logs on your firewall - you need to look for anything out of the ordinary (particularly on port 53).

[CHR1S]

Can you be more specific with your crash wording? Were they off? Were they hung? Were they fine but had no network connections?
Also why did you have to recover the user accounts? Were the drives wiped?
Im sure you get the idea?

[Michael]

6 x DNS servers does sound overkill to be honest, unless the network is absolutely massive. I'm talking thousands and thousands of workstations.

The fact you've had to restore 1500 users (presumably you're using AD) indicates the problem could be something else. Out of curiosity, are the six servers in the same physical location?

[FN-GM]

It is a bit of an overkill, 6 DNS servers? It Could be down to a bug on one server and its replicated to the others. We have had that where our parent domain passed down a fault. The way i look at it is the more servers you have running DNS the more likely you will have a problem.

[byron67]

First and most importantly, thanks to all of you for your input so far. It really has been appreciated!!!

Yeah, 6 DNS servers is overkill. The guy I replaced was also running hourly, 6 hourly and daily back ups as well...I've been trying to untangle the configuration he had set up and had every intention of reducing the number of DNSssss we had. This crash revealed that our ADserver1 was not actually the primary AD serve but was the BackupServer. And, yep, they are all in the same location. There are...wait for it...21 servers in my school. Not school district, but in my school! I gleefully shut down 3 last week and have plans to shut down 7 more over the next month or two.

As for the state of the servers when we walked in Monday morning. They were all on and seemingly working. But when we went to the servers all tools listed under the Administrator Tools folder were gone. My Control Panel only had a few icons shown (no discernible pattern as to which ones were there or missing). The AD list of users was empty. Users could not logon. No IP addresses were being servered as our DHCP server was affected (it was also running DNSs). But, any server not running DNS wasn't affected which is why I was asking if it could have been something that targeted the DNS.

[Dos_Box]

21 Servers!!! It sounds like each server had a specific role, instead of putting several roles on one server each had its own specific task. Time for some virtualization if need be I think.
The symptoms you describe do indeed sound like DNS issues, but for a single school you probably need 3 DNS servers max, and only 2 of them as domain controllers. How many workstations do you have BTW?
As a starting point I would first of all get all of the FSMO roles back in one place (check they are working correctly to begin with - http://support.microsoft.com/kb/324801) remove DNS on all but the domain controllers and then check that replication occurs correctly. DNS problems are very awkward and take quite a while to sort out and troubleshoot. Its a shame you are on the other side of the world to me as I could talk you through it using our messaging client (not yet live for mainstream usage).