+ Post New Thread
Results 1 to 14 of 14
Windows Thread, 6 DNS Servers Crash! WTF??? in Technical; Hey Community, I'm looking for input from the community if possible. My situation is: Monday morning I walk in to ...
  1. #1
    byron67's Avatar
    Join Date
    Nov 2007
    Location
    Hong Kong
    Posts
    12
    Thank Post
    3
    Thanked 1 Time in 1 Post
    Rep Power
    0

    Angry 6 DNS Servers Crash! WTF???

    Hey Community,
    I'm looking for input from the community if possible. My situation is: Monday morning I walk in to work to find that 6 of my Windows 2003 R2 servers have crashed. It's now Tuesday, lunch time and we've been able to recover all user accounts (~1500 of them) and shortly all of our user data will be back online. (I'm taking a lunch break now...)

    This is where I'm hoping someone can provide guidance, cause what I don't understand is why this happened. I've been able to identify what time it all started, but not much beyond that. The common factor is that it seemed to affect all my servers that were running DNS services. (I have a slight over kill situation in that the network I just took over had 6 DNS servers on it.)

    Do any of you out there have any experience with a series of servers crashing almost simultaneously? It seems to have originated with my primary Domain Controller. Is it possible that this communicated "crash" instructions to the other servers and they ran like lemmings ran off the edge of the cliff???

    Given that I had to rebuild network access ASAP, I didn't get to save many logs. Does anyone have a suggestion on how I might be able to go through and identify how this happened?

    Thanks for your help!
    Sean

  2. #2

    Dos_Box's Avatar
    Join Date
    Jun 2005
    Location
    Preston, Lancashire
    Posts
    10,467
    Thank Post
    606
    Thanked 2,192 Times in 1,006 Posts
    Blog Entries
    23
    Rep Power
    633
    The only thing that I know would do that if is the DNS settings themselves were incorrectly setup. This would populate problems across many servers.
    Ensure that the DNS settings on each server point only to themselves and that the forwarders are correctly setup to point to external DNS servers as a starting point.

  3. #3
    eejit's Avatar
    Join Date
    Jun 2005
    Location
    Ireland
    Posts
    606
    Thank Post
    52
    Thanked 12 Times in 12 Posts
    Rep Power
    22
    Quote Originally Posted by Dos_Box View Post
    Ensure that the DNS settings on each server point only to themselves and that the forwarders are correctly setup to point to external DNS servers as a starting point.
    Are you sure about the DNS servers only pointing at themselves? I have two DNS servers and in the TCP/IP properties of the LAN NIC on both of them I point to itself as the "preferred DNS server" and to the other DNS server as the "Alternate DNSserver". Is that not right?

  4. #4
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,462
    Thank Post
    10
    Thanked 496 Times in 436 Posts
    Rep Power
    113
    Quote Originally Posted by eejit View Post
    Are you sure about the DNS servers only pointing at themselves? I have two DNS servers and in the TCP/IP properties of the LAN NIC on both of them I point to itself as the "preferred DNS server" and to the other DNS server as the "Alternate DNSserver". Is that not right?
    That should be fine, I do the same here. It's a great help when you restart the DCs individually as they can look at the other one for dns while their own is still starting up.

  5. Thanks to DMcCoy from:

    FN-GM (19th February 2008)

  6. #5
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,462
    Thank Post
    10
    Thanked 496 Times in 436 Posts
    Rep Power
    113
    As to the crashing, it could be automatic updates, but Windows 2003 also has some denial of service vulnerabilities with its dns service.

  7. #6

    Dos_Box's Avatar
    Join Date
    Jun 2005
    Location
    Preston, Lancashire
    Posts
    10,467
    Thank Post
    606
    Thanked 2,192 Times in 1,006 Posts
    Blog Entries
    23
    Rep Power
    633
    The alternates (as long as they are part of the same domain) should be picked up from the name servers tab in domain controllers DNS settings. There isn't really a requirement for the domain controller to have them listed in the NIC properties.
    I've just remembered another thing that will stuff a domains DNS servers is that if a domain controller has had its date and time adjusted (for whatever reason) then when replication occurs it kills AD due to the date change. I last saw this when some bright spark decided to roll back their DCs clock in an attempt to get some trial software to continue working! Their network was killed stone dead. A complete reinstall was required.

  8. 2 Thanks to Dos_Box:

    byron67 (19th February 2008), FN-GM (19th February 2008)

  9. #7
    eejit's Avatar
    Join Date
    Jun 2005
    Location
    Ireland
    Posts
    606
    Thank Post
    52
    Thanked 12 Times in 12 Posts
    Rep Power
    22
    Quote Originally Posted by Dos_Box View Post
    The alternates (as long as they are part of the same domain) should be picked up from the name servers tab in domain controllers DNS settings. There isn't really a requirement for the domain controller to have them listed in the NIC properties.
    Ah! You're assuming load balancing - I'm assuming crash

    I have the alternate in for the time when the local DNS is not available.

  10. #8
    byron67's Avatar
    Join Date
    Nov 2007
    Location
    Hong Kong
    Posts
    12
    Thank Post
    3
    Thanked 1 Time in 1 Post
    Rep Power
    0
    Quote Originally Posted by DMcCoy View Post
    As to the crashing, it could be automatic updates, but Windows 2003 also has some denial of service vulnerabilities with its dns service.
    Thanks for your input. Maybe it's paranoia on my part, but I am worried that my network was "intruded" over the weekend. Do you know where might I look for evidence of a DOS attack?
    Thanks,
    S

  11. #9

    Ric_'s Avatar
    Join Date
    Jun 2005
    Location
    London
    Posts
    7,599
    Thank Post
    109
    Thanked 765 Times in 596 Posts
    Rep Power
    181
    Quote Originally Posted by byron67 View Post
    Thanks for your input. Maybe it's paranoia on my part, but I am worried that my network was "intruded" over the weekend. Do you know where might I look for evidence of a DOS attack?
    Thanks,
    S
    A good place to start would be the logs on your firewall - you need to look for anything out of the ordinary (particularly on port 53).

  12. Thanks to Ric_ from:

    byron67 (19th February 2008)

  13. #10

    CHR1S's Avatar
    Join Date
    Feb 2006
    Location
    Birmingham
    Posts
    4,505
    Thank Post
    1,585
    Thanked 486 Times in 304 Posts
    Rep Power
    217
    Can you be more specific with your crash wording? Were they off? Were they hung? Were they fine but had no network connections?
    Also why did you have to recover the user accounts? Were the drives wiped?
    Im sure you get the idea?

  14. #11

    Michael's Avatar
    Join Date
    Dec 2005
    Location
    Birmingham
    Posts
    9,262
    Thank Post
    242
    Thanked 1,572 Times in 1,252 Posts
    Rep Power
    340
    6 x DNS servers does sound overkill to be honest, unless the network is absolutely massive. I'm talking thousands and thousands of workstations.

    The fact you've had to restore 1500 users (presumably you're using AD) indicates the problem could be something else. Out of curiosity, are the six servers in the same physical location?

  15. #12

    FN-GM's Avatar
    Join Date
    Jun 2007
    Location
    UK
    Posts
    16,038
    Thank Post
    888
    Thanked 1,727 Times in 1,490 Posts
    Blog Entries
    12
    Rep Power
    453
    It is a bit of an overkill, 6 DNS servers? It Could be down to a bug on one server and its replicated to the others. We have had that where our parent domain passed down a fault. The way i look at it is the more servers you have running DNS the more likely you will have a problem.

  16. #13
    byron67's Avatar
    Join Date
    Nov 2007
    Location
    Hong Kong
    Posts
    12
    Thank Post
    3
    Thanked 1 Time in 1 Post
    Rep Power
    0

    6 DNS Server Over Kill and More Tech Details

    First and most importantly, thanks to all of you for your input so far. It really has been appreciated!!!

    Yeah, 6 DNS servers is overkill. The guy I replaced was also running hourly, 6 hourly and daily back ups as well...I've been trying to untangle the configuration he had set up and had every intention of reducing the number of DNSssss we had. This crash revealed that our ADserver1 was not actually the primary AD serve but was the BackupServer. And, yep, they are all in the same location. There are...wait for it...21 servers in my school. Not school district, but in my school! I gleefully shut down 3 last week and have plans to shut down 7 more over the next month or two.

    As for the state of the servers when we walked in Monday morning. They were all on and seemingly working. But when we went to the servers all tools listed under the Administrator Tools folder were gone. My Control Panel only had a few icons shown (no discernible pattern as to which ones were there or missing). The AD list of users was empty. Users could not logon. No IP addresses were being servered as our DHCP server was affected (it was also running DNSs). But, any server not running DNS wasn't affected which is why I was asking if it could have been something that targeted the DNS.

  17. #14

    Dos_Box's Avatar
    Join Date
    Jun 2005
    Location
    Preston, Lancashire
    Posts
    10,467
    Thank Post
    606
    Thanked 2,192 Times in 1,006 Posts
    Blog Entries
    23
    Rep Power
    633
    21 Servers!!! It sounds like each server had a specific role, instead of putting several roles on one server each had its own specific task. Time for some virtualization if need be I think.
    The symptoms you describe do indeed sound like DNS issues, but for a single school you probably need 3 DNS servers max, and only 2 of them as domain controllers. How many workstations do you have BTW?
    As a starting point I would first of all get all of the FSMO roles back in one place (check they are working correctly to begin with - http://support.microsoft.com/kb/324801) remove DNS on all but the domain controllers and then check that replication occurs correctly. DNS problems are very awkward and take quite a while to sort out and troubleshoot. Its a shame you are on the other side of the world to me as I could talk you through it using our messaging client (not yet live for mainstream usage).

SHARE:
+ Post New Thread

Similar Threads

  1. PC Price Crash
    By Elky in forum Our Advertisers
    Replies: 0
    Last Post: 19th November 2007, 10:34 AM
  2. Exchange OWA SP2 Crash
    By Grommit in forum Windows
    Replies: 2
    Last Post: 11th September 2007, 10:13 AM
  3. AD/DNS Crash course Help
    By dave.81 in forum Windows
    Replies: 8
    Last Post: 3rd May 2007, 11:44 AM
  4. Erratic [Non Fatal] Crash
    By DaveP in forum Wireless Networks
    Replies: 7
    Last Post: 7th February 2007, 10:58 PM
  5. Scary Car Crash
    By eduabncs in forum Jokes/Interweb Things
    Replies: 3
    Last Post: 20th January 2007, 04:27 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •