Troubleshooting DNS Performance

[StephenPink]

Hi all,

Experiencing network difficulties at a site currently - all signs pointing to slow DNS resolution. iPerf/Fast.com out to internet consistently performs well but constant intermittent issues loading sites. 

Struggling to narrow this down - or reproduce when no one on site (so seems load related).

 

Was wondering if there were any recommendations for tools to help troubleshoot DNS performance, internally?

Cheers

[mavhc]

https://www.grc.com/dns/benchmark.htm

[StephenPink]

1 hour ago, mavhc said:

https://www.grc.com/dns/benchmark.htm

Couldn't seem to get that to look at local DNS - just wanted to test performance to other internet based DNS servers?

[mavhc]

When I run it, and then ignore dns failure, as most dns is blocked here, and go to Nameservers tab, the local dns server is there, plus there's an add/remove button.

 

Results page

   10. 1. 22.  33 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.007 | 0.009 | 0.013 | 0.001 | 100.0 |
  + Uncached Name | 0.020 | 0.043 | 0.237 | 0.047 | 100.0 |
  + DotCom Lookup | 0.020 | 0.026 | 0.033 | 0.003 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
vs
     8.  8.  8.  8 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Cached Name   | 0.014 | 0.019 | 0.028 | 0.003 | 100.0 |
  - Uncached Name | 0.018 | 0.043 | 0.222 | 0.050 | 100.0 |
  - DotCom Lookup | 0.019 | 0.023 | 0.032 | 0.003 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                       dns.google
                       GOOGLE, US


    1.  0.  0.  1 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Cached Name   | 0.016 | 0.019 | 0.025 | 0.002 | 100.0 |
  - Uncached Name | 0.020 | 0.045 | 0.203 | 0.044 | 100.0 |
  - DotCom Lookup | 0.020 | 0.024 | 0.028 | 0.002 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                     one.one.one.one
                    CLOUDFLARENET, US


    1.  1.  1.  1 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Cached Name   | 0.016 | 0.019 | 0.028 | 0.002 | 100.0 |
  - Uncached Name | 0.020 | 0.048 | 0.217 | 0.050 | 100.0 |
  - DotCom Lookup | 0.020 | 0.023 | 0.027 | 0.001 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                     one.one.one.one
                    CLOUDFLARENET, US


    8.  8.  4.  4 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Cached Name   | 0.014 | 0.024 | 0.163 | 0.021 | 100.0 |
  - Uncached Name | 0.018 | 0.040 | 0.248 | 0.046 | 100.0 |
  - DotCom Lookup | 0.018 | 0.024 | 0.032 | 0.003 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                       dns.google
                       GOOGLE, US

 

[StephenPink]

Oh I missed that you could carry on - likewise we have most DNS blocked here. Anyway, still doesn't seem to work though;

 

 

Never had this before and have used this in the past successfully! Can't even "Run Benchmark" as greyed out... 

[mavhc]

Is this on the server that's running DNS? Weird. What happens if you put the LAN IP?

[StephenPink]

2 hours ago, mavhc said:

Is this on the server that's running DNS? Weird. What happens if you put the LAN IP?

Yup! Oh yeah that's the blanked out one - the internal IP of the other DNS server. Not sure what's going on here - might be related to Securly though as seems to do the same at another site not having performance issues... 

[mavhc]

Securly is installed on the DNS server?

[StephenPink]

No sorry - our internal DNS servers have forwarders set to Securly's DNS servers. Securly is a cloud-based proxy service, for filtering and monitoring. 

Why this would impact the DNS Benchmark tool though I don't know...  

[KK20]

Yep I get "The DNS server at this IP address does not provide domain name service answering client queries.  It should not be used for normal client-based resolution"  which is odd because I have the local DNS servers set on this machine at the moment and it is quite obviously working....  I removed all the default leaving on the local system servers.

[StephenPink]

Okay so still working on this. 99% sure it is a local network issue - but not sure what/where it could be affecting this?

I have written a PowerShell script to periodically run some DNS resolution checks, against a few different domains, and using different DNS servers - clearing DNS cache inbetween. Consistently, the poorest responses are from our internal DNS servers - going directly to the Securly DNS servers for the same query (which our internal DNS servers simply forward to) - is consistently much quicker. 

Generally response times are between 3 and 20ms against the external DNS servers. Periodically, the response times against our internal DNS servers hit 12,000ms+ and some completely timeout. 

Any ideas where to start looking? Nothing obvious on the DNS servers in Event Viewer etc, performance doesn't seem to be an issue in terms of CPU/RAM/disk usage etc.

Cheers

[machy]

21 minutes ago, StephenPink said:

Okay so still working on this. 99% sure it is a local network issue - but not sure what/where it could be affecting this?

I have written a PowerShell script to periodically run some DNS resolution checks, against a few different domains, and using different DNS servers - clearing DNS cache inbetween. Consistently, the poorest responses are from our internal DNS servers - going directly to the Securly DNS servers for the same query (which our internal DNS servers simply forward to) - is consistently much quicker. 

Generally response times are between 3 and 20ms against the external DNS servers. Periodically, the response times against our internal DNS servers hit 12,000ms+ and some completely timeout. 

Any ideas where to start looking? Nothing obvious on the DNS servers in Event Viewer etc, performance doesn't seem to be an issue in terms of CPU/RAM/disk usage etc.

Cheers

can you use Wireshark and see if the local DNS server is being bombarded by something? 

Even though the CPU Usage might be fine, the service could be playing up 

 

Have you tried running the test quires on the Local server itself?

[StephenPink]

9 minutes ago, machy said:

can you use Wireshark and see if the local DNS server is being bombarded by something? 

Even though the CPU Usage might be fine, the service could be playing up 

 

Have you tried running the test quires on the Local server itself?

Nothing jumped out on the Wireshark captures I've done so far - however am far from an expert so that's not necessarily a good check! Any advice on specifics I should be looking for?

Good idea - I have run the same script now directly on the DNS server, and interestingly the resolution times when looking up using itself, are still periodically 12,000ms+.... 

[machy]

With Wireshark if you filter by DNS _ws.col.protocol == "DNS" and just see if you seem to be getting a silly amount of requests for the same addresses, from the same ips. 

We had a OneDrive mapper have a bad update a while ago, and it killed all the local DNS servers every time someone logged on as for 10 minutes it generated thousands of requests for the same set of address.

(with some exceptions) a PC shouldn't keep asking for the IP Address, so if a Client asks for the IP for Google.com it shouldn't ask for google.com again for x amount of time.

 

So if you see 10.1.1.55 Keep asking for "google.com" even though its getting a response, somethings funky 

In effect you would be D/DOSing your Local servers, and Securly isn't bothered as they can probably handle your years worth a minute 

 

If that makes sense? 

 

If its still that resolution time when run on the server (for the sake of clarity are you using local host over the IP?) 

That might rule out it being a network issue and it being a server/service one

 

 

 

[StephenPink]

On 08/01/2026 at 11:15, machy said:

With Wireshark if you filter by DNS _ws.col.protocol == "DNS" and just see if you seem to be getting a silly amount of requests for the same addresses, from the same ips. 

We had a OneDrive mapper have a bad update a while ago, and it killed all the local DNS servers every time someone logged on as for 10 minutes it generated thousands of requests for the same set of address.

(with some exceptions) a PC shouldn't keep asking for the IP Address, so if a Client asks for the IP for Google.com it shouldn't ask for google.com again for x amount of time.

 

So if you see 10.1.1.55 Keep asking for "google.com" even though its getting a response, somethings funky 

In effect you would be D/DOSing your Local servers, and Securly isn't bothered as they can probably handle your years worth a minute 

 

If that makes sense? 

 

If its still that resolution time when run on the server (for the sake of clarity are you using local host over the IP?) 

That might rule out it being a network issue and it being a server/service one

 

 

 

Thank you for this - really helpful. 

So - nothing obvious still in the Wireshark, I didn't spot excessive repeated Client IPs or anything. 

And good point to use the loclhost address - have tried both it's IP and localhost IP, and yes still getting the intermittent slow responses... so perhaps not network related then??

[machy]

With your DNS forwarding, do all your servers go directly to Securely or do they ask each other first 

IE Server A Doesn't have it, so it asks Server B then Server B Asks Securely, my thought process being is if you have a loop.

 

A Couple more risky options

 

Turn of the DNS service on all but one server, see what happens, potentially hopping through the list of your servers 

Set up something like Pi Hole temporarily, and this should let you see lookup counts etc iirc  

 

My thinking being is are you getting to many requests or are they not being handled right

[psydii]

I wonder if get-dnsstatistics might yield useful data? 

 

Also in the final example in this article:

 

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/dns-client-resolution-timeouts
 

the timeout seen is 12ish seconds, which matches what you are seeing.

 

And, (per that article) “

Any Name Error response by any of the DNS servers will cause the process to stop - client doesn't retry with the next server if the response was negative. Client tries new servers only if the previous are unreachable.”

 

Which leads me to the question: are you sure when you have a timeout the client is even  able to reach the dns servers?

 

Other random things that jump to mind based on other’s suggestions earlier in the thread:

 

What other roles do your DNS server hold? How many clients are connected/using those services? If virtual, is the performance latency at the host level ok?

 

is it possible there is another device with the same IP/MAC address as the server on the network somewhere?

 

is there any network flood protection kicking in at a switch level?

 

There is also dns logging that you can turn on- it is extremely verbose so you will need to plan/use careful to not run out disk quite quickly.

[StephenPink]

1 hour ago, machy said:

With your DNS forwarding, do all your servers go directly to Securely or do they ask each other first 

IE Server A Doesn't have it, so it asks Server B then Server B Asks Securely, my thought process being is if you have a loop.

 

A Couple more risky options

 

Turn of the DNS service on all but one server, see what happens, potentially hopping through the list of your servers 

Set up something like Pi Hole temporarily, and this should let you see lookup counts etc iirc  

 

My thinking being is are you getting to many requests or are they not being handled right

 

Hm that's a good point... so they're setup as per Microsoft best practice (although am aware this always a hotly contested debate)

- DC1

Primary DNS: DC2

Secondary DNS: 127.0.0.1

 

- DC2

Primary DNS: DC1

Secondary DNS: 127.0.0.1

 

And then both have their forwards set to Securly, and root hints disabled. 

 

Not sure how that could cause a loop though?

 

 

Risky options indeed haha! I agree though trying to distinguish if a load issue or an error issue would be helpful - I did temporarily increase VM resources (CPU/RAM) which didn't make any noticable difference.
 

 

1 hour ago, psydii said:

I wonder if get-dnsstatistics might yield useful data? 

 

Also in the final example in this article:

 

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/dns-client-resolution-timeouts
 

the timeout seen is 12ish seconds, which matches what you are seeing.

 

And, (per that article) “

Any Name Error response by any of the DNS servers will cause the process to stop - client doesn't retry with the next server if the response was negative. Client tries new servers only if the previous are unreachable.”

 

Which leads me to the question: are you sure when you have a timeout the client is even  able to reach the dns servers?

 

Other random things that jump to mind based on other’s suggestions earlier in the thread:

 

What other roles do your DNS server hold? How many clients are connected/using those services? If virtual, is the performance latency at the host level ok?

 

is it possible there is another device with the same IP/MAC address as the server on the network somewhere?

 

is there any network flood protection kicking in at a switch level?

 

There is also dns logging that you can turn on- it is extremely verbose so you will need to plan/use careful to not run out disk quite quickly.

Assuming you meant Get-DNSServerStatistics, nothing that means too much to me. Other than the fact that the fact that the "TimeElapsedSinceLastClearedStatisticsBetweenRestart" and "TimeElapsedSinceServerStartBetweenRestart" seems to be permanently stuck at "00:00:16"

So to answer the first question "are you sure when you have a timeout the client is even able to reach the dns servers?" I guess when running the DNS testing script directly on the server, I am remoted onto it and still stay connected, and the script keeps running so yes?

Roles: They are Domain Controllers and DNS Servers and that's it (oh and one is an NTP server) Yes virtual, and performance all seems okay

 

Duplicate IP/MAC: IP no, and the second DC is also affected. MAC address is interesting though, I'll see what I can find there... 

Network: No that I've seen, however we are partway through a large network project so not all the infrastructure is easily visible yet

Logging: Thanks, I'll have a read up and see what I can find if there's a way to perhaps be quite specific with what to log etc

[StephenPink]

Okay so using perfmon, looks like average figures for the below are;

- Total Query Received/sec: ~1,400

- Recursive Queries/sec: ~685

 

Neither of which seem particularly high, which seems to indicate not a load issue.

Looking at the debug logging (definitely not 100% sure what I'm looking for) but spotted every so often a bunch of entries with this;

- [8281   DR SERVFAIL] 65

- [8281   DR SERVFAIL] A 

 

This doesn't seem to be for consistent domain lookups though... and not sure what exactly the above means as yet!

Cheers

[machy]

Back to stupid ideas, could you set one of your DCS to have the fwders of 

1.1.1.3, 1.0.0.3 (Cloudflare for families, so should be safe enough for testing) then hammer that DC with your test tool 

 

More stabbing in the dark

But my brain goes to is seculry rate limiting by IP

Your requests should be a rounding error to them, but we all know edu providers aren't smart