Watchguard frequently dropping DNS requests / Smoothwall

[slancaster]

Morning all,

 

we have been having problems for ages now and can't find what the problem is.

 

We use Smoothwall for our filtering, and Watchguard for our Firewall. On a regular basis (daily, sometimes every other day), we loose connectivity from our smooth wall to our watch guard (shown in Smoothwall as "gateway failed"). After 18 minutes, the connection resumes. This resumes without touching Watchguard or Smoothwall - no reboots required.

Luckily Smoothwall have set up a failover port, which kicks in and connects to a different port on the Watchguard during the outage so our end users at worse see a 20 second "blip" which is solved by a refresh.

The dropouts happen around the same time of day when staff arrive / start to log on en-mass (between 8 and 9am).

 

To cut a long story short, Watchguard say "they can't see anything wrong" and Smoothwall say "Watchguard are dropping DNS packets". I say this is crazy and we need it fixing!

 

The dropouts only appear during term time, so for example over christmas break we didn't have any users logged on to our domain and we didn't see any dropouts in the logs.

If i connect a laptop directly on the WAN link then there are no ping drops.

 

So... could the problem be linked to a rogue machine logging in and doing something funky? Is it one user that runs a certain service which throws things off? Do I need to go and hit the Watchguard with a cricket bat and hope the problem goes away?

On a separate note, but an example of weird stuff happening, years ago we had a machine running OS X Lion, and every time that machine connected to our SAN, it sent the SAN in to a panic. We patched the SAN and things seemed ok. Now I'm not saying this is SAN related, but it shows how such a small thing can cause a huge problem.

 

ANY help would be muchly appreciated.

Edited January 15, 2016 by slancaster

[MatthewL]

Have you got ports on both ends set to auto negotiate as we had some issues years back and had to force the speed of the port to stop something similar, I think in the end we used a crossover cable to resolve the issue.

[ReBoot]

When I first setup my school's broadband I had a similar issue with DNS packets being dropped. This turned out to be a firmware bug in the Draytek router I was using and an update to later firmware sorted things out.

 

I did take me a few weeks to track down the issue and resolve it. I was able to get dns service stats on the local proxy server we use which showed what was happening.

 

I would suggest 4 approaches to diagnosing the problem:

Maybe possible to get DNS service stats on the smoothwall as I did to confirm what's happening.

 

If both your suppliers are denying responsibility you may have to employ something like wireshark to look at packets between the Smoothwall and Watchguard.

 

Its possible packets are being dropped by another component such as your router as I experienced.

 

Is it possible to take the watchguard out and test without it or substitute another firewall.

 

Good Luck !

Edited January 15, 2016 by ReBoot

[twin--turbo]

Yep Wireshark it..

 

TT

[slancaster]

Many thanks fellas, i'll investigate and report back.

[slancaster]

Im new to Wireshark and just installed on my Mac to have a look.

To capture traffic between the Smoothwall and Watchguard, would it be a case of plugging a laptop directly into the Watchguard and monitoring traffic on that interface? Watchguard would give the laptop and IP.

 

thanks

[ReBoot]

Unfortunately probably not. There is some info here on using wireshark. You need to make sure the layer 2 traffic you are interested in is replicated on the interface where you have wireshark. Putting a switchport in promiscuous mode achieves this. Traffic to the Smoothwall will probably not be on another Watchguard port because of MAC learning.

 

https://wiki.wireshark.org/CaptureSetup/Ethernet

[slancaster]

Thank you.

[slancaster]

Thinking about it, Smoothwall ran a packet capture for us a while back. This showed packets being sent from Smoothwall but not coming back to Smoothwall. Watch guard however seemed to think everything was ok from there end.

[slancaster]

Our Watchguard is set to Auto... just checking Smoothwall now...

[slancaster]

Still having these drop outs. Here is so more info just in case anyone can suggest anything - getting desperate now!

 

The drop outs USUALLY occur in the morning, around 8.30/8.40am - this is when a lot of admin staff arrive and login. By "drop out", I mean our smoothwall detects that the main gateway to our Watchguard is down. Smoothwall then re-directs traffic to a failover port (without this we would be screwed). After 18 minutes (its always 18 minutes), the main gateway is back online and Smoothwall connects back to it.

When Smoothwall moves to the failover port, this connects to a different port on the Watchguard.

 

Now, Smoothwall say that packets are getting sent FROM the Smoothwall TO the Firewall (Watchguard) and are not getting a reply. This isn't ALL packets, just some.

 

Over half term, when there where literally about 3 staff in the building, we did NOT have a drop out at all, so these problems must either get triggered by something / someone, or are the result of some kind of built up.

 

 

So, is there any kind of service that takes 18 minutes (a long time) to rebuild / restart? Is the problem with Smoothwall, Watchguard or something Internal?

Lets just say its giving me a headache....

[Xenon2]

Hi slancaster,

I think we are experiencing similar issue to you back in 2016. Did you manage to resolve it?

 

 

Still having these drop outs. Here is so more info just in case anyone can suggest anything - getting desperate now!

 

The drop outs USUALLY occur in the morning, around 8.30/8.40am - this is when a lot of admin staff arrive and login. By "drop out", I mean our smoothwall detects that the main gateway to our Watchguard is down. Smoothwall then re-directs traffic to a failover port (without this we would be screwed). After 18 minutes (its always 18 minutes), the main gateway is back online and Smoothwall connects back to it.

When Smoothwall moves to the failover port, this connects to a different port on the Watchguard.

 

Now, Smoothwall say that packets are getting sent FROM the Smoothwall TO the Firewall (Watchguard) and are not getting a reply. This isn't ALL packets, just some.

 

Over half term, when there where literally about 3 staff in the building, we did NOT have a drop out at all, so these problems must either get triggered by something / someone, or are the result of some kind of built up.

 

 

So, is there any kind of service that takes 18 minutes (a long time) to rebuild / restart? Is the problem with Smoothwall, Watchguard or something Internal?

Lets just say its giving me a headache....

[MatthewL]

I've run Watchguard for years never really had an issues, has been an issue recently with an update that broke NAT but there was a workaround. Is the Watchguard on the latest update available or are you miles out?

[Xenon2]

Thanks Matthew,

Watchguard is on the latest version but Smoothwall isn't because when I put the latest update it on, it seemed to exacerbate the issue.

[10101010]

I had a major issue with users authenticating on Smoothwall and it turned out to be the disk performance issue Might be worth checking the disk wait time which you can find by running Top if you log in as root and SSH session.

 

[Xenon2]

Thanks 10101010, that is a really interesting lead. Our "wa" fluctuates quite wildly. I have seen it go up to 6.7. It will be interesting to compare the figures when the load has been removed. I have sent Smoothwall a screenshot.

 

Thanks again.

[ibpalle]

The wait states show delays when waiting to put or retrieve content from disks or RAM - it does fluctuate and is supposed to. The load average of 6.7 isn't high - once you get into the 20-30s the web interface will start becoming slower and that's where it may indicate an issue.

 

If the problem can somewhat be attributed to DNS, have a look at the DNS path on your network, you may be able to optimise this.

 

Internal DNS servers - what are they suing as forwarders? ISP DNS , root servers, Watchguard DNS proxy or Smoothwall DNS proxy?

 

Clients on youir domain will likely use your internal DNS servers but what about BYOD users?

 

What we normally do when the Smoothwal is the firewall as well, is set the Smoothwall to resolve externally using ISP or other publicly available DNS services and internally via conditional DNS forwarders, use your internal AD DNS servers.

 

Clients are then set to use either internal AD DNS or the Smoothwall as DNS and then youir internal AD DNS servers are set to use Smoothwall as their DNS forwarder.

 

This will reduce the amount of external lookups and make the Smoothwall system a DNS cache for your entire network. I assume the Watchguard have a DNS proxy service as well, in which case, that can be used instead of the Smoothwall.

[Xenon2]

Thanks ibpalle,

Much appreciated. I've answered the questions within the post as best I can...

 

The wait states show delays when waiting to put or retrieve content from disks or RAM - it does fluctuate and is supposed to. The load average of 6.7 isn't high - once you get into the 20-30s the web interface will start becoming slower and that's where it may indicate an issue.

 

If the problem can somewhat be attributed to DNS, have a look at the DNS path on your network, you may be able to optimise this.

 

Internal DNS servers - what are they suing as forwarders? ISP DNS , root servers, Watchguard DNS proxy or Smoothwall DNS proxy?

Currently no forwarders set.

 

Clients on youir domain will likely use your internal DNS servers but what about BYOD users?

 

BYOD, Guest and Chromebook IPs are assigned by Smoothwall DHCP. DNS is set to the first address in each of those ranges.

What we normally do when the Smoothwal is the firewall as well, is set the Smoothwall to resolve externally using ISP or other publicly available DNS services and internally via conditional DNS forwarders, use your internal AD DNS servers.

 

We have it configured like this.

Clients are then set to use either internal AD DNS or the Smoothwall as DNS and then youir internal AD DNS servers are set to use Smoothwall as their DNS forwarder.

 

Internal clients are set to use internal DNS (not sure what to do with the Smoothwall DHCP clients?)

This will reduce the amount of external lookups and make the Smoothwall system a DNS cache for your entire network. I assume the Watchguard have a DNS proxy service as well, in which case, that can be used instead of the Smoothwall.

[ibpalle]

It may be an idea to try using forwarders on your internal AD DNS servers - point them to the Smoothwall as DNS forwarder - it should reduce the outgoing DNS requests and possibly make for a smoother DNS path generally.

 

For the Smoothwall DHCP clients I am assuming the first IP in the subnet is the Smoothwall itself, so we should be good there.

[Xenon2]

Thanks ibpalle,

I've added the forwarder.

 

Please find the config on Smoothwall DHCP. Is any further config required/recommended for this?

 

[ibpalle]

For the BYOD I tend to add your AD domain name as a search domain in DHCP settings. I assume the 172.16 60 1 is the Smoothwall itself? Then in the network - firewall - Smoothwall access make sure that the services for DNS, port 80 and 442 is allowed as well along with any other service needed (possibly RADIUS)

[Xenon2]

Thanks ibpalle, you are a legend! It is looking better already.

 

By search domain, do you mean the "Domain name suffix:" option?

It is a shame that official Smoothwall channel wasn't as helpful!

 

Thanks again!

[ibpalle]

Yes, I meant the domain name suffix.

 

Looks like I need to run a DNS training session for our support team Glad it's looking better for your network.