Intermittent drop in internet connection

[icemonkey]

So this has been an intermittent problem for a few weeks now and honestly, I'm stumped.Every now and then, we lose our internet connection for a few minutes.

 

We've had a successful line test from the ISP and they have confirmed it's not an issue at their end.

This all began as what appeared to be a DDoS, our S8 was showing hundreds (if not thousands) of external hits at the same time as the drop outs. Following DDoS mitigation and changing our S8's external IP, this is no longer happening yet the drop outs continue.

 

I've been working with our ISP and Smoothwall to look at the possible causes. Smoothwall are convinced it's a problem upstream of our S8, the ISP are convinced it's being caused by a device on the LAN.

Neither Smoothwall nor I can find anything in the logs to support the ISP's claim.

To confuse matters, we had no issues over half term which supported the idea of an internal device being the cause but then we've just had 8 term-time days with no issues. This was until yesterday when we had 4 instances between 8:15am and 9:30am.

 

In terms of the services which are affected - Uptime Robot is showing me connection timeouts for Remote Access (VM), VLE (VM), ISP Primary & Smoothwall (thus internet connection). Basically anything with an external IP. Internal services are unaffected.

 

Now we're past the "You need to speak to the ISP / You need to speak to Smoothwall" debate, Smoothwall have been really helpful but I'm looking for some ideas of what else I could/should be looking at!?All I'm being told from the ISP is that they can see CPU utilisation alerts on the router which match the times of the outages and as far as I'm aware, nothing has changed on our network.

Edited March 10, 2020 by icemonkey
Formatting

[paulkerton]

Now we're past the "You need to speak to the ISP / You need to speak to Smoothwall" debate, Smoothwall have been really helpful but I'm looking for some ideas of what else I could/should be looking at!?All I'm being told from the ISP is that they can see CPU utilisation alerts on the router which match the times of the outages and as far as I'm aware, nothing has changed on our network.

 

If your network is segmented, you should be able to see where the issue lies... at least what area of the network it's happening in.

[SchoolsBroadband]

A very simplistic test can be done below.

 

Does your ISP install a managed router that sits in front of the Smoothwall box? If so why not do some constant pings from inside your network to the external IP from your router and then also from another Internet connection send constant pings to your ISP's router.

 

That way you can tell which bit of the kit (i.e. smoothwall or router) could be dropping the packets.

 

Good luck

 

Dave

[Chockster]

Check whether a NIC is dropping on the smoothwall box. A couple of years ago we had a very similar set of symptoms over the course of a few weeks. It turned out to be a failed upgrade of the driver on one of the NICs; it looked to all intents and purposes that someone was pulling the cable and immediately replacing it. Pings would find the box but dropped packets were still happening. I eventually found this in the Kernel log:

 

Sep 20 14:04:12 smoothwall Kernel [ 9764.209964] e1000e: ethE NIC Link is Down

Sep 20 14:04:12 smoothwall Kernel [ 9764.277459] bonding: bond0: link status definitely down for interface ethE, disabling it

Sep 20 14:04:12 smoothwall Kernel [ 9764.277462] bonding: bond0: making interface ethC the new active one

Sep 20 14:04:12 smoothwall Kernel [ 9764.326012] e1000e: ethC NIC Link is Down

Sep 20 14:04:12 smoothwall Kernel [ 9764.377500] bonding: bond0: link status definitely down for interface ethC, disabling it

Sep 20 14:04:12 smoothwall Kernel [ 9764.377784] bonding: bond0: now running without any active interface!

Sep 20 14:04:15 smoothwall Kernel [ 9767.375993] e1000e: ethE NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

Sep 20 14:04:15 smoothwall Kernel [ 9767.378898] bonding: bond0: link status definitely up for interface ethE, 1000 Mbps full duplex

Sep 20 14:04:15 smoothwall Kernel [ 9767.378904] bonding: bond0: making interface ethE the new active one

Sep 20 14:04:15 smoothwall Kernel [ 9767.379499] bonding: bond0: first active interface up!

Sep 20 14:04:15 smoothwall Kernel [ 9767.552037] e1000e: ethC NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

Sep 20 14:04:15 smoothwall Kernel [ 9767.578988] bonding: bond0: link status definitely up for interface ethC, 1000 Mbps full duplex

 

Smoothwall support dialled in and reverted the driver. We moved to sophos a few weeks later.

 

Chris

Edited March 10, 2020 by Chockster

[icemonkey]

A very simplistic test can be done below.

 

Does your ISP install a managed router that sits in front of the Smoothwall box? If so why not do some constant pings from inside your network to the external IP from your router and then also from another Internet connection send constant pings to your ISP's router.

 

That way you can tell which bit of the kit (i.e. smoothwall or router) could be dropping the packets.

 

Good luck

 

Dave

Yes, we have a router on site which is managed by the ISP with unfiltered traffic down to our smoothwall.

 

Didn't think of pinging both internally and externally - just let it run for about 10 minutes initially and I'm getting a 0.2% packet loss (negligible but still greater than 0) from the external connection. Internal is reading as 0%.

 

Interestingly, I've spoken to our ISP again and they say their DDoS mitigation system is still flagging up significant traffic hitting our old external IP on UDP port 53. This IP is no longer listed as as interface on our Smoothwall box which is why I haven't been able to see it in the logs. They're now looking at dropping any traffic heading here from their end so fingers crossed we'll see an improvement.

 

- - - Updated - - -

 

Check whether a NIC is dropping on the smoothwall box. A couple of years ago we had a very similar set of symptoms over the course of a few weeks. It turned out to be a failed upgrade of the driver on one of the NICs; it looked to all intents and purposes that someone was pulling the cable and immediately replacing it. Pings would find the box but dropped packets were still happening. I eventually found this in the Kernel log:

 

Sep 20 14:04:12 smoothwall Kernel [ 9764.209964] e1000e: ethE NIC Link is Down

Sep 20 14:04:12 smoothwall Kernel [ 9764.277459] bonding: bond0: link status definitely down for interface ethE, disabling it

Sep 20 14:04:12 smoothwall Kernel [ 9764.277462] bonding: bond0: making interface ethC the new active one

Sep 20 14:04:12 smoothwall Kernel [ 9764.326012] e1000e: ethC NIC Link is Down

Sep 20 14:04:12 smoothwall Kernel [ 9764.377500] bonding: bond0: link status definitely down for interface ethC, disabling it

Sep 20 14:04:12 smoothwall Kernel [ 9764.377784] bonding: bond0: now running without any active interface!

Sep 20 14:04:15 smoothwall Kernel [ 9767.375993] e1000e: ethE NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

Sep 20 14:04:15 smoothwall Kernel [ 9767.378898] bonding: bond0: link status definitely up for interface ethE, 1000 Mbps full duplex

Sep 20 14:04:15 smoothwall Kernel [ 9767.378904] bonding: bond0: making interface ethE the new active one

Sep 20 14:04:15 smoothwall Kernel [ 9767.379499] bonding: bond0: first active interface up!

Sep 20 14:04:15 smoothwall Kernel [ 9767.552037] e1000e: ethC NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

Sep 20 14:04:15 smoothwall Kernel [ 9767.578988] bonding: bond0: link status definitely up for interface ethC, 1000 Mbps full duplex

 

Smoothwall support dialled in and reverted the driver. We moved to sophos a few weeks later.

 

Chris

I've had a look and there's nothing similar in our kernel log but definitely sounds like it's worth watching out for in future!