Smoothwall Gateway going down

[Benjaminbl12]

Hello world,

 

Just a query if anything for the most part - Our Smoothwall got an update last Thursday and we've found that since at 10:20am each day our internet goes down for 2 minutes. Looking at Smoothwall when this happens it states the gateway has gone down. We use pfSense as our Firewall and this hasn't been a problem since that update was released. We're using the update Leeds-42.

 

Any thoughts/ideas?

 

Kind regards,

Ben

[2097]

Do you have any machines that have direct internet access ? When smoothwall goes down try a directly connected internet machine and see if the internet still works.

It could be a ddos attack as the free accounts available online for "testing" ddos normally only allow short DDOS intervals.

[Mako]

Possibly related - I opened a thread in the Cloud Services forum about O365 being inaccessible today. Nobody else on 'Geek having an issue. After some troubleshooting, I've pinned it on the Smoothwall. If I bypass it, everything worth absolutely fine. Lightning fast.

 

It seems to be crapping out when there's a lot of users accessing the internet, any Microsoft services becomes unavailable. The only difference between yesterday and today is that it installed Leeds-42 overnight. I don't think SchoolsICT are identifying any issues at their end.

 

Phones also having trouble, so there's a call logged with the comms team but I think it's the Smoothie.

 

While I'm not having the gateway failing, our common ground is issues since the update.

[lmrogers]

Not sure if it is related but I recently had an issue with our Smoothwall where our WAN interface kept dropping intermittently and would cause issues with slow internet, audio on phone calls dropping and random disconnects. After eventually getting a Smoothwall engineer assigned to the case and pointing them in the right direction - http://www.edugeek.net/forums/internet-related-filtering-firewall/214100-intermittent-drop-internet-connection.html - they had to roll back a driver update on the external interface on the Smoothwall. Since this has been done we have not had the issue.

[E_G_R2]

@Mako we've had the said same problem with our soon to be replaced Smoothwall. Seems that the amount of Teams traffic is causing it to lock and only a reboot will fix the issue. OK our Smoothwall may be a tad under specked for the job in hand but the current level of support is a joke (2 weeks for a support call to be responded to, had to go to the account manager)

[Mako]

@E_G_R2 was the rest of internet browsing fine? Everyone here is absolutely fine browsing the web, it's only MS related services (microsoft.com, office.com etc).

[Primus]

Oh great we've once again got different schools seeing difefferent latest releases. The latest available to my 2 schools is Leeds-39.

[tom_newton]

@Primus if you can PM me your details i'll make sure your schools all get the updates simultaneously. We have a phased rollout and sometimes folk get straddled between 2 phases.

@Benjaminbl12 It seems unlikely that 42 has caused this - but i'd like to look more deeply - if you can let me know a support ticket or school name?

[howartp]

Interesting timely thread, don't know if it's related.

 

We had authentication popups at around 9.45 yesterday morning - we rebooted SW (no change), DC1 (no change) and DC2 (problem solved).

 

This morning at 11am then again about 12noon we had the same issue, but only for about 30 seconds - which is why this thread caught my eye.

 

we did have the same auth problem the week before Half Term around 8.15am and had to reboot SW, DC1 and DC2 to go away.

 

We're on Leeds 37 at present.

 

EDIT: Just literally happened again, 1.30pm, as I hit submit on this post!

Edited November 13, 2020 by howartp

[Mako]

@howartp Auth pop ups are usually when it's struggling to cope with the amount of traffic and there's a hardware bottleneck somewhere (or there's an actual issue with your DCs but I rarely find that's the case!). What's your resource usage at during those times?

 

There's another thread somewhere in the Smoothie forum where the latest boxes are running crappy 5.6k HDDs instead of SSDs. I've had a look at mine and it indeed does have 5.6k rpm HDDs in it. Even before today we get auth popups when the internet is being hammered, our resource usage sometimes goes up to 15 or even 18, I believe the box is struggling. I've raised it with SchoolsICT who are chasing up the supposed free replacement drives from Smoothwall. Naturally if it solves nothing then I'm resigned to having to upgrade the box as maybe it just can't cope.

 

The fact that there's a significant increase in online resource use isn't helping. Office 365 and our VLE are being used multiple factors more than before Lockdown 1.0 so the amount of traffic is becoming an issue. Then there's also streaming services like Teams that were not in use before.

 

Sorry @Benjaminbl12 , we might be derailing your thread.

[ibpalle]

Just a query if anything for the most part - Our Smoothwall got an update last Thursday and we've found that since at 10:20am each day our internet goes down for 2 minutes. Looking at Smoothwall when this happens it states the gateway has gone down. We use pfSense as our Firewall and this hasn't been a problem since that update was released. We're using the update Leeds-42.

 

Do you have more than one interface with a gateway set and if you do, are both internet connections?

[E_G_R2]

@E_G_R2 was the rest of internet browsing fine? Everyone here is absolutely fine browsing the web, it's only MS related services (microsoft.com, office.com etc).

 

Yes other sites were fine but anything MS such as teams and exchange online was blocked.

[supportman]

I've been tracking this on our site as well for the last week or so....

 

Our smoothwall seems to have issues for about 10mins every day at around the same time, I'm really not sure what is causing it but the interface becomes unresponsive and internet around school goes down or is very slow.

 

Any ideas how to troubleshoot this? I assumed it was our ISP at first until reading this!

 

I get this in the Kernal logs

 

[TABLE="width: 1782"]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.250825] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.251984] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.255814] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.256341] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.256581] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.263998] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.266961] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[/TABLE]
[TABLE="width: 1781"]
[TR]
[TD="class: date"]Nov 16 13:48:18[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604513.437496] net_ratelimit: 1784 callbacks suppressed[/TD]
[/TR]
[/TABLE]

We are on Leeds 39

Edited November 16, 2020 by supportman

[SpaceInvader83]

I've been tracking this on our site as well for the last week or so....

 

Our smoothwall seems to have issues for about 10mins every day at around the same time, I'm really not sure what is causing it but the interface becomes unresponsive and internet around school goes down or is very slow.

 

Any ideas how to troubleshoot this? I assumed it was our ISP at first until reading this!

 

I get this in the Kernal logs

 

[TABLE="width: 1782"]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.250825] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.251984] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.255814] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.256341] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.256581] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.263998] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[TR]
[TD="class: date"]Nov 16 13:46:48[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604423.266961] nf_conntrack: table full, dropping packet[/TD]
[/TR]
[/TABLE]
[TABLE="width: 1781"]
[TR]
[TD="class: date"]Nov 16 13:48:18[/TD]
[TD="class: machine"]smoothwall[/TD]
[TD="class: section"]Kernel[/TD]
[TD="class: contents"][604513.437496] net_ratelimit: 1784 callbacks suppressed[/TD]
[/TR]
[/TABLE]

We are on Leeds 39

Have you tried running htop from an ssh session at the time of the issue? Just to see what processes are hammering your box

[howartp]

From North Yorkshire Police:

 

Not sure if you’re aware of an ongoing sustained mass port scanning / attempt compromise of public sector networks. I am informed that there is a large discussion ongoing on CiSP about this. The current advice from NCSC is to block all connections / attempts from IP range: 45.146.164.0/23. Attacks seem to be varied but include scanning and brute force attempting SFTP and SSH services with credential stuffing.

 

I just wanted to make you aware and ask if you could maybe forward this message onto your other local network manager contacts.

 

Many thanks

[ibpalle]

The Network » Settings » Advanced area has a setting for increasing the conntracking size in case you want to try that.

[supportman]

Thanks both, I have blocked the IP and increased our Connection Tracking size to "Auto" which is about 20 times more than it was manually set to.

 

Lets see how it goes today....

[andyk1975]

I am glad I have stumbled upon this thread!

 

We are having exactely the same issues with our Smoothwall box - and have for a number of weeks. The internet slows down, then authentication box start popping up on users PC's, then it last 30 mins or so and then picks up again.

 

It happens the same time everyday - Thursday's particualry for some reason.

 

I have spoken with smoothwall, who are changing our authentication method to idex next week, but reading this thread it makes me wonder whether it is a hardware issue on the Smmothwall Box and not an issue with our authenitication / DC's. It is a newish box - replaced in Feb 2020, and is running Leeds 39.

 

I have blocked the IP that has been mentioned, but our connection tracking table was already set to Auto.

 

Very strange and difficult to pin point exactely what is happening.

[Andycat]

I am glad I have stumbled upon this thread!

 

We are having exactely the same issues with our Smoothwall box - and have for a number of weeks. The internet slows down, then authentication box start popping up on users PC's, then it last 30 mins or so and then picks up again.

 

It happens the same time everyday - Thursday's particualry for some reason.

 

I have spoken with smoothwall, who are changing our authentication method to idex next week, but reading this thread it makes me wonder whether it is a hardware issue on the Smmothwall Box and not an issue with our authenitication / DC's. It is a newish box - replaced in Feb 2020, and is running Leeds 39.

 

I have blocked the IP that has been mentioned, but our connection tracking table was already set to Auto.

 

Very strange and difficult to pin point exactely what is happening.

 

Our path was the same, almost exactly, including changing the authentication method. It was the HDD all along!

[supportman]

I am glad I have stumbled upon this thread!

 

We are having exactely the same issues with our Smoothwall box - and have for a number of weeks. The internet slows down, then authentication box start popping up on users PC's, then it last 30 mins or so and then picks up again.

 

It happens the same time everyday - Thursday's particualry for some reason.

 

I have spoken with smoothwall, who are changing our authentication method to idex next week, but reading this thread it makes me wonder whether it is a hardware issue on the Smmothwall Box and not an issue with our authenitication / DC's. It is a newish box - replaced in Feb 2020, and is running Leeds 39.

 

I have blocked the IP that has been mentioned, but our connection tracking table was already set to Auto.

 

Very strange and difficult to pin point exactly what is happening.

 

Our internet is totally fixed now, whether it was the IP block of that specific IP or the connection increase on the Smoothwall it seems to have fixed it.

 

Caused us a nightmare too for a number of weeks with unexplained slowdowns or outages...

 

I do wonder if this is something smoothwall should be warning about more widely as it seems a limitation of the settings on their hardware.

[andyk1975]

Glad it is all sorted. How did you go about getting the hardrive replaced? We have 2 x 1TB drives in our box, so obviously would need 2 x 1TB SSD. Is this something Smoothwall provided and replaced? Was there a charge?

[Andycat]

Glad it is all sorted. How did you go about getting the hardrive replaced? We have 2 x 1TB drives in our box, so obviously would need 2 x 1TB SSD. Is this something Smoothwall provided and replaced? Was there a charge?

 

We had a support call open, and we got the specs of the drive (it was a 5400rpm CCTV HDD) so they sent out a 7200rpm (non ssd) but it's fine so far.

[Mako]

I've had exactly the same response as @Andycat .Although I went through SchoolsICT who logged it with SW on my behalf (we don't have direct SW support). A SW engineer remoted in and monitored the box and confirmed the wait times on the disks were far too excessive. I received new 7200 1TB drives on Thursday. Disappointed they're not SSDs, the logic makes no sense. Someone is coming to rebuild the SW this week.

[robk]

I made this change on our new s10 a week ago, the difference is amazing.

 

30% waits and 12-16 load ave (and box not working)

 

Down to 5% and 6-8 load ave (box now working).

 

So worth it if support suggest the swap.

[howartp]

We’ve got two 7200s arriving this week also.

 

They won’t send SSDs as “they don’t know what the failure modes are” - whatever that means,

 

I’ll monitor our waits/loads before and after - I’ve never paid attention as I’ve no idea what scale or meaning they have, but I can compare before/after.