Smoothwall S9 filtering randomly falls over

[timbo343]

I need to get this out with the hope someone other than smoothwall might be able to help because im on my knees with this smoothwall box.

 

We have 4 month old Smoothwall S9 running Leeds 61 and between the hours of 9pm and 3pm the thing like to play the game of "i'll stop filtering and kill the proxy causing it to fail" however if we bypass the proxy everything is ok.

 

There is no set time or pattern to this. Today it decided to fall over at 9.34am then at 10.40, 10.51 and 11.42. Yesterday it fell over twice and wednesday it fell over 5 times all random times, times we are unable to pinpoint any major events.

 

Each time it falls over we have to either do a soft reboot or the soft reboot hangs on "Azure indicators not dectect" and we have to pull the power lead out the back.

 

Smoothwall have looked over the box time and time again and have been on the box whilst it's fallen over yet the box gives no warning to when it's about to fall over.

 

All we get is no web pages load and then a series of redis errors with no error code.

 

We have tried all sorts. We think we have solved it until it falls over again. I think we are up to Plan AG at the moment and whatever we do does not have an effect on the box.

 

To be honest it would be a great miracle if we had one full day whilst school is on flow and the box stayed up.

 

It's not like the box ia under load or the internet lines are busy, we have 2 FTTP lines running at around 900mbps each. Whivh are hardly being pushed to their limits.

 

I connected into work the other day at 5.30 and replicated 185 users logging on and browsing the net at once, it didnt fall over.

 

It doesnt fall over on an evening (tell a lie, it gave a couple of redis errors on Monday 5th September around 18.15 and 21.45)

 

It's doesnt always fall over between 9 and 3, it could be between 11 and 2.

 

We are seeing an issue on a few user accounts the 9 google app dots give an error but we dont know if that is related. Asking the user to move computers, resetting their AD or asking to try a different browser makes no difference, yet the users that have the issue say it works fine at home.

 

We are running the latest versions of chrome and edge.

 

Ive rebuild the box with a clean ISO Leeds 57, updated it to Leeds 61, restored my config and the issue remains.

 

I'm on my knees with this now and the school is starting to get pig sick of it.

 

I'm receiving an S9 on Wednesday to see if it's the box. If it still does it, it could be updates 58, 59, 60 or 61 or my config. We never had this issue one leeds 57.

Edited September 16, 2022 by timbo343

[Smokebomb]

We are having a similar issue with our appliances since the start of term (and also on Leeds 61). We are working with support to try and get to the bottom of it and I’ll share any solution we receive with you here.

[timbo343]

We are having a similar issue with our appliances since the start of term (and also on Leeds 61). We are working with support to try and get to the bottom of it and I’ll share any solution we receive with you here.

Thank god im not alone... if i find a solution i shall post back too but Smoothwall seem to be all out of ideas at the moment.

 

I do have another thread on here about a random NT Auth\Anonymous Logon issue that seems to be hitting us at the moment - not sure if it's related (https://www.edugeek.net/showthread.php?t=229387)

 

I have a test box coming from Smoothwall on wednesday so see how that fairs, though if i can, im tempted to keep that on Leeds 57 if at all possible - it all depends if i can restore my leeds 61 config files to leeds 57 because a few bits changed over summer such as VLANs, HTTPS cert, general config bits.

[Smokebomb]

My current suspicion of the cause of the crashes is our BYOD devices, as this only started up again when the students returned. We have over a 1000 devices connected to our guest WiFi at any one time during the week (we’re a college). The crashing started with the block page server locking up; we can see this in the system report, filtering on web filter. Next week we are going to try the advice in the following kb to drop connections rather than displaying the block page on our guest WiFi vlans: https://kb.smoothwall.com/hc/en-us/articles/360005418299-Dropping-the-Connection-Instead-of-Showing-the-Block-Page Edited September 18, 2022 by Smokebomb

[timbo343]

My current suspicion of the cause of the crashes is our BYOD devices, as this only started up again when the students returned. We have over a 1000 devices connected to our guest WiFi at any one time during the week (we’re a college). The crashing started with the block page server locking up; we can see this in the system report, filtering on web filter. Next week we are going to try the advice in the following kb to drop connections rather than displaying the block page on our guest WiFi vlans: https://kb.smoothwall.com/hc/en-us/articles/360005418299-Dropping-the-Connection-Instead-of-Showing-the-Block-Page

Ah yeah we had that issue on our S8. When we looked at the SSH CLI to see what was spiking the loads, we saw the blockpage service flood the system.

 

We made alterations to the blockpage module for our BYOD and LAN networks and from what i remember this helped on the loads..

 

If you have access to the CLI SSH on the box, once logged in, type top and then press 1 on the keyboard. This will show the top processes and also give you stats of what each of the processor's loads are at.

Edited September 18, 2022 by timbo343

[Chclark]

We had a issue at one of my school's a while back where the smoothwall would fall over, after alot of trial and error it turned out it was the schools CCTV they had at the time it was a web-based one which wrote hundreds and hundreds of little files back over the internet at the smoothwall would get overwhelmed, smoothwall confirmed this so we sections the cameras off and bypassed the smoothwall all was fine then. You haven't got anything similar going through it have you?

 

The only other odd one we have at the moment is another site which if any port using any authentication is used we start seeing drop outs they have to be pushed onto a port which sets filting on one of the groups which isn't ideal. Still waiting for some answers on this, doesn't matter the authentication ntlm or Kerberos they all do the same.

[timbo343]

We had a issue at one of my school's a while back where the smoothwall would fall over, after alot of trial and error it turned out it was the schools CCTV they had at the time it was a web-based one which wrote hundreds and hundreds of little files back over the internet at the smoothwall would get overwhelmed, smoothwall confirmed this so we sections the cameras off and bypassed the smoothwall all was fine then. You haven't got anything similar going through it have you?

 

The only other odd one we have at the moment is another site which if any port using any authentication is used we start seeing drop outs they have to be pushed onto a port which sets filting on one of the groups which isn't ideal. Still waiting for some answers on this, doesn't matter the authentication ntlm or Kerberos they all do the same.

Funny you should say that about the CCTV. We've had a new HikVision installed over the summer. It's on it's own network (even with it's own switches so completely away from our network) and it has 2 NICs in it. One configured for the Camera network and the other on a VLAN on our network.

 

Apparently it's not using HikCentral and was only allowed access out to the net via the smoothwall for windows updates, then all traffic was blocked, apart from traffic to the TeamViewer ports that they requested to be opened - i might pull the patch lead for this box incase it's something coming from the CCTV server.

 

The only other thing we installed over the summer was Barracuda Asset Manager Scanning that a company is doing an audit of our devices across the trust.

[Chclark]

I wouldn't expect hik to cause it if you have a DVR these where cameras that saved to the cloud and wrote the files as millions of images and this would crap out the smoothwall. Edited September 19, 2022 by Chclark

[jontyplatt]

I have recenlty changed my Smoothwall box to a 10000rpm as both SSD and a 7200rpm proved unreliable.

[ebeecroft]

Following with interest. We have had no end of ongoing problems with our S10 (currently Leeds-61) and have been unable to pinpoint the issue. I did wonder whether a rebuild might sort it but reading here makes me think it may not. We have had the Smoothwall disks upgraded from their original CCTV tier disks to enterprise 7200RPM versions. Edited September 20, 2022 by ebeecroft

[ibpalle]

The 'Drop connection' blockpage option can be useful with large BYOD numbers. A lot of the categories like 'Computing' and 'Social Networks' etc generate a lot of blocks from devices without the CA installed for example. None of those types of applications are capable of showing a blockpage anyways so applying the drop connection blockpage option to categories that show a large number of blocks on your BYOD network can be a good resource saver. The Smoothwall does not have to generate sometimes millions of blockpages that no-one ever sees.

 

Find the categories causing the most blocks, then go to the Guardian - blockpages - manage policies and add a new policy to use the 'Drop connection' option for the blockpage, applied to the BYOD location and the categories found to generate the most blocks.

[Smokebomb]

Had to failover at 9am today, then again at 9:23am. No solution from Smoothwall yet. How are things going for you timbo343?

[timbo343]

Had to failover at 9am today, then again at 9:23am. No solution from Smoothwall yet. How are things going for you timbo343?

 

Heya!

 

Had the webproxy / guardian fall over this morning at 9:08am.

 

I've restored to blocking MAC addresses on the wireless network incase it was a student trying their luck.

 

I've got Smoothwall Level 3 looking at this now.

 

We know it's not the box or leeds57 so we are now thinking it's something on the student machines or even student profiles because it's happening when the students are on the computers - not so much logging on though.

 

We are running Windows 10 LTSC2021 and we have Sophos running too.

[timbo343]

We have had the web proxy fall over again at 10:03 and the script command Smoothwall gave us didnt work so had to forcefully reboot the smoothwall though we didnt have to pull the powerlead.

[timbo343]

Our box failed at 10:45am again and had to reboot it.

 

We have seen a lot of traffic to play.google.com when using Chrome browsers and we have play.google.com blocked for students. Since 11am, we have auth-bypassed play.google.com. We blocked it for students because they were finding playable games on the site.

[timbo343]

Our box failed at 10:45am again and had to reboot it.

 

We have seen a lot of traffic to play.google.com when using Chrome browsers and we have play.google.com blocked for students. Since 11am, we have auth-bypassed play.google.com. We blocked it for students because they were finding playable games on the site.

 

It's not play.google.com - web filter fell over at 12:10

[timbo343]

We are now trying Sophos.

In Central, Computer policies > Threat Protection > Base Policy > we have disabled "LIVE PROTECTION" and "Real-Time Scanning - Internet" to see if this makes any difference.

[ChrisC]

I've seen some appliances update to Leeds 62 overnight and it's in the update notes that under certain conditions the guardian service could error out/stop responding.

 

Could be worth trying to update?

[timbo343]

I've seen some appliances update to Leeds 62 overnight and it's in the update notes that under certain conditions the guardian service could error out/stop responding.

 

Could be worth trying to update?

 

Thanks, i have a box which is on 62 and that did not stop the issue. I'm thinking the issue is Sophos. we haven't had the proxy / webfilter / guardian fall over since 12:10 this afternoon. We were expecting to fall over around 14.15 to 14.30 but it never happened.

 

Let's see what tomorrow brings.

[ChrisC]

If that seems to have helped, (if you haven't already) it could be worth adding sophos.com to a do not filter rule along with other related URLs/domains it could be calling as part of the real-time scanning.

 

If every URL is being passed to/looked up by sophos then I theory I guess you're at least doubling the amount of processing required by the smoothwall per address visited?

[timbo343]

If that seems to have helped, (if you haven't already) it could be worth adding sophos.com to a do not filter rule along with other related URLs/domains it could be calling as part of the real-time scanning.

 

If every URL is being passed to/looked up by sophos then I theory I guess you're at least doubling the amount of processing required by the smoothwall per address visited?

We run a Caching and message relay server onsite so wonder if it has something to do with that [emoji2371]. I have a Sophos Central Custom Category which is already set to Do Not Filter for Everyone / Everywhere.

 

We've had issues with sophos in the past so i wont be surprised if it is Sophos - in saying all this, it might be another red-herring and might have to go back to the drawing board in the morning.

[ChrisC]

Do you have a source exception in guardian for the caching server?

 

I'm assuming you don't see a huge volume of your traffic actually coming from that box according to usage reports etc on the smoothwall?

[timbo343]

Do you have a source exception in guardian for the caching server?

 

I'm assuming you don't see a huge volume of your traffic actually coming from that box according to usage reports etc on the smoothwall?

I have the server listed in as a source exception but dont configure port 801 on the server. Sophos Central is in the exceptions list for the wrb proxy too - we have sophos.com and all the other sophos domains added into the same category.

 

We have seen a lot of activity from out Caching server to ncs2-cloudstation-eu-west-1.prod.hydra.sophos.com which is obviously Sophos and i guess that is Central.and will be telling sophos who the last user was etc.

 

I mean when 300 students use the computers that server will be busy sending packets out to sophos.

 

Let's not count our chickens just yet.

[ebeecroft]

Interested to hear how you get on today Tim. Fingers crossed it doesn't fall over with Sophos turned off.

[timbo343]

@ebeecroft - see our thread on the NYNM - it fell over again at 9:19am this morning.

 

The script that smoothwall gave me has been edited to restart Auth and the script worked a treat.