Smoothwall - high processor load averages

[timbo343]

This week we have seen the Processor Load averages hit around 15 / 16 and the only way to bring them back down is to restart the device.

 

Currently it's sat around 8 / 9 and has today averaged around 11 / 12.

 

Nothing has changed on the smoothwall so i'm struggling to find out where the issue is sitting.

 

It normally sits at around the 3 / 4 mark so it's higher than usual. We are on update Leeds-55

 

Wondered if anyone else's is running high.

[MartinT]

Check your System logs to see if you're getting NIC drops. This happened to us and caused high load averages necessitating lots of web proxy restarts.

 

The culprit was a bad Linux NIC driver causing it. Smoothwall had to dig in and patch our S8 appliance.

 

Our load average is currently 2.7.

[timbo343]

Check your System logs to see if you're getting NIC drops. This happened to us and caused high load averages necessitating lots of web proxy restarts.

 

The culprit was a bad Linux NIC driver causing it. Smoothwall had to dig in and patch our S8 appliance.

 

Our load average is currently 2.7.

 

Thanks, i've spoken Smoothwall about it and sent across my system logs. All the logs look fine. The only thing i could see that happened when the processor average increased was on WEB FILTER:

 

05:02:17 Web filter Reclaiming memory from old session due to timeout being reached

05:02:19 Web filter Reclaiming memory from old session due to timeout being reached

08:19:04 Web filter All sockets in session ignored due to closure/error state

09:10:20 Web filter Unable to connect to redis. Error: events: revents: connecting,timeout

09:10:20 Web filter Failed to connect to blockpage server

09:10:20 Web filter Unable to connect to redis. Error: events: revents: connecting,timeout

 

Other than that, i cannot think what the issue could be.

 

I would have expected a processor load average around 3.

[Mako]

What are the disk wait times? The amount of remedies over the years to reduce the stress on the Smoothwall does get tiresome. The latest thing was the whole HDD/SSD debacle. Dropping bad traffic, reducing logging and catching troublesome software have all been contributing factors in reducing the load.

 

I can't remember what site it was, it was during Lockdown 1.0, but whenever a user was on the website it was constantly sending data to the device. A class of kids would generate thousands of hits a minute and I had to end up putting it into auth bypass to stop the logging. Some kind of maths site, if I recall.

[slugshead]

our s14's sit at around 4.0-20.0

 

But if you run Top when ssh'd in, datasto+ is never less than 99% and guardian is never below 50%, snort is always around 50% too

[timbo343]

We keep getting a load of these in the MONITOR logs:

 

Feb 8 13:30:18 firewall Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources. /var/log/dansguardian3/access.log

Feb 8 13:30:19 firewall Monitor Notice: System message rates have slowed, resuming normal operations. /var/log/dansguardian3/access.log

Feb 8 13:31:47 firewall Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources. /var/log/dansguardian3/access.log

Feb 8 13:31:48 firewall Monitor Notice: System message rates have slowed, resuming normal operations. /var/log/dansguardian3/access.log

Feb 8 13:33:21 firewall Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources. /var/log/dansguardian3/access.log

Feb 8 13:33:22 firewall Monitor Notice: System message rates have slowed, resuming normal operations. /var/log/dansguardian3/access.log

Feb 8 13:36:00 firewall Monitor 1F:133600:Warning: System load average is 9.37 9.23 8.78

Feb 8 13:36:39 firewall Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources. /var/log/dansguardian3/access.log

Feb 8 13:36:42 firewall Monitor Notice: System message rates have slowed, resuming normal operations. /var/log/dansguardian3/access.log

Feb 8 13:39:10 firewall Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources. /var/log/dansguardian3/access.log

Feb 8 13:39:12 firewall Monitor Notice: System message rates have slowed, resuming normal operations. /var/log/dansguardian3/access.log

Feb 8 13:40:58 firewall Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources. /var/log/dansguardian3/access.log

Feb 8 13:41:00 firewall Monitor Notice: System message rates have slowed, resuming normal operations. /var/log/dansguardian3/access.log

 

Yet when we look in the system logs there isn't a lot of messages.

 

The drives were changed a couple of years ago when the boxes came with 5400rpm drives and im not sure on the wait times.

 

Here is a screengrab of the processor utilisation and you can clearly see that something today triggered something off but what i don't know!

 

[timbo343]

our s14's sit at around 4.0-20.0

 

But if you run Top when ssh'd in, datasto+ is never less than 99% and guardian is never below 50%, snort is always around 50% too

 

Here are the TOP processes:

 

top - 14:15:18 up 4:48, 1 user, load average: 4.43, 6.74, 7.83

Tasks: 298 total, 2 running, 295 sleeping, 0 stopped, 1 zombie

%Cpu(s): 31.0 us, 6.5 sy, 0.0 ni, 51.5 id, 9.4 wa, 0.0 hi, 1.6 si, 0.0 st

KiB Mem: 16373968 total, 13672056 used, 2701912 free, 67904 buffers

KiB Swap: 7994364 total, 368936 used, 7625428 free. 9464600 cached Mem

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

11706 datasto+ 20 0 218924 119644 5368 R 98.2 0.7 143:50.61 datastore.+

6896 guardian 20 0 2706008 343836 9436 S 70.9 2.1 217:49.20 dansguardi+

20069 root 20 0 528040 470852 5252 D 33.6 2.9 1:26.75 datastoret+

11507 smooths+ 20 0 308504 159008 10008 S 13.0 1.0 35:48.92 smoothwall+

7255 auth 20 0 102860 49340 3964 S 11.3 0.3 37:48.58 authd4

6876 squid 20 0 316060 222832 11396 S 5.3 1.4 14:10.44 squid

6875 squid 20 0 281528 187648 10712 S 2.0 1.1 10:11.17 squid

3167 root 20 0 118820 17732 5328 S 1.3 0.1 1:39.56 networkmod+

[timbo343]

So now school has finished the processes are down at below 4 now..

 

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

6896 guardian 20 0 2706008 341300 8568 S 12.3 2.1 244:05.70 dansguardi+

11706 datasto+ 20 0 218924 119356 5368 S 35.9 0.7 218:15.17 datastore.+

11507 smooths+ 20 0 308108 158328 10228 S 33.3 1.0 46:58.68 smoothwall+

7255 auth 20 0 102996 49452 3964 S 2.0 0.3 42:41.86 authd4

6876 squid 20 0 316056 224268 11604 S 1.3 1.4 15:57.46 squid

6875 squid 20 0 281528 189272 11144 S 0.7 1.2 11:27.92 squid

[MartinT]

We just upgraded to a new S9 appliance yesterday (from an old UTM1000). Now the Load Average is sitting at around 1.2 and page responses are lightning fast.

[kennysarmy]

Issues with our Smoothwall on-prem today.

 

Seeing similar messages as above in the logs.

 

12:23:28 Monitor Caution: Large amounts of system messages are being generated, entering burst processing mode to preserve system resources.

 

Awaiting second line to call us back, currently trying to get hold of Smoothwall support but no one is answering the phone, I've been trying for 20 minutes now!!!

@ibpalle @tom_newton

[tom_newton]

Other than the error messages, what issues are you seeing?

 

If you can ping me the ticket # by email that would help