Smoothwall Caching Options

[DGardiner]

So just updated our smoothwall and this has appeared - or atleast ive not noticed it before?

 

Is there any reason not to just max the cache out (2Gb)? any recomended values here

 

 

 

[Joeloman]

The settings depend entirely on your hard drive and connection.

If you use an nvme pci ssd, it is probably always max because it is so fast.

But if you have a regular sata disk and have a very fast connection, it usually pays to reduce the size of the cache and also set the smallest object to at least 10KB, as it is often faster to download from the Internet.

 

Today, web pages change very often and bandwidth increases, which reduces the need for a proxy for reduced need for bandwidth.

[DGardiner]

The settings depend entirely on your hard drive and connection.

If you use an nvme pci ssd, it is probably always max because it is so fast.

But if you have a regular sata disk and have a very fast connection, it usually pays to reduce the size of the cache and also set the smallest object to at least 10KB, as it is often faster to download from the Internet.

 

Today, web pages change very often and bandwidth increases, which reduces the need for a proxy for reduced need for bandwidth.

 

NVME + 4gb internet and the machine has ram to spare, just trying to squeeze some extra performance!

[Joeloman]

Then it's probably just a matter of running max cache, then you can try setting 10KB as the smallest object.

To see if it gets better or worse..

Usually depends on the type of pages that are frequently visited and whether they are fast pages.

[milesla]

Depending on your raw speed and user base, just switch off caching as it can cause havoc with some sites we have found. We're on a 1Gb line plus a few thousand users 1:1 devices

[Joeloman]

I'll try to explain a little more:We in Sweden rarely have any problems today, as almost all schools have at least 10 Gb, but about 5-8 years ago 1 Gb more or less was standard.

So then we often had to make settings so that everything would work better/faster.

Today, the Proxy cache goes via RAM memory (previously via disk) RAM memory is probably always faster than downloading from the Internet.

(We have the cache on in systems with 400 Gb).

 

What can happen in a system if you increase all the values is that the RAM runs out and that's when the disk comes in again.

When you only have 1 Gb and a lot of users, the proxy cache is very useful, but you have to keep an eye on the total RAM usage if you have a regular disk.

Firstly, under Network settings, make sure that the ARP table size has the correct value (also consider phones and printers, etc.)

If you have many users, it can often be worth increasing the SYN backlog queue size.

The main thing that eats memory in Smoothwall is Guardian and the more different user groups, the more memory is used.

 

The best feature Smoothwall introduced to make the Internet respond faster is the System internal DNS server extremely many problems are solved by using this..

 

Smoothwall has a lot of good reports that can help and see how it looks with the memory, is it possible to increase it, does it memory is cheap and maybe you can switch to faster memory?

 

If we go back to the proxy:

In a system with 1 Gb and many users, I would set Global cache size: to max (was previously 1500 Mbytes)

Max and min object size that can be stored in the cache: Max object size: 20480 Kbytes (so that the cache is not filled by many large files)Min object size: 30 Kbytes (so that the cache is not filled with only small files)

You have to test yourself when it comes to min and max, no system is the same.

 

When we come to web pages that are not displayed correctly with the cache running, it is often home-made web pages today, the person who makes the page has "forgotten" to use "no-cache" in their code.

Try putting these pages under Do not cache these domains

Edited November 10, 2023 by Joeloman

[ibpalle]

I have stated using caching with min size 10MB and max size 250 MB - the intent is to cache larger files and not bother with smaller ones but I haven't done any testing on the performance as such, it just sounds like a reasonable thing to do

[tom_newton]

Some good points from my Swedish friend Joel, and our colleague ibpalle I would also throw into the mix that caching is surprisingly CPU intensive, so be sure to balance that out

[PotNoodleTech]

Sounds like Sweden is quite far ahead of us in the UK for connectivity then!

[localzuk]

Is caching much use these days when the majority of sites are served via HTTPS?

[Joeloman]

As long as Smoothwall is the man in the middle (certificate installed on the users), the content is read in HTTPS and then the cache is also used.

 

Yes, that's right, we have a lot of bandwidth in Sweden... sometimes preschools with 20 children have 10 Gb... as that is standard in that municipality.

Unfortunately, this means that everyone is extremely spoiled for bandwidth so everything from AP points to web filters must work without much delay.

Which often means that the customer uses their own servers with a lot of memory and powerful CPUs.

 

The ISP we cooperate with delivers at least 100 Gb to all new customers since 2019, however, the customer can choose a 10 Gb router.

 

I have private 1.2 Gb at home, via the broadband and will soon be able to get 5 Gb for the same cost.. if I want 10 Gb it will cost extra..

 

We also have good 5G in our larger cities, usually at 700-900 mbit, many like me have free surfing amount and extra data card for Chromebook, iPad, router in the caravan, etc. for £1.45 per month

(normal price about £7, but if you've been a customer since 1982 you have some extra benefits..)

Edited November 15, 2023 by Joeloman

[DGardiner]

Some good points from my Swedish friend Joel, and our colleague ibpalle I would also throw into the mix that caching is surprisingly CPU intensive, so be sure to balance that out

 

how much will this scale though, looking at our smoothwall vm with 40 cores there either seems to be only one thread in use and not much spread around

[DGardiner]

Is caching much use these days when the majority of sites are served via HTTPS?

just thinking more stuff like chromeos updates realisticly thats probably our biggest issue now, after haf term when the kids get in the schools and all turn on their chromebooks its hamering the internet (ive got an order in to raise the lines to 1gb but isp are dragging their feet)

[Joeloman]

Chrome's updates can use Smoothwall Proxy. See settings

Cache updates to reduce bandwidth

https://support.google.com/chrome/a/answer/3168106?hl=en#zippy=%2Cuse-peer-to-peer-automatic-updates%2Cspeed-up-updates%2Ccache-updates-to-reduce-bandwidth

[tom_newton]

So, it *is* possible to cache stuff still as Joel says, but it's increasingly hard. We use squid cache in Smoothwall and it's pretty single threaded (unlike the webfilter which is aggressively multithreaded) - and the reason the cache is pretty single threaded is cache consistency. This is why it's a hard balancing act to get the right proportion of cache and performance. We will be improving the mt performance of auth in maiden (probably GA in a few weeks), and then the subsequent castle will have a lot of updates (we're doing a big kernel and OS update).

[DGardiner]

webfilter which is aggressively multithreaded

 

Im seeing one dansguardian proccess hammering one thread and not much elsewhere, though there are multiple proccesses only one seems to be doing the chunk of the work?

[tom_newton]

Im seeing one dansguardian proccess hammering one thread and not much elsewhere, though there are multiple proccesses only one seems to be doing the chunk of the work?

 

Can you show me what it looks like? It's threads not processes - so as long as you're seeing 1 entry in top using more than 100% CPU things are good

[DGardiner]

Can you show me what it looks like? It's threads not processes - so as long as you're seeing 1 entry in top using more than 100% CPU things are good

so this look fine? its not the worst ive seen it but if it looks right!

 

[ibpalle]

That's what it looks like in htop - in top you would see a single process at 100+%. AFAIK here you see the individual threads. Naming is confusing in these apps sometimes.

[tom_newton]

That single core working hard is likely to be the one doing reporting - that can be multi-processed but I think the default is one