How slow is your Smoothwall? UTM or own hardware?

[lmgtfy]

I was wondering how many people use Smoothwall on their own hardware vs. how many use Smoothwall on a UTM?

 

Basically since April this year we have been having performance issues with our Smoothwall box causing it to either to drop the Web proxy service or cause web pages to take up to 2 minutes to load a single page.

 

We currently run the Smoothwall box on our own hardware which is a dual quad core Xeon running at 2.33MHz (Intel L5410) 8GB of RAM, and dual 1TB Barracuda ES.2 SATA drives, along with 4 Intel® PRO/1000 Network cards (e1000e)

 

We only see these performance issues when Smoothwall reports in User Activity that there are over 500-600 users using Smoothwall. We are currently licenced for 1120 as we have roughly 1000 computers/laptops/tablets plus we allow some use of BYOD. Does anyone have Smoothwall running with this amount of users and if so what do you run it on and do you have any problems.

 

I have being using Putty to run Top to see what might be causing these issues and this is what I find. Dansguardian3 runs at between 500-700% and datastore runs at 100% all other processes jump up and down never running above 100%. The memory usage is very small but then I guess this is because Smoothwall is only 32bit and can never currently go over 4GB anyway.

 

I have had a number of support calls open with Smoothwall since April and mainly waiting for the new database update in the Summer which was going to help with these load issues but unfortunately it has not. The latest news is there are no software problems and it is down to our hardware being not man enough to cope with the load. Ok fair enough I thought there are three ways we can go with. One buy the Smoothwall UTM3000, two buy are own better specification server, three look at other products.

 

Please don’t get me wrong I like the Smoothwall software what it does and when it works it works well in terms of functionality but we have had Smoothwall for 3 years now and although all these new updates come out bring new features it seems to need ever increasing resources. We first had the Smoothwall running on a Intel Core 2 Duo similar specs to UTM1000 which ran well for a year but then upgraded due to slow downs and now less than a year later we are looking at a massive upgrade again.

 

The UTM3000 sounds like a nice box albeit rather expensive at £5000 to buy but should Smoothwall really need such high specifications to run smoothly (pardon the pun) we currently have 8 virtual servers running on a server of this spec going from heavy use SQL servers to heavy use file stores and web servers it seems crazy. I don’t think we would buy the UTM either way because last week we were quoted £3000 for a Dell PowerEdge R620 running 16GB of RAM and Dual Xeon E5-2640 processors with all the bells and whistles including a 3 years NBD warranty.

 

Please can anyone offer any advice to the situation we are in? Having just paid out over £3000 for our annual Smoothwall costs I can’t see us getting another £5000 to pay for a hardware upgrade.

 

 

Thanks

 

Chris

[buzzard]

That's an interesting issue you have there, ours ran on a much lower specced box than that, albeit with far fewer users. I've had other solutions though on other boxes (in a previous post) with lower hardware specs and didn't have those issues. I guess rebuilding the box and reloading the config isn't an option, or have you tried that already?

[lmgtfy]

@buzzard Just interested to know what are the other solutions you have had running, at this point I'm willing to try anything. Looked at Lightspeed but they only do it as an appliance I believe and seeing as we have just renewed to Smoothwall we would not have this money again until September. As to rebuilding I would gladly do this in the half term but as support have told me its a hardware issue is there any point I ask myself?

[buzzard]

Just tried to look at the spec on the UTM3000 but gave up, wtf has happened to the smoothwall site! In one of my old posts (with about 1200 NOR) originally they had an RM Smartcache2 which ran well but was limited in capability, we then replaced it, but I'm struggling to remember with what, I have used Light speed and found it very good but there was another one we used.... hopefully it will come back to me! I have found Smoothwall the best all rounder, as I said I've only used that up to about 300-400 users but that ran on an old Dell Poweredge 840 again no reall issues on that, that (if i remember correctly) was a single quad core Xeon with about 8Gb Memory on a PERC5 controller (only one Intel 1Gbps NIC).? But it hardly taxed the system so I do struggle to see how/why your box is struggling so much, which I'd try a rebuild tbh, I did something similar to another smoothwall box because we couldn't work out a niggle, the rebuild and a restore of the config sorted out that issue.... If you're happy technically and have the time it's worth a try!

[lmgtfy]

The UTM3000 spec is I believe dual Xeon E5645 processors, 12GB of RAM (only 4GB would really be used at the minute being 32bit) and 2 1TB 15RPM and 4 Gbit network cards. Nice spec but £5000 I think I could buy two. Sorry

[tom_newton]

Chris,

 

There could be a couple of issues in play here.

First, we have seen connection tracking overloads on some sites - if you haven't already, go adjust the conntrack table (networking/advanced IIRC) - if it's set to 65000 or so (auto) double it.

 

If it's not that, there could be a few other things pegging out the box's processor. It does sound like you have an unusually busy system there. Would certainly be worth working out what's the bottleneck before laying out any cash either on our tin, or a 3rd party's (which may well be cheaper in some cases!)

[lmgtfy]

@tom_newton We did have the connection tracking problem and this issue was sorted out via a support call at the start of September but now the problem is when we have over 500 users page loads get increasing longer upwards of 2 minutes loading per page.

 

I have had a support call open (ref 00045814) and the last response I have had said the problem is purely down to our hardware not being up to the job. I would love to find out the bottleneck before we change hardware etc.

[Achandler]

Interestingly we had an ongoing issue with the web proxy failing which meant we had to restart the whole UTM1000 box fairly regularly. After a year of re-installing the box, even replacing it with a new box, Smoothwall support were unable to find a solution for us.

 

I never saw the processor spiking a lot though.

 

We "fixed" the issue by sorting our core switch out, we set up proper access lists etc. Since then the Smoothwall box didn't crash once, so it would appear there was to much traffic hitting smoothwall (a lot of it duff) which was causing the problem. We did however have our smoothwall UTM providing DHCP for our guest wireless, so it mgith have been guest wireless traffic that wasn't good.

 

Just a thought.

[tom_newton]

Thanks or that - it may be there's just not enough cores to handle the load after all - but I will check out the ticket.

[tom_newton]

To be fair it seems that support may have got this one right - the box is often seeing 600% CPU usage in Guardian alone, and load averages of 10+ on an 8 core box. Load average is "number of tasks waiting for CPU" - you have effectively 8 CPUs with 10/11 tasks waiting at busy times, so 2-3 are going to end up not served in a timely manner. You may just have a great deal of traffic - is there perhaps, as a previous poster speculated, one or two users applying dubious workloads?

 

We do have the 64bit release coming in a week or so, but I am not sure it will do a lot for you, given that your machine is already pretty effective in using its CPU cores to the max.

[buzzard]

@tom_newton why is such a high spec machine required to do this job? I know when I've used SW and other filtering solutions it never needed such a beefy box? Has there been more features in SW recently or is it possible to turn off features not being used, I'm not disagreeing with your post, I'm just curious!

[tom_newton]

@buzzard - it isn't, generally - it must be that the OP is shifting a serious amount of traffic. In actual fact Guardian has become slightly more efficient over the last 18 months.

[zag]

Check for users on your network using P2P software such as torrents.

 

We currently have a problem with a few BYOD users using 90% of our bandwidth and inevitably the CPU on the filter that goes along with it.

 

Check the reports for any sites that are blocked millions of times.

[twin--turbo]

What sort of site has 500-600 constantly bashing the filter? is this realy the true concurrancy?

[buzzard]

There does seem to be something amiss here, do you have any traffic management going on on the core network at all? where is all that traffic coming from? I'll see if I can look at my old place with SW setup to see what they have concurrently and compare to server performance, @tom_newton does SW have some metrics to compare this against?

[twin--turbo]

Also what are the top 10 sites for hits and bandwidth.

[lmgtfy]

@tom_newton I believe we still have the same amount of computers/tablets/laptops around 1000 but there could be two load differences between now and two years ago. The first being that we now have a 100Mb leased internet line whereas before we had 30Mb so perhaps the demand for internet is instead of be held back by our external connection perhaps smoothwall is having troubles keeping up (I doubt this one is true) and the second being BYOD which as I'm sure you know has taken off. We only have around 50 BYOD "surfing" the internet at once but a lot of devices are automatically connecting to the wireless where smoothwall is the DHCP on this VLAN. In regards to dubious activity I hope not we have everything locked down quite tightly but I will certainly try and monitor activity more closely.

 

@twin--turbo we are quite a large school here 2800 users, 1000 devices easily and during peek times we do see 600 users accessing the internet be it for simple Google research or more bandwidth eating activities such as MyMaths the top 10 sites today are as follows:

 

Hits

http.00.s.sophosxl.net 262686

kh.google.com 196649

Google 183069

encrypted-tbn0.gstatic.com 111055

encrypted-tbn1.gstatic.com 110999

encrypted-tbn2.gstatic.com 110962

encrypted-tbn3.gstatic.com 110608

clients1.google.co.uk 90338

static.bbci.co.uk 50298

 

Bandwidth

Google 6.42GiB

kh.google.com 1.50

s.ytimg.com 0.92

apis.google.com 0.90

http://www.reactiongifs.com 0.65

bits.wikimedia.org 0.52

cdn-the-2.musicradio.com 0.51

downloads.bbc.co.uk 0.49

r14---sn-aigeznlz.c.youtube.com 0.48

 

I think it may be case of switching off BYOD for a couple of weeks see how many people it annoys and two how much faster all other traffic goes.

 

Thanks for all your help and suggestions everyone. As I have said I like Smoothwall and the amount of time I have put into it setting up all the different features and polices as well as learning the interface I don't really want to move away as I still think the functionality is the best in class but we do need to resolve the issue.

[tom_newton]

Try turning off the "sophos web protection" feature on your desktop AV - it looks like it is responsible for a *lot* of requests.

 

Can you do 2 things for me - confirm the processor you are running on, and run "top" on the commandline at peak time, and tell me how much CPU time squid is using.

 

I suspect the difference between now and a couple of years ago is the number of web requests needed to complete the same "job" - google has doubled in "weight" over just a few years, and the likes of BBC are considerably request-heavier too, with a lot of in page updates.

[buzzard]

Wow that sophos one has quite a few hits...... also 2800 kids, fair enough I bet you've your hands full!!! Watching with interest where this ends up!

[andyrite]

 

We do have the 64bit release coming in a week or so, but I am not sure it will do a lot for you, given that your machine is already pretty effective in using its CPU cores to the max.

 

Will this be an upgrade or new install?

[Gaz]

Seems your not the only one that has problems with the Sophos requests. It appears to be related to the web protection feature.

Excessive DNS lookups from sophosxl.net - SophosTalk community

 

Perhaps its Sophos related?

[twin--turbo]

Seems your not the only one that has problems with the Sophos requests. It appears to be related to the web protection feature.

Excessive DNS lookups from sophosxl.net - SophosTalk community

 

Perhaps its Sophos related?

 

What you think an AV program may becausing massive trouble with computer performance... Never heard that before LOL...

[Gaz]

Almost sounds like sarcasm.

[lmgtfy]

@tom_newton Thanks for your help I will try turning off Sophos web protection and see if that helps at all. The processor we have currently is a dual quad core Xeon running at 2.33MHz (Intel L5410) its not the newest thing in the world and your right the internet has changed a lot generally in the way it was used from two years ago so if we need to beef it up then that's what we will do. I guess the only concern is not ever being able to tie down exactly what is causing the big CPU loads yes dansguardian is running at 600% but that's all we know. Will run top tomorrow when busy to see what squid is doing but I can't ever remember it being up there in the top 3 CPU hogs but I will check and let you know.

 

Cheers

[zag]

What you think an AV program may becausing massive trouble with computer performance... Never heard that before LOL...

 

I still have no clue why anyone uses 3rd party AV stuff. Its a complete con these days.

 

We use Microsoft forefront endpoint protection and it works almost invisibly on our systems.