Proxy performance hit after bandwidth upgrade?

[pete]

So we got upgraded to 10mb yesterday (which works out at 4-6mb once it's past the border firewalls) and all was well, many isos were downloaded.

 

Until about 11am this morning when the proxy started grinding like a ...grindy thing.

 

The proxy is well past warranty and replacement date and I've finally got time to start replacing it, but did anyone else notice proxy thrashage when their bandwidth got upgraded?

 

No the kids don't know about the upgrade, so deliberate thrashage is unlikely.

[webman]

Which proxy OS+software/what spec server is it running on?

[localzuk]

Also, is it a dedicated 10mb or a shared one? (ie. do you connect to another organisation for your pipe)

 

We have a 10mb pipe at our school, which is connected to the local college which has a 100mb pipe out. They share theirs with 3 other places as well as us and have 1000 or so students themselves. Our connection is not dedicated 10mb but rarely dips so low that you'd notice it being 'slow'.

[Geoff]

We have a 2.5Ghz Celeron with 1.5G of ram running Ubuntu Linux, Squid and Dansguardian. It copes quite happily with a 100mbit link.

[pete]

Server is a ML350 with dual P3 800s and 1.5G ram, single 100mb NIC and running scsi disks - it's an old Netlinc everything + kitchen sink job

 

it's running nt4, with exchange 5.5

Proxy is Symantec Web Security 3.0

>

 

Pipe is via EMBC and is 10MB to Netlinc / EMBC and then we share whatever pipe they have with other embc schools (roughly 4-6mb on the Internet, as mentioned).

 

The proxy handled the load fine until yesterday and is still puttering along at about 20-50% cpu use and 700MB ram +/- 50mb. This is normal, but site access is fabulously intermittent.

 

Access via a separate Netlinc proxy is also slightly intermittent (was also fabulously intermittent earlier today).

 

Rebooting the server did make site access slightly more reliable for about an hour over lunch time, but it then slowed again. Some sites will nearly always render ok (yahoo), others (theregister.co.uk, google, bbc) hardly ever do.

 

It's doesn't appear to be a DNS issue, since running nslookup in debug mode on the proxy provides the correct results in an acceptable amount of time.

 

The connection itself doesn't appear to be flaky either, since I can get isos via ftp without any problem (no, I'm not stealing all the bandwidth )

 

The only thing that's changed wrt the proxy and network this term is the changing over from a 2MB BT box to a 10MB fibre connection plus new router.

 

I'm waiting for Netlinc & embc to come back to me.

[Ric_]

@pete: Perhaps the speed with which the data is now coming in is too much for your poor old box to process?

[Geoff]

Yes, sounds like an IO bottleneck to me.

[pete]

That's kind of the conclusion I'm coming to. T' Internet has become reliable since the kids have gone home. I've increased the concurrent connections limit in SWS, so we'll see if that helps as a stop-gap (at the expense of CPU and ram usage) until I spec out the new server.

 

I'm waiting on embc to get some info as to whether us --> our proxy --> border firewall --> Internet is now routed differently to us --> Netlinc proxy --> border firewall --> Internet.

 

But I think you guys are right.

[pete]

@Geoff

Are you doing running ClamAV or similar on that Dansguardian box?

 

Whatever I replace it with, I'd like scanning (yes, I'll take a performance hit), we currently have two sets of av defs - symantec on the proxy and mcafee on everything else.

[Geoff]

Yes, ClamAV. It does complicate the setup though, I'd recommend you hold off on the idea until you have basic filtering working.

[tom_newton]

We find that turning on Clam in SchoolGuardian hits the CPU quite hard, it does relieve some of the load on disk, as the system tends towards cpu-bound.

 

You can probably revitalise this box with some more modern, quicker disks anyway.

[FreeWill]

I think the issue you have with your net is a Netlinc issue.

 

We are also a partially Netlinc school - we have a proxy the same as yours, but we left them approx 2 years ago for everything apart from internet connectivity. We got upgraded to 10Mb BT/Kingston Fibre at the end of last year, and our internet seems to be slow, certainly not 10Mb 1:1 speed.

 

The connection we have between us and Netlinc certainly runs as near to 10Mb as it can, but I'm sure that Netlinc's connection to the net isn't fast enough to serve all the schools at the full 10Mb.

 

During peak times all the schools in the area are pulling such a huge amount off the net it tops out - I find after 3.30ish the net becomes faster, when the Primary schools finish.

[pete]

Spent most of today either on the phone with Affiniti / embc or in the server room fscking with the proxy. And the result? None of us know wtf is going on.

 

From their end, proxy looks fine, pulling 6mb-ish and from my end I'm not getting bad packets, high cpu load or disk thrashage. This morning it was fine until 12:45, when it just went slow - 1 in 10 pages loaded.

 

But there's no _reason_ it did that at 12:45 - the bandwidth and computer usage was _exactly_ the same as at 12:10 (when period started), if not lower.

 

I've got one last trick to try and then I've run out of ideas.

 

I am now going to blow up spaceships.

[kylewilliamson]

Change your lan card.

[pete]

Guess what? $school_near_us has been having the same problem since their 2mb got upgraded to 10mb.

 

I'll still swap the nic when it goes down today, but I sense a cockup.

 

smt are asking if they can use the nuclear option --- I'm very tempted to let them.

[Geoff]

Sounds like there's a bandwidth contention issue at your ISP/LEA/RBC.

[pete]

Aye, that's what I pointed out to the helldesk.

 

I've also told them to find out how quickly they can switch us back to the working 2mb connection - it's better than a 10mb connection that doesn't.

 

Edit: @FreeWill - have you nagged Netlinc / EMBC about this recently? Would you mind doing so again?

 

And the power has just gone out.

 

edit02: The uplink interface at the exchange is set at 10mb, when it should be 100mb - every school around here is competing for 10mb.