Squid physical configuration

[Ignatius]

I'm looking into configuring Squid on a network that is used by about 20 clients. I plan to have the Squid PC with a static IP address on the network (172.16.0.0/16), configure the clients to use the proxy and configure the ADSL router to block all internet traffic except from the Squid PC. That bit's OK.

 

My question relates to what happens if the number of clients increases significantly because the single Gb network card would have a lot of in/out traffic. The possibilities that I have thought of are:

 

1. Have a second (third or fourth) Gb NIC in the Squid PC (say, 172.16.0.250, 172.16.0.251, 172.16.0.252 and 172.16.0.253) and configure the appropriate proportion of the clients to use the different NICs, hence providing balancing.

 

2. Have a second Gb NIC in the Squid PC - one on the 172.16.0.0/16 network and the second connected to the ADSL router via a 192.168.0.0/30 network. I realise that I would have to provide routing in the Squid PC and suspect that I'd have to provide NAT.

 

Which of these solutions would be preferable? I suspect that the former would be more simple. If the latter solution would be used, would it kick in on a network of a particular size (i.e. number of clients) or would there be some other time when I should (or must) use such a configuration?

 

Finally, other than having several Squid proxies, are there any other solutions that would be necessary in a large network?

[ChrisH]

I'm guessing you could just use different ports with squid rather than loads of NICs, especially where you only have 20 clients.

[glennda]

I doubt you need to worry about the throughput of a 1gb network card. I currently have about 600+ pc's going through one squid box. then about 200+ on another (due to different filter policies for staff/students). I have never had a problem with the capacity of the squid box more the internet connection on the other side. I'm assuming you don't have a 1gb internet connection!

[Guest Guest]

Unless your internet connection is >1gb then you don't need to worry, thats where your bottle neck is going to be. The only reason I can think of to have more than 1gb of bandwidth on your squid box would be if you cached high bitrate media and had tonnes of clients.

[MatthewL]

I have 1000+ devices connecting to my server on a 1Gb connection and its fine, I wouldn't worry about it.

[Ignatius]

I'm grateful for everyone's comments and am reassured that I wouldn't have any problems with the amount of traffic passing in and out through a single 1Gb NIC. I'm relieved because I didn't particularly want to have to delve into configuring iptables!

[tom_newton]

Yeah - the NIC'sthe last of your worries, virtually every subsystem would flake before the NIC choked. That, however would be a long way off 20 PCs, or probably 2000

[Ignatius]

Yeah - the NIC'sthe last of your worries, virtually every subsystem would flake before the NIC choked. That, however would be a long way off 20 PCs, or probably 2000

I realise that a network of 20 PCs would manage without any problems. I was really "pontificating" about what would be appropriate in a much larger network, such as a school, college or university where the number of clients could well be into four figures.

[dhicks]

Have a second (third or fourth) Gb NIC in the Squid PC (say, 172.16.0.250, 172.16.0.251, 172.16.0.252 and 172.16.0.253) and configure the appropriate proportion of the clients to use the different NICs, hence providing balancing.

 

If you're going to have multiple NICs, you could just team them - I think that provides both fail-over and load balancing, all in one easy-to-do setup. As already pointed out, though, no matter how many client computers you have there's little point having a network connection to your router that's faster than your Internet connection. I suppose you might see some improvement from a faster network connection to your Squid box if you were getting a lot of cache hits - maybe a whole class viewing a YouTube video or something?

 

--

David Hicks

[tom_newton]

I realise that a network of 20 PCs would manage without any problems. I was really "pontificating" about what would be appropriate in a much larger network, such as a school, college or university where the number of clients could well be into four figures.

 

Admittedly most of the systems I see have filtering - which will discourage saturation of the NIC, but I have seen one of synetrix's squid test boxes. They have quality Gig network cards, but the throughput was limited by other aspects - which is why each box has a couple of top end xeons and a bunch of solid state storage and 10s of gigs of RAM. For this reason, it is generally the case that cache/filter boxes end up clustered before you do anything with the >1 NIC. Additionally, of course, in larger networks, a cluster gives resilience.

 

The only recent >1 NIC install we have done teamed 2 NICs for failover, not performance - this was in a large cluster already.

[Ignatius]

If you're going to have multiple NICs, you could just team them - I think that provides both fail-over and load balancing, all in one easy-to-do setup.

I suppose my initial theoretical way of dealing with a large number of clients would be to configure 25% to have one of the four NIC IP addresses as the proxy, hence reducing the traffic that would be borne by a single 1Gb NIC. I appreciate the comments in this thread though that my over-complicated solution wouldn't be necessary in the hypothetical scenario that I proposed.

[Ignatius]

They have quality Gig network cards, but the throughput was limited by other aspects - which is why each box has a couple of top end xeons and a bunch of solid state storage and 10s of gigs of RAM.

Before I started this discussion, I hadn't really thought of other areas that might create a bottleneck. I understand now that, in the mega network scenario, all hardware components must be capable of high throughput. A chain is, after all, as strong as it's weakest link.

[tom_newton]

Before I started this discussion, I hadn't really thought of other areas that might create a bottleneck. I understand now that, in the mega network scenario, all hardware components must be capable of high throughput. A chain is, after all, as strong as it's weakest link.

 

Spot on. It's actually remarkably difficult to saturate a modern NIC in all but the most network heavy of scenarios. Some interesting corner cases arrive when you've lots of little packets though (say heavy heavy voip users - we're talking carrier grade stuff here).

[Ignatius]

I wasn't sure whether to start a new thread ... I'm happy to do so if necessary.

 

I've made progress in configuring Squid. A single NIC (eth0 172.19.0.250/16) and using port 3128. I blocked all outward traffic in my ADSL router except from the Squid box and working everything's fine.

 

The next stage was to make it transparent. Here's what I've done:

 

I changed the client Default Gateway to the Squid IP address

I removed the proxy in my web browser then restarted it

I changed squid.conf to have the line "http_port 3128 transparent"

I executed "iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128"

I cleared the Squid log (/var/log/squid/access.log) then restarted Squid

 

I visited some sites using the client and the browsing works with the sites added to the Squid log file.

 

BUT

 

When I try to access e-mail via my e-mail client (Outlook 2003), it fails. However, if I change the DG to the "real" one on my ADSL router (172.19.0.1), Outlook sends and retrieves mail correctly.

 

I suspect that I need some other iptables command ... unless there's something else that I've missed. Does anyone have any suggestions?

 

Sorry if I should have started a new thread. I'm happy to do so.

 

EDIT - I've now noticed that https traffic is blocked if I have the Squid box as the Default Gateway. I understand that Squid doesn't deal with secure sites so I guess I'd have to configure https traffic to bypass the transparent proxy.

Edited February 7, 2011 by Ignatius
After some research

[glennda]

you would need to also setup port 443 to redirect to port 3128 like you have done for port 80 traffic - its not tested but i think it should work.

[dhicks]

EDIT - I've now noticed that https traffic is blocked if I have the Squid box as the Default Gateway. I understand that Squid doesn't deal with secure sites so I guess I'd have to configure https traffic to bypass the transparent proxy.

 

This thread might come in handy:

 

http://www.edugeek.net/forums/nix/11689-squid-transparent.html

 

I'd love to report that the above was produced after a careful study of the IPTables documentation, and came from my deep understanding of all things routing and filtering related, but as it happens I seem to remember spending a weekend cutting-and-pasting stuff cobbled together from assorted Google searches until it worked...

 

Edit: the above is now 3 years old, glennda's idea might well now work if support for filtering HTTPS has been added to Squid.

[Ignatius]

Thank you both for your input. As far as I know, Squid doesn't work with https. I'll check out the thread from 2007. I figured that I'd have to have some iptables rules to check the destination ports. If they're 25, 110 or 443, send the traffic straight to the real DG (172.19.0.1) but, if the destination port is 80, send the traffic to port 3128 of the same box (I have the Squid and iptables on the same box).

 

If this logic is correct, I might need some help with the iptables syntax!