Squid3.4.8 and accessing youtube

[notalot]

Hi, sorry if this is in the wrong place please feel free to move it.

 

We have just finished setting up a Squid3 server on Debian.

 

It works fine 99% of the time but some sites it refuses to load at all, YouTube happens to be one of them.

 

When you browse to http://www.youtube.com the page just sits loading, nothing comes up on screen (I've left it for 30 minutes with nothing loading) and it never times out. We have the same problem if we try to go to Google (even though google is the homepage) it loads google for the first time then doesn't do it again.

 

I have no idea what the problem is with it as I'm still very new to Squid, we have tried a few things.

 

Turning off the Squid Cache

Adding the DNS_4_first setting to the config

Setting debian to use IPv4 first.

Checking that the file descriptors are within limits

 

Looking in the Access.log file the connection request doesn't show up.

 

See below a copy of our config (the cache has been turned back on as it doesn't appear to fix the problem).

 

#Recommended minimum configuration:

shutdown_lifetime 2 seconds

cache_peer UPSTREAMFILTER parent 80 0 no-query no-digest
never_direct allow all
acl localnet src 10.60.148.0/22
acl SSL_ports port 443 563 1863 5190 5222 5050 6667
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http

acl CONNECT method CONNECT

http_access deny to_localhost
icp_access deny all
htcp_access deny all
http_port 8080 
hierarchy_stoplist cgi-bin ?

access_log /var/log/squid3/access.log squid


#Suggested default:
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern .		0	20%	4320
# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid3

##############################################################################
##Auth Settings
##############################################################################

### negotiate kerberos and ntlm authentication
auth_param negotiate program /usr/local/bin/negotiate_wrapper -d --ntlm /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=DOMAIN --kerberos /usr/lib/squid3/negotiate_kerberos_auth -d -s GSS_C_NO_NAME
auth_param negotiate children 10
auth_param negotiate keep_alive off

### pure ntlm authentication
auth_param ntlm program /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=DOMAIN
auth_param ntlm children 10
auth_param ntlm keep_alive off

### provide basic authentication via ldap for clients not authenticated via kerberos/ntlm
auth_param basic program /usr/lib/squid3/ext_ldap_group_acl  -R -b "dc=DOMAIN,dc=local" -D squid@DOMAIN -W /etc/squid3/ldappass.txt -f sAMAccountName=%s -h dc01.DOMAIN
auth_param basic children 100
auth_param basic realm Internet Proxy
auth_param basic credentialsttl 10 minute


### acl for proxy auth and ldap authorizations
acl auth proxy_auth REQUIRED

#   aclname             acltype  typename activedirectorygroup
acl allowedsites        dstdomain "/etc/squid3/allowedsites.txt"
acl blockedsites        dstdomain "/etc/squid3/blockedsites.txt"
acl exceptedsites       dstdomain "/etc/squid3/exceptedsites.txt"
acl prioritysites       dstdomain "/etc/squid3/prioritysites.txt"


### http_access rules

# allow unrestricted access to prioritysites
http_access allow prioritysites

# enforce authentication, order of rules is important for authorization levels
http_access deny !auth

# prevent access to basic auth prompt for BlockedAccess users
http_access allow allowedsites
http_access deny blockedsites

# Access
http_access allow localnet

# block all other access
http_access deny all

 

I would appreciate any help in getting to the bottom of what's causing this.

[psydii]

Try using the developer tools in the browser to identify what elements in particular are taking a long time to load.

[jinnantonnixx]

I haven't worked with Squid for a while, so I'm pretty rusty.

 

Is there a reason why you're using Kerberos and NTLM?

Kerberos is by far the preferred protocol. Even though your config offers Kerberos before NTML, it is the client browser that decides on the protocol that it ultimately uses.

 

Squid exceptions are logged in the file 'cache.log'.

This command will list the last events logged by squid. The -f option keeps on polling the file, showing you a live view until you press Ctrl-C to escape.

tail -f /var/log/squid/cache.log

 

Have you configured the squid stats tool? This is a CGI script which should be installed by default, but not necessarily configured.

 

First find the file

locate cachemgr.cgi

Copy the file to this location:

/var/www/cgi-bin/cachemgr.cgi

 

Start a browser and connect to

http://your_squid_server/cgi-bin/cachemgr.cgi

 

There's a whole heap of useful goodies in here.

 

The following is a paste from my notes - it must have seemed important to me at the time.

 

The most useful menu items are:

 

 

NTLM User Authenticator Stats

This shows the array of NTLM authenticators and their status. If they are busy, they will show the current requests. Pay attention to the 'Queue Length' field. If there are requests backing up, they will show as a non-zero number. This indicates that the server is too busy.

 

Negotiate Authenticator Statistics

This shows the array of Kerberos authenticators and their status.

 

5 Minute Average of Counters

This shows a comprehensive range of statistics. Of particular interest:

 

 

client_http.requests metric. Regular inspection of these values tells us what a normal load is, but as a rule of thumb, we can regard 150 requests/sec as being on the heavy side.

 

client_http.all_median_svc_time. This metric shows how long it took to complete a single HTTP transaction. A good value is around 0.01 seconds.

 

dns.median_svc_time. This is the measure of time taken to resolve DNS cache misses. A good value is under 0.05 seconds. High values indicate a problem with the primary DNS server (or network).

 

 

How's your DNS server - is it stressed out?

If you can, you might want to try using Google's public DNS servers just to see if this has any effect. You'll need to restart the Squid service, so do this out-of-hours.

In the squid.conf, add

dns_nameserver 8.8.8.8  8.8.4.4

Edited December 8, 2015 by jinnantonnixx

[notalot]

Thanks to both of you,

 

Psydill - The developer menu doesn't show anything just that the request was started.

 

jinnantonnixx - Thas a massive help.

 

We were using a config that was from a guide and we haven't thinned it down yet just wanted to get things working then look at improving them, as it stands this is in development currently and not live.

 

From your information it turns out that there is a DNS related issue with Google and YouTube but not with any other site I tried. Its a little odd but I would guess it has something to do with the LA filtering forcing safesearch.

 

If I run the DNS on the server through 8.8.8.8 then it works fine but bypasses some of the filtering which I really don't want to do.

 

Now its just a case of working out how to resolve the issue.

[jinnantonnixx]

You might want to try these settings in the squid.conf.

negative_ttl 1 minute
positive_dns_ttl 1 hour
negative_dns_ttl 1 minute

 

These set negative hits to have a short life time, so that if something is wrong with your DNS, and a negative hit is encountered (by error), then the negative hit isn't remembered for long.

 

squid : negative_dns_ttl configuration directive