Dansguardian not letting HTTPS through

[KK20]

I have dansguardian 2.9.9.4 squid 2.7.stable3 on a single box. The network map is as follows:

. users 192.168.2.x -> DG (192.168.1.5:8080) + squid (127.0.0.1:3128) -> router

HTTP works fine (and blocks where necessary) but HTTPS timeout after a minute or so. Browser is IE with manually configured proxy address

If I configure clients to bypass DG by using port 3128 (and adding the necessary http_port 192.168.1.5 in squid) then I can access HTTP and HTTPS just fine.

I use NTLM authentication, the box is part of the domain and I have no filterlists other than default at the moment. ClamAV is not installed.

here are my configs

dansguardian.conf

reportinglevel = 3
languagedir = '/etc/dansguardian/languages'
language = 'ukenglish'
loglevel = 2
logexceptionhits = 2
logfileformat = 1
anonymizelogs=off
filterip = 192.168.1.5
#filterip = 192.168.1./255.255.0.0
filterport = 8080
proxyip = 127.0.0.1
proxyport = 3128
accessdeniedaddress = 'http://YOURSERVER.YOURDOMAIN/cgi-bin/dansguardian.pl'
nonstandarddelimiter = on
usecustombannedimage = on
custombannedimagefile = '/usr/share/dansguardian/transparent1x1.gif'
filtergroups = 2
filtergroupslist = '/etc/dansguardian/lists/filtergroupslist'
bannediplist = '/etc/dansguardian/lists/bannediplist'
exceptioniplist = '/etc/dansguardian/lists/exceptioniplist'
showweightedfound = on
weightedphrasemode = 2
urlcachenumber = 1000
urlcacheage = 900
scancleancache = on
phrasefiltermode = 2
preservecase = 0
hexdecodecontent = off
forcequicksearch = off
reverseaddresslookups = off
reverseclientiplookups = off
logclienthostnames = off
createlistcachefiles = on
maxuploadsize = -1
maxcontentfiltersize = 256
maxcontentramcachescansize = 2000
maxcontentfilecachescansize = 20000
filecachedir = '/tmp'
deletedownloadedtempfiles = on
initialtrickledelay = 20
trickledelay = 10
downloadmanager = '/etc/dansguardian/downloadmanagers/fancy.conf'
#downloadmanager = '/etc/dansguardian/downloadmanagers/trickle.conf'
downloadmanager = '/etc/dansguardian/downloadmanagers/default.conf'
contentscannertimeout = 60
contentscanexceptions = off
authplugin = '/etc/dansguardian/authplugins/proxy-ntlm.conf'
authplugin = '/etc/dansguardian/authplugins/proxy-basic.conf'
recheckreplacedurls = off
forwardedfor = off
usexforwardedfor = off
logconnectionhandlingerrors = on
logchildprocesshandling = off
maxchildren = 120
minchildren = 8
minsparechildren = 4
preforkchildren = 6
maxsparechildren = 32
maxagechildren = 500
maxips = 0
ipcfilename = '/tmp/.dguardianipc'
urlipcfilename = '/tmp/.dguardianurlipc'
ipipcfilename = '/tmp/.dguardianipipc'
nodaemon = off
nologger = off
logadblocks = off
loguseragent = off
softrestart = off
mailer = '/usr/sbin/sendmail -t'

squid.conf

http_port 127.0.0.1:3128 transparent
#the next line is for testing only.  We only want people connecting
# to port 8080 to go through dansguardian but for testing we can
# allow people to bypass dans at port 3128
http_port 192.168.1.5:3128 transparent

auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 5
authenticate_ttl 180 seconds
auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours

cache_peer 127.0.0.1 parent 8080 0 no-query login=*:nopassword

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl localserver src 192.168.1.5
acl to_localhost dst 127.0.0.0/8
acl localnet src 192.168.1.0/255.255.0.0
acl ntlm proxy_auth REQUIRED
acl SSL_ports port 443          # https
acl SSL_ports port 563          # snews
acl SSL_ports port 873          # rsync
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 631         # cups
acl Safe_ports port 873         # rsync
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT

http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow CONNECT SSL_ports
http_access allow ntlm
http_access allow localhost
http_access allow localserver
http_access deny all

icp_access allow localnet
icp_access deny all

hierarchy_stoplist cgi-bin ?
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern (Release|Package(.gz)*)$        0       20%     2880
refresh_pattern .               0       20%     4320
acl shoutcast rep_header X-HTTP09-First-Line ^ICY\s[0-9]
upgrade_http0.9 deny shoutcast
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
extension_methods REPORT MERGE MKACTIVITY CHECKOUT
hosts_file /etc/hosts
coredump_dir /var/spool/squid

Any ideas? Thanks in advance...

edit: not sure if this question is best served in here or in *nix - can I point from *nix to here or is that a board no-no?

[KK20]

Right. I have a very very dirty workaround.

Since I run GPO with pretty much everything locked down - the users cannot change the connections properties so all my systems are run from WPAD with respect to the proxy.

I have changed the proxy setup in the clients so that the proxy is 192.168.1.5:8080 for all *but* https which is 192.168.1.5:3128

This isnt ideal and doesnt fix the underlying problem but it at least gets me working until I can find a better fix ....

[wuzzy]

I have the exact same problem, apparently dansguardian does not support https traffic and it's just "killing" it, there must be some way to force it to just bypass https traffic..

[KK20]

If you set reportinglevel to 3 then you can see that DG is grabbing the HTTPS but only knows the IP not the username (unlike the HTTP) so DG is clearly grabbing the https but not tunnelling it correctly.

I might look at adding squidguard just to deal with HTTPS - possibly on a whitelist based on usernames.

[OutToLunch]

Not sure about the cache_peer 127.0.0.1 parent 8080 0 no-query login=*:nopassword
line?

Have dg -> squid running fine here for http/https without that. Are you creating a wierd loop that's killing something?

[KK20]

Indeed that was a redundant line for when I was experimenting with squid->DG (That line sends the NTLM authentication to squid so that the clients do not need to see a pop up box asking for their usernames and passwords - but I found that squid wouldnt log the usernames hence me needing to go DG->squid)

Ive commented it out now as it isnt needed with DG->squid but it still makes no difference unfortunately.

In either case here is a snippet from the DG access.log

2009.7.17 8:28:45 testaccount 192.168.2.123 http://www.google.co.uk/csi?v=3&s=we...s.1993,ol.2123 GET 0 0 1 204 text/html -
2009.7.17 8:28:45 testaccount 192.168.2.123 http://mail.google.com/mail/?hl=en&tab=wm GET 439 0 1 302 - -
2009.7.17 8:28:56 - 192.168.2.123 https://www.google.com:443 CONNECT 0 0 1 200 - -

[KK20]

For reference I managed to fix this. *but* I soon went back to my kludge!

I had originally installed DG from the debian repo - version 2.9.9.4

I removed this and compiled 2.10.1.3 and the HTTPS passthrough works perfectly. BUT DG doesnt filter the https! No matter what I tried it will not filter it - I presume because it cannot garner the username from the HTTPS header (being encrypted) therefore is defeated by the NTLM.

So I went back to my original split proxy of https going through squid directly (and thus using the blocklists for proxies and games) and all other traffic through DG.

This isnt perfect as all HTTPS are treat at the same maximum security but it works.

[j17sparky]

For reference I managed to fix this. *but* I soon went back to my kludge!

I had originally installed DG from the debian repo - version 2.9.9.4

I removed this and compiled 2.10.1.3 and the HTTPS passthrough works perfectly. BUT DG doesnt filter the https! No matter what I tried it will not filter it - I presume because it cannot garner the username from the HTTPS header (being encrypted) therefore is defeated by the NTLM.

So I went back to my original split proxy of https going through squid directly (and thus using the blocklists for proxies and games) and all other traffic through DG.

This isnt perfect as all HTTPS are treat at the same maximum security but it works.

It wont filter HTTPS because it cant decrypt the traffic, thats the whole point in HTTPS. If any proxy the traffic passed through could decrypt it then the protocol wouldnt be very good.

[KK20]

Agreed but it should still filter the https traffic based on the header. In fact there should therefore be an option in DG for a filterlist *for* https traffic. What is to stop people using https://proxy.org/ and bypassing everything?

Squid seems quite happy at blocking https traffic (hence my kludge using a split proxy.pac file).

[KK20]

OK! I revisited this since I had some spare time on my hands and it now works!

I am using compiled versions of squid and DG on a fresh installation of debian.

path is:

users -> switches -> DGBOX (2 NIC as a bridge) -> router

The bridge is using ebtables/iptables to force traffic to DG 8080 (https included) and can act as a proxy, squid is local traffic only so no bypass. There is a .PAC *and* .WPAD file available on the network. I have rolled out I.E. settings via GPO to accept the .PAC and the .WPAD is there for visitors via DHCP.

I am using NTLM with the usernames being grabbed from an LDAP scraping script (10 seconds execute time) that is running hourly. The list self creates and updates with an administrator exception list.

MOST traffic that tries to circumvent the proxy (i.e. browsers on a USB pen) will simply return "not authenticated" and thus they will have to automatically detect the proxy via the .WPAD on the DS. I can loan out authentication usernames as necessary for visitors.

DG *will* filter https traffic based on URL now so no more pesky https proxies! Obviously one bright 6th former decided to set a proxy up at home but luckily he was on a static IP and he was disciplined over it. I dont plan on adding domestic ISP ranges to the blocklist anytime soon....

This finishes a 3 month journey into linux and can happily say that it works and means I can finally throw that fortinet box in the bin.