Smoothwall - Psiphon VPN

[stgoodyeara]

Im network manager at an iPad 1:1 school so i've been locked into a game of "Whack a mole" with the never ending stream of VPN clients that are available on the internet. Smoothwall support has been pretty good to date helping me get my filtering and firewalling rules in place to effectively block lots of VPN services......that is until the students came across Psiphon. This is a very clever VPN that has managed to evade the smoothwall. What it seems to be able to do is connect using what appears to be a never ending stream of different URL's that does actually exist, some wizardry goes on somewhere and bingo the students browsing is all cloaked under this made up URL and guardian merrily lets it through. Ive logged a case with smoothwall who have said something along the lines of:

 

"Its new type of VPN technology the entire industry is struggling with them. Leeds release is due so this might help"

 

All well and good, but this vpn according to my MDM is on about 40% of devices so has basically rendered the smoothwall as a filtering tool as much use a tooth ache. Bandwidth utilization is off the scale. The broadband, Wifi & Smoothwall performance is awful because everything is getting hammered, safeguarding officers are going mental, class teachers are going mental about snapchat notifications going off all the time....what do i do now? Anyone else in this boat?

[FN-GM]

what do i do now?

 

Have you considered non technical steps such as behaviour management?

[stgoodyeara]

Yes. Thats the current approach but isn't really having much impact. A portion of the students are byod and not managed on the mdm. I have no way of identifying who out of these byod devices are using the vpn as the url in smoothwall reporting is different for each connection. So any broad stokes in terms of sanctions will be seen as unfair as it isn't a complete picture and some will be seen to be "getting away with it"

[ozydave]

Bin all the iPads

[caffrey]

We have a VPN spot check , but relies on savvy teachers so it lasted about a week

 

I have a very slow manual way to check so I don't do it very often - check bandwidth usage on Unifi if there's a lot for a user then I do a complete user audit trail report in Smoothwall - if that's empty I report it

 

Lost the battle with whackamole VPNs years ago - I just gave up, situation like you said is getting worse and will get impossible with things like cert pinning. Not sure what the answer is anymore unless like @ozydave said is just to bin BYOD as it just doesn't work

Edited March 14, 2018 by caffrey

[bossman]

This is the very reason why in America the largest single deployment of IPads in the world was scrapped because within 30 secs of use the students had found a way round the security measures which had been put in place using VPN's.... they all now have been issued with chromebooks and all seems well with the deployment...

 

Unfortunately this does not help your situation as a large number of devices are BYOD.... there in lies your problem...until you can effectively manage these devices then i would say the only thing left open to you is to change your infrastructure to be able to manage which apps have access to the network via layer 3 application blocking, Meraki switches would help in this situation as the rules you could apply to all switches would not allow the running of these VPN's fullstop!

 

Expensive....but only other option is as @ozydave states "Bin the use of IPads" instead of BYOD make it CYOD and give the students a choice of devices they can use which you can control.....

[Opendium_Steve]

We managed to figure out how to block Psiphon a few weeks ago and according to the customers the filtering is working ok for now. It is a game of Whack-A-Mole though - each time a VPN client comes along we have to drop everything and figure out how to block it. The previous one of concern was Hotspot Shield, which was quite difficult to block without causing too much collateral damage.

 

I don't think you're going to find a purely technical solution to the problem of VPNs though. I think you have to attack the problem from multiple angles:

 

Ensure that there are serious repercussions for anyone caught using a VPN. If there's no penalty for getting caught, why wouldn't the students try it on?

 

With school controlled devices, use the MDM to prevent random VPN apps from being installed if possible. Also, use the MDM to identify users who have installed VPNs onto these devices and punish them.

 

VPN use also needs to be rolled into the school's internet safety education. A discussion with the students about the safety implications of VPNs could be useful: they may be bypassing the school's antivirus filtering; how trustworthy is the VPN provider, given that they can snoop on all of the students' traffic? etc.

 

A recent incident highlighted to me that the "you must not bypass the school's filtering" clause in an acceptable use policy isn't good enough even for students who want to play by the rules, because they often just don't think enough about what they are doing. A student was caught using a VPN and when it was pointed out that they had agreed not to bypass the school's filtering when they signed the AUP, the student said "I wasn't bypassing the filters, I just wanted to use Facebook [which is blocked during lesson times]".

 

For what it's worth, Psiphon sends POST requests to https://.psiphon3.net/ with a user agent of "Go-http-client/1.1". Blocking these requests doesn't stop it from working, but you should be able to use them to identify devices that have Psiphon installed.

 

Also consider whether your filtering is too restrictive - if you block too much, you push students into using VPNs and 3G/4G and lose any chance you have of safeguarding them. On the other hand, if you're less restrictive they may be able to get to some slightly iffy sites (which they would've got to anyway via VPN/3G/4G) but by keeping them on your network you retain some ability to block the worst stuff and safeguard them by monitoring their access.

 

And yes, I'm afraid everything's going to get a whole lot harder with the rise in popularity of certificate pinning along with Google making a concious decision to make Android 7 incompatible with HTTPS interception. It is particularly concerning that Google have completely ignored our attempts to open a dialogue with them on the issue, even when we tried to discuss the issue through our contacts at the IWF.

[caffrey]

First honest answer I've seen from a provider, this is after a lengthy sales call with Sophos who assured me that everything would be fine. Basically avoid BYOD at all costs or there will be safeguarding issues.

[free780]

If the ipads are under an MDM can the app installation not be prevented?

[caffrey]

If you have an MDM... And if you can get the MDM profile on all the BYOD devices and even then I don't think you can prevent app installation - correct me if I'm wrong

[Opendium_Steve]

First honest answer I've seen from a provider

 

No problem - we always try to be honest, even when the answer may not be what you wanted to hear!

 

a lengthy sales call with Sophos who assured me that everything would be fine.

 

Did they say anything of substance or did they just wave the problems away as non-issues?

 

Basically avoid BYOD at all costs or there will be safeguarding issues.

 

I think it depends what you want to get in terms of safeguarding. Whether or not the school does BYOD, students *will* use their own phones, you just get to choose whether they are using 3G/4G or the school's network.

 

So you can avoid BYOD entirely, let them use 3G/4G and wash your hands of the issue - if it doesn't happen on the school network, it isn't the school's problem. This obviously means you may miss safeguarding opportunities since you have absolutely no control over their internet access.

 

Or you can offer a BYOD scheme and accept that you're not going to have complete control - this gives you some scope for blocking the worst stuff and allows some monitoring for safeguarding purposes, just possibly not as much control and monitoring as you'd like. On the other side of the coin this has the potential for opening up the school to liabilities when a student uses the internet inappropriately, because the connection is now a service the school are providing rather than the parents' responsibility.

 

These are hard questions and I don't have any easy answers I'm afraid, but this is the balancing act we all have to play.

[Opendium_Steve]

Oh, one thing I did mean to mention - obviously web traffic goes through your web filtering system, but you often need to poke holes in your firewall for other protocols. Make sure that any holes you poke are either restricted to specific destination IP addresses (e.g. allowing traffic to 17.0.0.0/8 is a common and fairly safe one since that whole network is Apple's), or the ports you open are protected by layer 7 deep packet inspection.

 

Any ports you leave wide open that haven't got these protections will very quickly be abused by VPNs - I've seen NTP and DNS ports being used by VPNs, amongst others (we tend to intercept DNS and NTP traffic and redirect it to local servers instead of allowing it out to the internet).

[stgoodyeara]

If the ipads are under an MDM can the app installation not be prevented?

 

No Apple API's that MDM system use are not fit for purpose in education. You can prevent ALL app installation or you can ALLOW all app installation. There is no middle group. If i prevent app installation and then use the MDM to manage app installations it will erode the usability of the devices and as the 1:1 scheme moves forward more people will BYOD rather than buy through the school.

 

In terms of safeguarding its all about "taking reasonable measures" i accept we can never beat all VPN and simple chucking ipad in the bin is like saying lets go back to chalk boards in classrooms. As long as we can demonstrate we have take reasonable steps to safeguard the students then nobody can ask any more. The issue i have with Psiphon isn't the safeguard aspect, im sure in time smoothwall and other filter providers have/will work out how to stop it. The issue i have is that its over usage in the school is have service level impacts on other services. So the purpose of this post was a plea for the resolution not to 1:1 device scheme bash. Im aware of all the pit falls of 1:1 as are my SMT and we've made the decision that the impact on T&L is worth the pit falls.

[bossman]

@SteveHill

 

With the latest switch ingress from Meraki you should be able to effectively block VPN applications from running via layer 7 application management tools which would shape the traffic before it actually reaches your proxy server would it not?

 

I don't have Meraki switches as such but have researched their capabilities within the LAN and also the WAN regarding BOYD....

 

https://meraki.cisco.com/solutions/mobile-device-management

 

https://meraki.cisco.com/technologies/layer-7-visibility

 

Surely this would be the way forward to prevent the use of VPN's on the networks?

 

Or am I totally off the mark here?

[stgoodyeara]

Oh, one thing I did mean to mention - obviously web traffic goes through your web filtering system, but you often need to poke holes in your firewall for other protocols. Make sure that any holes you poke are either restricted to specific destination IP addresses (e.g. allowing traffic to 17.0.0.0/8 is a common and fairly safe one since that whole network is Apple's), or the ports you open are protected by layer 7 deep packet inspection.

 

Any ports you leave wide open that haven't got these protections will very quickly be abused by VPNs - I've seen NTP and DNS ports being used by VPNs, amongst others (we tend to intercept DNS and NTP traffic and redirect it to local servers instead of allowing it out to the internet).

 

Ive done all this thanks, and this process although its a time consuming one is worth while. I will not allow any policies on the firewall that open any ports without a defined destination. This was a body of work i did when the smoothwall new firewall came out and was pretty successful in killing most VPN's on the market at the time. VPN's generally just had lightweight port sniffers built into them and would just sniff out open ports on the firewall. So my BYOD network is only allowed ports 80 & 443 through guardian and nothing else. I wont even allow email based protocols as ive seen these port being exploited by VPN's and opening email ports to all the possible mail providers specific IP addresses wasn't a viable option so i advise students to either use office 365 accounts provided by the school or login to the email services via webmail. Until Psiphon came along i was pretty confident that VPN's weren't being used. Ive got a few students that are acting as "a spy in the camp" and they will email me when word begins to spread that a new working VPN is found in exchange for anonymity and access to clash of clans on a lunch time.

 

The issue with Psiphon is that it is very cleaver almost a new generation of VPN, it seems to package up the users browsing traffic and somehow bluff guardian into thinking it is legitimate traffic. On the webfilter reports you just seem traffic going to a host of randomly generated URL's how the VPN manages to do this i have no idea.... but its clever i'll give it that.

[stgoodyeara]

Touching back on the CRAP apple API's for MDMs. There is an option in the mdm "Allow adding VPN configurations" i have disabled this so you'd think that would stop VPN configurations being added....wrong.... what actually happens is its prevents access to the adding VPN routine it doesn't actually stop the vpn configuration being added by an app installation. So while a user can't manually setup an VPN apps can install an VPN profile on the devices.

[stgoodyeara]

I think it depends what you want to get in terms of safeguarding. Whether or not the school does BYOD, students *will* use their own phones, you just get to choose whether they are using 3G/4G or the school's network.

 

Our AUP that parents and students sign state that we don't permit the usage of hotspot on mobile phones and that if students access inappropriate content on these hotspot type connections that as the parent fundamentally provided the connection in proving the student with the phone that they accept the responsibility of safeguarding content delivered by these connections.

 

The grey area here is "Student A" uses "Student B's" hotspot.....are Student B's parents responsible for the well being of Student A who is not related to them?

[fiza]

Is there any way to run a report in Smoothwall to see if our BYOD users have the app?

[Opendium_Steve]

With the latest switch ingress from Meraki you should be able to effectively block VPN applications from running via layer 7 application management tools which would shape the traffic before it actually reaches your proxy server would it not?

 

Layer 7 deep packet inspection (DPI) is definitely worth having, and the main filtering providers all offer it on their filtering products. I would tend to do this kind of traffic control on your internet gateway device - Meraki seem to be doing it on your core network switches, which just seems like a very odd place for it. DPI catches a lot of the "low hanging fruit", but certainly isn't a magic bullet.

 

The big problem VPNs of recent times try really hard to make their traffic indistinguishable from legitimate traffic, and this will fool all of your filtering. Once you've identified that a VPN is getting through the filtering, your filtering provider then needs to put in the development effort to analyse the traffic, figure out how to fingerprint it so it can be blocked. And since it looks almost identical to legitimate traffic, identifying and blocking the VPN traffic without causing a load of collateral damage by blocking legitimate traffic too is really difficult.

 

The last two VPNs we analysed were Psiphon and Hotspot Shield. Of these, I'd say that Hotspot Shield was probably the hardest to deal with, and to give you an idea why, I'll briefly describe how it works:

 

1. The VPN client connects to an IP address on port 443 - these IP addresses are hosted in popular cloud services, there are vast numbers of them and they change regularly. So blocking them isn't really an option.

2. The VPN client and server exchange HTTPS handshake traffic which has been previously recorded from the network traffic generated by a legitimate client accessing a legitimate website. The VPN system contains a large number of these recorded handshakes, changes them frequently, and since they are identical to legitimate HTTPS handshakes, you can't block the connection based on anything they contain.

3. As far as anyone snooping on the traffic (such as your switches or filtering system) knows, an encrypted connection has now been set up based on that handshake traffic and anything that follows would normally be encrypted with the associated keys and therefore unsnoopable. However, there is no way for the snooper to verify that the traffic being exchanged is actually encrypted with the appropriate keys (since the snooper doesn't have access to those keys). So in actual fact, the VPN client and server can send any traffic they like, and that's exactly what happens.

 

To a passive inspection system, this traffic is indistinguishable from legitimate traffic and therefore you can't block it without also blocking the legitimate application. Of course, if you man-in-the-middle the HTTPS session, the VPN client can't connect. Unfortunately, because of certificate pinning, you can't man-in-the-middle every HTTPS connection, and eventually the VPN client stumbles upon something you didn't intercept and successfully connects.

 

There is absolutely no way any kind of filter is going to automatically block that traffic without also blocking all the legitimate traffic - it requires a developer to spend time figuring out how the VPN is working and come up with a way of blocking that traffic without impacting legitimate browsing. Anyone who tells you "our system can magically block any VPN which is invented in the future" is selling you snake oil - the real test is how quickly a vendor can react and develop a solution once a new VPN has been identified.

Edited March 16, 2018 by elsiegee40
Edited for commercial

[stgoodyeara]

@SteveHill

 

With the latest switch ingress from Meraki you should be able to effectively block VPN applications from running via layer 7 application management tools which would shape the traffic before it actually reaches your proxy server would it not?

 

I don't have Meraki switches as such but have researched their capabilities within the LAN and also the WAN regarding BOYD....

 

https://meraki.cisco.com/solutions/mobile-device-management

 

https://meraki.cisco.com/technologies/layer-7-visibility

 

Surely this would be the way forward to prevent the use of VPN's on the networks?

 

Or am I totally off the mark here?

 

Again i might be totally off the mark here too. But my interpretation of Layer 7 packet filtering is its only as good as the definitions that it has. Smoothwall Firewall has layer 7 packet filtering capabilities but its let down by weak definitions. Psiphon VPN is defined as a Application on the layer 7 tool in smoothwall but the recent re-write of the software has resulted in it no longer working.

 

Layer 7 is just another "whack-a-mole" playground where VPN developers try to cook up ways of disguising data and layer 7 signature providers frantically find ways to identify them.

[Opendium_Steve]

Again i might be totally off the mark here too. But my interpretation of Layer 7 packet filtering is its only as good as the definitions that it has.

 

Yes, and also the fact that these VPNs try their hardest to look like legitimate traffic. Anyone who says "our product can stop all VPNs, including those that haven't been invented yet" is selling you snake oil. Sometimes it may be possible to do traffic flow analysis to figure out which machines are using VPNs, but for the advanced VPNs there's no way an automated system is going to be able to figure out how to tell the difference between each VPN connection and each legitimate connection in order to block it without also blocking the legitimate stuff. The real test is how quickly a vendor can put developers onto the job and figure out a solution once someone has discovered a VPN that's getting through.

 

Layer 7 deep packet inspection will get the low hanging fruit, but without the vendor constantly keeping on top of new VPNs it'll never stop the more advanced ones.

 

Layer 7 is just another "whack-a-mole" playground where VPN developers try to cook up ways of disguising data and layer 7 signature providers frantically find ways to identify them.

 

Exactly.

Edited March 16, 2018 by ZeroHour

[stgoodyeara]

Is there any way to run a report in Smoothwall to see if our BYOD users have the app?

 

If you do a webfilter report for Psiphon3.net you'll see the requests going through, i highlighed the ones below to lookout for. Despite them being blocked the VPN still connects. But you can atleast formulate a list of whos using it. Im having little success trying to get SMT to impose a sanction that will apply to over 2/3 of the student body.

 

[Opendium_Steve]

Although the MDM can't stop the installation of Psiphon, can you perhaps have the MDM automatically remove it if it is installed? (run this as a regular job and it's almost as good as preventing the installation in the first place).

 

Regarding sanctioning 2/3 of the students, maybe arrange an amnesty: remove it from your device within a week and we'll say no more, anyone who doesn't gets sanctioned? (Clearly explaining that the amnesty is a one-off and anyone caught with a VPN in future will be punished).

[DGardiner]

we have the app store dissabled on all our ipads, if you want an app i gets delivered via MDM or you do without.

 

we have 1:1 also

[fiza]

If you do a webfilter report for Psiphon3.net you'll see the requests going through, i highlighed the ones below to lookout for. Despite them being blocked the VPN still connects. But you can atleast formulate a list of whos using it. Im having little success trying to get SMT to impose a sanction that will apply to over 2/3 of the student body.

 

[ATTACH=CONFIG]48137[/ATTACH]

Thanks. Looks like we are not getting any hits on that so that's good.