Blocking use of VPNs.

[Opendium_Steve]

Well, if you managed to install a VPN endpoint at adobe.com you probably deserve to get unfiltered internet access

Ok, this is a real world example - its exactly how Hotspot Shield works. In this example I'll use "get.adobe.com" as the forged web server that it is purporting to connect to:

 

1. User installs VPN client on their phone when not on the school network, so there's no chance of preventing this.
   
2. VPN client downloads a list of server IPs while not on the school network, so again, no chance of preventing this.
   
3. User logs onto school network.
   
4. VPN client connects to a server IP on port 443, from the list it downloaded in (2).
   
5. VPN client sends a TLS client handshake that contains a server name indication (SNI) of get.adobe.com. This handshake is a replay of a recorded handshake to the real get.adobe.com service.
   
6. VPN server sends a TLS server handshake and presents a certificate for get.adobe.com. Again, this is a replay of a real recorded handshake.
   
7. Client and server now have an encrypted connection which they use as a VPN.
   

 

So, looking at this, its very difficult to see how the proxy server can detect and block this VPN.

 

The whole TLS handshake (steps 5 and 6) is a recording of a real handshake to get.adobe.com, so there is nothing "odd" about it that can be used to distinguish it from a legitimate connection.

 

Since the server doesn't have the legitimate service's private key, the encryption used for the VPN connection (7) is not in any way related to that was apparently negotiated in the TLS handshake. But that doesn't matter, because there's no way for the proxy server to know what the encryption should have looked like and compare the two. At this point, all it knows is that there is an encrypted data stream between the two endpoints - it can't know if that matches the encryption that was negotiated in the handshake, or if it is completely unrelated to that.

 

 

So, options:

 

We can watch the DNS traffic and see if the host actually looked up get.adobe.com, and whether the DNS response included the IP address that the host is connecting to. But:

• The client may have done the DNS lookup before the device logged onto the school wifi, and is using a cached copy of the DNS result. This would result in blocking a legitimate client.
  
• If we did see a DNS request/response, how long should we consider it to be valid for? Just looking at the record's TTL isn't good enough because applications frequently completely ignore the TTL.
  
• Some legitimate apps are just badly written and don't use DNS.
  
• (We frequently see legitimate web traffic with no matching DNS traffic preceding it)
  

 

We can actually do a DNS lookup for get.adobe.com ourselves and see if the IP address being connected to is listed. But:

• These days the big services don't present the same records on every request.
  
• This is especially true if the DNS requests were made before the phone logged onto the wifi - anycast DNS is frequently used to direct people at their "nearest" server, and the nearest server for someone on 4G is probably not the same as for someone on your wifi.
  
• TTLs are often very short.
  
• All this means that the IP addresses you get back from doing a DNS lookup are probably not the same as those that the client got.
  

 

We can do a DNS lookup for get.adobe.com ourselves and connect them to that IP instead of connecting them to whatever IP the client was trying to connect to. But:

• This just plain tends to break legitimate things, because the client ends up connected to a server other than the one it expected to.
  

 

We can decrypt (man in the middle) the HTTPS connection, after which it becomes very obvious that the encryption being used doesn't match what was negotiated. But:

• Some legitimate apps (and Adobe's software update system is one) are designed to not be compatible with decryption, so if you want the legitimate app to work you have to exclude this host name from being decrypted.
  
• Although I've specifically talked about get.adobe.com here, the reality is that the VPN forges various different host names until it finds one that works.
  

 

You could build a list of IP addresses that it connects to and block them. But:

• Those IP addresses change very frequently. The advent of cloud services means that you can spin up a new bunch of virtual machines (with new IP addresses) to use as VPN servers every few hours.
  
• This assumes they aren't sharing IP addresses with legitimate services, which certainly isn't always the case.
  

 

All this assumes that you don't want to allow any non-web traffic, which makes things even more difficult!

 

I don't want to be dismissive, but I do see a lot of comments along the lines of "just block everything except legitimate traffic" that make the problem sound simple. This completely glosses over the fact that you first have to be able to identify which traffic is legitimate. Some of the VPNs are designed to make it really hard to tell their traffic apart from the legitimate stuff, and much of the legitimate software that people want to use is badly designed or just plain broken, so you're already having to bend the rules a bit to even allow that to work.

 

As it turns out, it was possible to engineer a way to block this type of VPN, but doing so involved quite a lot of analysis to work out how this specific one worked and figure out how to block it without affecting the legitimate traffic. Every so often a new VPN pops up which works in a slightly different way and you have to go back to the drawing board and engineer a fix for that specific situation too. To state the obvious: if there were an easy fix to block all VPNs in one go, all of the filtering vendors would be doing it already.

Edited December 16, 2019 by Opendium_Steve

[mavhc]

> VPN client connects to a server IP on port 443, from the list it downloaded in (2).

 

Well, it connects to the school's proxy.

 

I'm assuming you're decrypting https, and not using transparent proxy, and not allowing dns

 

> Some legitimate apps are just badly written and don't use DNS.

 

What happens if you block everything of the form http(s)://ip.address.octet.octet[:/]*

 

> We can do a DNS lookup for get.adobe.com ourselves and connect them to that IP instead of connecting them to whatever IP the client was trying to connect to. But:

> This just plain tends to break legitimate things, because the client ends up connected to a server other than the one it expected to.

 

If you did that from the start of the connection would it matter?

 

> We can decrypt (man in the middle) the HTTPS connection, after which it becomes very obvious that the encryption being used doesn't match what was negotiated. But:

> Some legitimate apps (and Adobe's software update system is one) are designed to not be compatible with decryption, so if you want the legitimate app to work you have to exclude this host name from being decrypted.

 

But if you negotiate the connection to adobe.com etc, then you don't need to decrypt it, you know it's legit.

 

Question is how much stuff does that break?

[Opendium_Steve]

> VPN client connects to a server IP on port 443, from the list it downloaded in (2).

 

Well, it connects to the school's proxy.

 

I'm assuming you're decrypting https, and not using transparent proxy, and not allowing dns

 

This is where the assumption is flawed. Absolutely no devices reliably support connecting to a non-transparent proxy, so if you are relying entirely on a non-transparent proxy, lots of stuff isn't going to work. Even Windows machines have apps that ignore your proxy settings, mobile devices are worse. Not to mention the hassle of manually configuring proxy settings on every device (because nothing pays attention to WPAD these days, argh!).

 

> Some legitimate apps are just badly written and don't use DNS.

What happens if you block everything of the form http(s)://ip.address.octet.octet[:/]*

 

Ok, if you're using a non-transparent proxy, the client will send "CONNECT :443 HTTP/1.1" to the proxy. Mostly, the will be a hostname, but that is by no means guaranteed - some clients send an IP address instead. If you block CONNECTs to IP address literals, some legitimate stuff will break. After the CONNECT happens, the client sends a TLS handshake which should include the host name its trying to connect to (the SNI).

 

Since relying on a non-transparent proxy to handle everything is generally unrealistic these days, so most schools use a transparent proxy (possibly in addition to the non-transparent one, more usually instead of it where BYOD is concerned). A transparent proxy doesn't get the "CONNECT" request, so you have to rely entirely on the SNI contained in the TLS handshake.

 

In the example I gave, the TLS handshake is a replay of a real handshake. That SNI can't be trusted, but the proxy doesn't know that. Also note that some legitimate apps either don't include an SNI at all, or include an SNI that's just plain wrong.

 

> We can do a DNS lookup for get.adobe.com ourselves and connect them to that IP instead of connecting them to whatever IP the client was trying to connect to. But:

> This just plain tends to break legitimate things, because the client ends up connected to a server other than the one it expected to.

 

If you did that from the start of the connection would it matter?

 

Yes - some apps expect to actually connect to the IP address they thought they were connecting to, and break if they connect to a different IP. Some apps just plain set the SNI to the wrong thing, or an invalid one. It shouldn't be like that, but app developers are idiots.

 

> We can decrypt (man in the middle) the HTTPS connection, after which it becomes very obvious that the encryption being used doesn't match what was negotiated. But:

> Some legitimate apps (and Adobe's software update system is one) are designed to not be compatible with decryption, so if you want the legitimate app to work you have to exclude this host name from being decrypted.

 

But if you negotiate the connection to adobe.com etc, then you don't need to decrypt it, you know it's legit.

 

Question is how much stuff does that break?

I'm not sure I understand what you're saying here - the proxy (whether transparent or not) sees a TLS handshake from the client, containing "get.adobe.com" as the SNI. It can either passively inspect that, or decrypt it.

 

If you passively inspect it, you get to see the whole TLS handshake (SNI from the client, and the certificate from the server), but you don't get to decrypt the subsequent connection. As mentioned, the TLS handshake that the proxy sees is a real handshake to a legitimate service, which is being replayed. There's no way to know that the connection isn' t to that real service.

 

If you decrypt it, you get to see the whole TLS handshake and get to decrypt the subsequent connection. This is great, but certain clients are designed to not allow this and will therefore break.

 

 

The upshot of all this is: yes, you can engineer a system that knows exactly what host name its connecting to by doing things like insisting that *everything* goes through a non-transparent proxy, and that *everything* uses a host name, rather than IP address in its CONNECT request, and you can insist on decrypting *everything*. But you'll break a lot of legitimate software in the process, and in my experience that isn't seen as an acceptable way to run a school internet connection. Even if you know exactly which host name its using, you're still left figuring out which are legitimate hosts and which are VPNs. You can obviously set up a walled garden and only allow known-good sites, but for a "default-allow" policy you're going to have an ongoing battle figuring out which hosts are VPNs when they change every few days.

 

In the real world, we're left having to accommodate badly designed or completely broken apps, and generally "the app is broken" isn't seen by end-users as an acceptable reason why something doesn't work on the school network when it works fine on a home internet connection. So we end up working around all the mess caused by terrible legitimate apps, which creates lots of extra scope for the VPNs to go undetected too.

Edited December 16, 2019 by Opendium_Steve

[mavhc]

> This is where the assumption is flawed. Absolutely no devices reliably support connecting to a non-transparent proxy, so if you are relying entirely on a non-transparent proxy, lots of stuff isn't going to work.

 

Fair enough, so if you want to know you've blocked everything, you've also broken a lot of stuff.

 

> Ok, if you're using a non-transparent proxy, the client will send "CONNECT :443 HTTP/1.1" to the proxy. Mostly, the will be a hostname, but that is by no means guaranteed - some clients send an IP address instead. If you block CONNECTs to IP address literals, some legitimate stuff will break. After the CONNECT happens, the client sends a TLS handshake which should include the host name its trying to connect to (the SNI).

 

And now more stuff breaks

 

> I'm not sure I understand what you're saying here - the proxy (whether transparent or not) sees a TLS handshake from the client, containing "get.adobe.com" as the SNI. It can either passively inspect that, or decrypt it.

 

I'm saying you don't let the client decide what ip get.adobe.com is, the proxy server finds it, avoiding the fake ip.

 

So it will work, just it breaks way more than I expected.

 

If it's only for devices where users can install software, then it's just BYOD, so could be "here's a wifi connection we let you use for free, a lot of stuff won't work, that's your fault for keep trying to bypass security", the managed devices can have a more liberal policy.

[Opendium_Steve]

If it's only for devices where users can install software, then it's just BYOD, so could be "here's a wifi connection we let you use for free, a lot of stuff won't work, that's your fault for keep trying to bypass security", the managed devices can have a more liberal policy.

Yep, and that might work for you, but wouldn't work for a lot of schools. You're rapidly getting to the point of saying "we can block VPNs perfectly by just unplugging the network"

[mavhc]

Not really, every website that's allowed will still work.

 

I never really got the point of BYOD anyway, why are pupils bringing devices to connect to the internet via the school?

[DrBeaker]

Not really, every website that's allowed will still work.

 

I never really got the point of BYOD anyway, why are pupils bringing devices to connect to the internet via the school?

 

In a business it made sense at one point. Staff getting tablets, phones etc in order to support the business and be flexible etc.

 

Now with GDPR, encryption, proxy servers etc some of the el-cheapo devices people use it makes no sense. Especially moreso in schools where people will have el-cheapo devices that might not work with the network/guest security e.g. HP Stream devices and Fire tablets used to be a PITA for us.

[Opendium_Steve]

I never really got the point of BYOD anyway, why are pupils bringing devices to connect to the internet via the school?

 

The case for BYOD seems more clear-cut for boarding schools, since the kids are under the school's care 24 hours a day and expect to be able to use their own devices for some of that time. But like it or not, it increasingly seems to be a thing that everyone's expected to offer.

[Primus]

I can name a few that will work fine with no access to DNS, everything closed except HTTPS and HTTPS directed to a transparent filtering proxy. I've done a lot of work dissecting how some of the more sneaky VPNs work - its basically down to a combination of some really clever tricks played by the VPN, combined with the DNS tricks that some of the really big services such as Cloudflair and Facebook use, which makes it difficult to detect and block the VPN without also breaking some services that use DNS tricks for legitimate reasons.

 

As far as I know, Smoothwall, Sophos and Lightspeed still can't successfully block Hotspot Shield, for example (someone correct me on this if I'm wrong though - I know SW have a KB article on blocking hotspot shield, but its years out of date and current versions don't work like that any more). Part of the trouble with mobile devices is that the VPNs often download a list of servers while on 3G and then use that list to find a server to connect to when they go onto the wifi network, so they don't rely on having working DNS, etc. on the wifi network.

 

Just to come back to this, we use a Smoothwall and I've currently got Hotspot Shield blocked - just tested, it cannot connect on our WiFi.

[caffrey]

BYOD here because the school couldn't afford to do 1:1, I'm trying to shift focus to 1:1 chromebooks

 

BYOD android is impossible to support especially with MITM, it just doesn't like it

[Wave9_Lee]

I can name a few that will work fine with no access to DNS, everything closed except HTTPS and HTTPS directed to a transparent filtering proxy. I've done a lot of work dissecting how some of the more sneaky VPNs work - its basically down to a combination of some really clever tricks played by the VPN, combined with the DNS tricks that some of the really big services such as Cloudflair and Facebook use, which makes it difficult to detect and block the VPN without also breaking some services that use DNS tricks for legitimate reasons.

 

As far as I know, Smoothwall, Sophos and Lightspeed still can't successfully block Hotspot Shield, for example (someone correct me on this if I'm wrong though - I know SW have a KB article on blocking hotspot shield, but its years out of date and current versions don't work like that any more). Part of the trouble with mobile devices is that the VPNs often download a list of servers while on 3G and then use that list to find a server to connect to when they go onto the wifi network, so they don't rely on having working DNS, etc. on the wifi network.

 

Just our twopenneth - this can be done on Sophos XG using IPS, cheers

[TwistedHelixis]

We tried for a while using Smoothwall, to some effect, but then every outside agency that comes in needs their VPN to work (NHS, Careers, Religious groups, HR, auditors etc etc...) so it was more trouble than it was worth.

 

Jumped over some posts, sorry if dupicating.

 

Can't you just have a seperate vlan or connection for agencies and vpn connections?

[MrGForse]

What I've done is make sure all certificates going through the web proxy are validated, the only DNS resolvers that can be used are the internal ones with additional DNS filtering, and any traffic that isn't HTTP, HTTPS, STUN, Video Streaming, iTunes, etc. is limited to 250-300kbits/s (compared to our soon-to-be gigabit connection). I also have "Log-on Hours" set in AD which works quite well with BYOD as the kids' devices will flat out disconnect after 22:00.

 

It'll get so frustrating for the kids that they'll just stop using their VPNs, while legitimate users will notice no difference. We're also in the middle of nowhere with thick walls so 4G isn't an issue.

 

After a few months of running reports on the type of traffic going through the firewall, VPN usage (e.g. 30GB on "DNS" or "NTP" per night) has shot right down.

 

It might be very overkill, but it works.

[tom_newton]

Its a whack-a-mole I'm afraid and anyone who tells you that their particular brand of filter/firewall can block *all* VPNs is lying. We recently surveyed a load of schools about VPNs and about 70% of the respondents said that they aren't worried at all by them because their filter blocks them all... That tells me that 70% of schools are a bit deluded.

 

The best you can do is have a filter/firewall that blocks the most popular ones, keep your ear to the ground and whenever you find a new one becoming a problem let your filtering supplier know and give them as much information as you can. You're reliant on having a supplier who can put their development team on the problem and react quickly to figure out how to block the new VPN.

 

I'm actually working on some traffic analysis code to block a new problem VPN at the moment. Some are trivial to block, but some use really sneaky and devious techniques to avoid detection...

 

Also, make sure VPN use is banned in your acceptable use policy, that suitable punishments are handed out, and that VPN use features in your online safety curriculum. Key points:

* VPNs have legitimate uses (e.g. when you're using untrusted networks);

* But you might be breaking an AUP by using one;

* And you might be putting yourself in danger by bypassing malware filters, online safety systems, etc.

* And that you have to trust the VPN provider - if its a free VPN, where are they getting their money? Are they snooping on/meddling with your data in order to monetise you?

 

Also, being very strict with your filters encourages VPN use (and 4G use), so have a think about whether you can relax your filtering a bit - monitoring instead of prohibition. Balance up the benefits of filtering vs. the safeguarding opportunities you miss if overzealous filtering pushes people onto VPNs/4G.

 

Sensible as usual from Steve

[tom_newton]

Just our twopenneth - this can be done on Sophos XG using IPS, cheers

 

I'll double check where we are with hotspot shield, but TBH, an IPS is going to be little use against it, as the traffic is encrypted, so stream based IPS rules are next to worthless.

Usually it's a case of enhancing web filter rules and being sure all your traffic goes via the proxy.

[Primus]

I'll double check where we are with hotspot shield, but TBH, an IPS is going to be little use against it, as the traffic is encrypted, so stream based IPS rules are next to worthless.

Usually it's a case of enhancing web filter rules and being sure all your traffic goes via the proxy.

 

Our Smoothwall is currently successfully blocking HotSpot Shield Tom