Schools Broadband : Our systems have detected unusual traffic...

[paulkerton]

Looks like the issue with Google Calendar / Google and unusual traffic is back yet again. This has been going intermittently since November, but this time it's affecting Google Sites so we can't use that tool anymore.

I tether off and it works just fine.

 

One day I hope that things will just work.

[DavidYoung]

Do you know how Schools Broadband present school IP addresses to the internet? Does each school have their own public IP address or are multiple schools behind a shared IP? If it is the latter then it is likely that something happening in one school such as compromised devices could be sending malicious traffic which is getting picked up.

 

Our provider gives each school their own public IP address and I think I have seen this just once in the last few years.

[paulkerton]

I believe we have our own IP address presented to the world, but it's very unlikely to be a compromised device as we are a Chromebook school.

[DavidYoung]

I believe we have our own IP address presented to the world, but it's very unlikely to be a compromised device as we are a Chromebook school.

 

That's strange then. I remember the time this happened to us with Google sites was when we moved to a new proxy server and suddenly had 5 schools presented behind a previously dormant IP address so I think needed a bit of warming up.

 

It's probably worth checking with SB that there isn't any traffic associated with your IP address that isn't from your school and that there haven't been any network/IP changes recently.

[PaddyNewman]

I believe we have our own IP address presented to the world, but it's very unlikely to be a compromised device as we are a Chromebook school.

 

Bear in mind ANY Chrome/Android based device is doing chit chat to Big Goog' and when a lovely student/visitor with an android device rocks up with naughty VPN software on there, the device has 2 connection mechanisms... you might have seen them when installing certs, WiFi or VPN and Apps.

 

Its talking home on BOTH of those mechanisms, when one (VPN and Apps) talks on public IP 1.2.3.4 and WiFi talks on 2.4.6.8, Big Goog is suspicious and will likely trigger this. We've demonstrated this, tested and confirmed this is what happens locally, hell of a load of chit chat from Android devices, my Pixel is constantly talking and I mean constantly... I have nothing to hide, but VPNs do, and Google does not like your traffic to be obfuscated or amended in any way so they can't shape those adverts and results.

 

I would also look at that as a possibility, sometimes it isn't even a student, but a visitor/parent/staff member.

 

I see it maybe twice a month over a few thousand schools, reports show odd activity. Given BYOD/Guest and lack of SSL inspection, that is my go to investigation point.

[FN-GM]

Bit of the sledge hammer approach but if you implemented IPv6 the concept of sharing a public IP goes. Each client will have it's own IP and these kind of things will be prevented.

[SchoolsBroadband]

evening @paulkerton.

@PaddyNewman is correct (thanks btw). We have also seen similar things happen to BBC when a user from a specific IP tries to use a VPN. This is not a Schools Broadband thing but a Google / website thing.

 

Another way around it in part is to give your BYOD network a different outgoing IP from the rest of the school if the use of a VPN comes from this network (assuming you have byod on a different VLAN?) You can also track via the Fortigate and Netsweeper logs who is trying to use some types of VPN via various means which we can help you with to an extent if you are struggling to hunt down the culprit. e.g. IP endpoint, URL's etc

 

I'm also assuming your school has its own external IP for all web traffic? Some of our school share for different legacy reasons but we do have numerous schools on their own IP.

 

Please so send me a PM and I'll go escalate for you.

 

Thanks

 

Dave

[DavidYoung]

Bear in mind ANY Chrome/Android based device is doing chit chat to Big Goog' and when a lovely student/visitor with an android device rocks up with naughty VPN software on there, the device has 2 connection mechanisms... you might have seen them when installing certs, WiFi or VPN and Apps.

 

Its talking home on BOTH of those mechanisms, when one (VPN and Apps) talks on public IP 1.2.3.4 and WiFi talks on 2.4.6.8, Big Goog is suspicious and will likely trigger this. We've demonstrated this, tested and confirmed this is what happens locally, hell of a load of chit chat from Android devices, my Pixel is constantly talking and I mean constantly... I have nothing to hide, but VPNs do, and Google does not like your traffic to be obfuscated or amended in any way so they can't shape those adverts and results.

 

I would also look at that as a possibility, sometimes it isn't even a student, but a visitor/parent/staff member.

 

I see it maybe twice a month over a few thousand schools, reports show odd activity. Given BYOD/Guest and lack of SSL inspection, that is my go to investigation point.

 

That makes a lot of sense, and explains our experience when moving to the proxy setup as that was a new IP address but a lot of the traffic would also be going directly out of the schools' own IP with a similar outcome. I wonder if Google just sort of got used to it after a few days once the traffic patterns were analysed.

 

I guess this is just one of the problems when are beholden to our new cloud overlords! @paulkerton, as it is affecting core Google Workspace applications (Sites and Calendar), have you reached out to their support?

[paulkerton]

Well I've spent a good chunk of this morning digging through our Audit Logs between 9am and 3pm from yesterday to see if there were VPNs causing this. We have one or two issues with staff, but I think I've found the issue.

It seems that SBB is at any particular time of the day assigning one of three IP addresses our way. So what's happening is we have people using Google and flipping between these three IP addresses, which is clearly making Google suspicious and triggering it.

 

 

I can go to http://www.whatismyipaddress.com and http://www.whatismyip.com at the same time and both will give me a different IP address.

[PaddyNewman]

That will cause very similar problems, if you have multiple addresses going out and a device is going over that constantly, it will ring alarm bells somewhere.

 

I don't know the infrastructure there, but if the option to NAT all your school based traffic to 1 non changing address is possible that would stop the flipping between IPs, load balancing/multiple routes out? However, could be a number of things depending on routing, firewalling or filtering. Depending on the Netsweeper set up, you'll could see various NAT IPs depending if you aren't in explicit modes or full source IP maintaining transparent modes, but likely something for SB to look at rather than poking my nose in too far