Netsweeper false positives on UK Cloud

[danrhodes]

Hi all,

 

Can anybody else confirm if they have these issues? We're struggling to get NSW to get to the bottom on them and provide a fix for us. It is causing no end of stress for the school DSLs/Heads who receive the prevent/porn/etc alerts.

 

See below for a write-up of the problem I have provided to the sites with the issues.

 

What is Happening?

Your site uses a web filtering system called Netsweeper to ensure that inappropriate websites are blocked and do not reach your users. This system checks website addresses against a list of known inappropriate sites. It also performs an additional check to ensure these sites are correctly identified. This extra check involves looking up the IP address of the website to see what other websites might be using the same server.

 

Where is the Problem?

The issue is with the additional IP address check, which can lead to false positives. Many websites today utilise Content Delivery Networks (CDNs) to enhance their performance, making their sites load faster and more reliably. A CDN hosts numerous websites on the same server. When Netsweeper checks an IP address, it might encounter one of these CDN servers, which can complicate the identification process.

 

Here's where things go wrong

- The CDN server can return the URL of any website it hosts as it doesn't not know the website you are attempting to access from just the IP address so returns a random virtual host url.

- Sometimes, the website returned by the CDN is inappropriate or blocked by Netsweeper (for example, a pornographic site).

- Even though the website the user tries to access is safe and appropriate, the system mistakenly flags it because it is inappropriate and hosted on the same CDN server.

 

Example of the Issue with CDN Servers

Let's consider an example with Cloudflare, a popular CDN service. When Netsweeper checks the IP address 172.67.216.176, the reverse lookup process resolves to the website sextasytube.net, among 500+ other domain names, otherwise known as virtual hosts. Cloudflare's servers host many different websites, and when the system performs the reverse lookup, it may return any of these hosted site addresses. If the returned hostname, such as sextasytube.net, is inappropriate, Netsweeper generates an alert. This does not necessarily block the original, safe website you were trying to access, but it does cause unnecessary alerts.

 

Additional Issues

Another issue causing alerts involves computers accessing the IP address 192.168.0.1. This is a standard internal IP address used in many local networks. However, due to incorrect DNS (Domain Name System) records, this IP address incorrectly points to the website whitepower.com. As a result, when computers try to access 192.168.0.1, Netsweeper mistakenly generates alerts because of this misconfiguration.

 

Example Timeline Leading to a False Positive Alert

1. User Logs into the Internet

- A user connects their device to the Internet. The device already has the Netsweeper filter agent installed.

 

2. User Attempts to Access a website

- The user Googles for a site or enters the URL of a website (e.g., newswebsite.com) in their web browser and hits enter.

 

3. DNS Resolution

- The device sends a request to a DNS server to resolve the domain name (newswebsite.com) into its corresponding IP address.

- The DNS server responds with the IP address, e.g., 172.67.216.176, which is managed by a CDN like Cloudflare.

 

4. Netsweeper URL Check

- Netsweeper intercepts the request and checks the URL (newswebsite.com) against its list of known inappropriate websites.

- The URL passes this initial check because it is a safe and allowed site.

 

5. Netsweeper IP Address Check

- Netsweeper performs an additional check by looking up the IP address (172.67.216.176) to see if it is associated with inappropriate content.

- A reverse IP lookup is performed to find the hostname associated with this IP address.

 

6. Reverse IP Lookup Result

- The reverse IP lookup returns a random hostname hosted on the same IP address.

- In this case, the lookup returns sextasytube.net (a pornographic site) hosted on the CDN server.

 

7. Alert Generation

- Because sextasytube.net is identified as inappropriate, Netsweeper generates an alert.

- The alert indicates that the IP address 172.67.216.176 is associated with inappropriate content due to the presence of sextasytube.net.

 

8. Alert Logging

- Netsweeper logs the alert internally, recording details such as the user's attempt to access newswebsite.com, the IP address involved, and the inappropriate domain found (sextasytube.net).

 

9. Alert Notification

- The system sends an alert notification to the designated members of SLT at your site

- The notification includes information about the user's access attempt, the IP address, and the inappropriate domain detected.

Edited June 21, 2024 by danrhodes

[sigma]

I have problems with Netsweeper porn FP's, related, but not on that scale.

 

We are on SBB's hosted Netsweeper. SBB have cloudflare.com and cloudfront.net (for example), as allowed domains in a shared list applied to all users for most customers. I've added decrypts for my reporting purposes, and from what I've observed, it seems work correctly, and has not given us any problems.

 

As I understand it(?), and bearing in mind we are apparently not on a current version, Netsweeper "for the most part" categorises domains rather that urls, so that say, a search for a tractor outline drawing categorised this as porn this week: https://www.munichre.com/content/dam/munichre/hsb/hsb-eil/insights/plant-guide/lifting/Plant_Guide_Lifting_Loading_Shovel_with_Back_Actor_068_HSBEIL.png/_jcr_content/renditions/cropped.3_to_1.png./cropped.3_to_1.png

 

(It is an outline drawing of a JCB type digger) but has been categorised this way because porn has been found on the domain previously.

 

We are not totally reliant on Netsweeper as we also have Impero, which is a very different tool but can filter content that is invisible to Netsweeper.

[PaddyNewman]

Netsweeper can categorise URLs and domains, but if you 'allow' something like cloudflare.com as a URL type, that's that. Everything *.cloudflare com will work.

 

If you have decryption off, everything *.cloudflare.com/porn will work too. It may still work with it on, but at least you can filter on something past that to override the allow.

 

Allow lists should be incredibly specific, being lax with them will open more than intended.

[sigma]

Netsweeper can categorise URLs and domains, but if you 'allow' something like cloudflare.com as a URL type, that's that. Everything *.cloudflare com will work.

 

If you have decryption off, everything *.cloudflare.com/porn will work too. It may still work with it on, but at least you can filter on something past that to override the allow.

 

Allow lists should be incredibly specific, being lax with them will open more than intended.

 

I'm glad you raised this. I totally agree. (With my ISP default settings and decryption on, porn would be a blocked wholeword, mis-spellings could still work).

 

My ISP's settings might explain why we don't see the same FP's at the OP.

 

I'm not sure how to tell if my ISP has rDNS switched on or not as I can't see those settings.

 

Back on topic, I can't see, even from the Netsweeper documentation (although I've probably missed something), how this should be set up.

 

There is brief mention of the issue in the online documentation:

https://helpdesk.netsweeper.com/docs/Flare/Netsweeper_Documentation/Content/Configuration/RDNS_Filtering/RDNS_Filtering.htm?Highlight=reverse%20ip#Reverse_DNS_with_Many