Internet Bandwidth Maxed

[ITGURU]

During the day i get periods where our internet connection is maxed out and causes lag. However it not NOT related to user activity.

Outside of working hours if i do a bulk wake on lan of all machines, as they turn on internet bandwidth will get maxed out for ages, known by traffic graphs.

Port mirroring and internet logs show that the top domain is ipv6.msftconnecttest.com visited x1 times for each machine.

Is this normal?

[Steve21]

That’s one of the urls windows uses to check for internet access so yep nothing wrong with that

 

If it’s specifically at boot time it’s likely to be updates of some kind, are you managing all updates for windows/apps/edge/chrome etc?

 

Sounds like one of the auto upgrade services is running without it being cached, so for example of every machine updates edge at once and you don’t have any delivery optimisation on

 

Steve

[ITGURU]

That’s one of the urls windows uses to check for internet access so yep nothing wrong with that

 

If it’s specifically at boot time it’s likely to be updates of some kind, are you managing all updates for windows/apps/edge/chrome etc?

 

Sounds like one of the auto upgrade services is running without it being cached, so for example of every machine updates edge at once and you don’t have any delivery optimisation on

 

Steve

I thought it was edge but isn't as have the settings in a GPO and when open edge to check for update it says managed by the orgnisation. Also in the web activity logs no other users are top of the list, i have 478 hits of the above url,(al the machines which have booted up) and next top domain has less than 20 hits so if was edge or any other application it would show in the top access domains for the last hour

[chaplic]

Msftconnecttest is part of NCSI so no surprise it’s accessed a lot, but we are talking a handful of bytes going over the wire. Number of page impressions is not equal to bandwidth, for example calling up iPlayer might be a few hits but GBytes if left on all day.

 

Delivery optimsation settings definitely worth a look too

 

My guess would be google chrome. I may or may not have ground a ftse100s internet connection to a halt a few weeks after rolling out chrome with innovative self updating settings…

[ITGURU]

Msftconnecttest is part of NCSI so no surprise it’s accessed a lot, but we are talking a handful of bytes going over the wire. Number of page impressions is not equal to bandwidth, for example calling up iPlayer might be a few hits but GBytes if left on all day.

 

Delivery optimsation settings definitely worth a look too

 

My guess would be google chrome. I may or may not have ground a ftse100s internet connection to a halt a few weeks after rolling out chrome with innovative self updating settings…

 

Definitely not chrome, we don't use it. If i look at my core switch uplink ports there is consistent traffic going out/from each uplink even to our IT suites of 30 machines until it all drops off together on the core switch uplinks and internet bandwidth.

[georgeescott]

Does your firewall log enough to tell the highest bandwidth usage vs highest requested domains?

Or does your core switch support NetFlow/sFlow?

[ITGURU]

Does your firewall log enough to tell the highest bandwidth usage vs highest requested domains?

Or does your core switch support NetFlow/sFlow?

Not firewall, but external web filter does. top domain is ipv6.msftconnecttest.com and shows every machine accessing the url

top domain is ipv6.msftconnecttest.com however even from home i get

ERR_NAME_NOT_RESOLVED

 

so i'm not sure whats going on

[georgeescott]

Not firewall, but external web filter does. top domain is ipv6.msftconnecttest.com and shows every machine accessing the url

 

Any other information your external web filter shows?

What are the other top domains being hit?

 

however even from home i get ERR_NAME_NOT_RESOLVED

so i'm not sure whats going on

 

I posted about this in a different thread: https://www.edugeek.net/showthread.php?t=231552

Most of the NCSI domains used by Microsoft don’t work. I’ve logged a ticket with MS about it.

As others have mentioned, this would primarily be a few bytes for the .txt file or just DNS lookups. It’s unlikely to be the issue.

Edited February 12, 2023 by georgeescott

[ITGURU]

Filter provider has advised me that if a domain is bypassed for SSL interception/authentication it wont show in the logs! So i suspect its one of the agents that doesnt have randomisation , Some testing to do this week.

thanks

[georgeescott]

Filter provider has advised me that if a domain is bypassed for SSL interception/authentication it wont show in the logs! So i suspect its one of the agents that doesnt have randomisation, Some testing to do this week.

thanks

 

Ah.. could well be it then. Start by going through your exemptions list I guess.

 

Worth looking at NetFlow/sFlow though, it helped me diagnose bandwidth issues as a previous place using a free instance of PRTG for the sensors/graphs.

 

Other than Windows Updates, you could look at Windows startup, task scheduler and services to spot anything running at startup. Autoruns is helpful http:// https://learn.microsoft.com/en-us/sysinternals/downloads/autoruns

[PaddyNewman]

Filter provider has advised me that if a domain is bypassed for SSL interception/authentication it wont show in the logs! So i suspect its one of the agents that doesnt have randomisation , Some testing to do this week.

thanks

 

Would be odd to do that...

 

However, your provider should be able to provide something akin to 'heaviest users' or at least a capture from a firewall level at periods of high load.

 

General web traffic, especially check ins to a website, will not swamp your connection. They are tiny requests. We have seen evidence of a single Windows 10 PC doing updates max out a 100Mb leased line, can't imagine anyone goes to MS directly these days for all their machines, but its easily swamped if you do.

[KK20]

If you have PC maxing out updates then either look at caching (if you have the necessary licensing) or set a policy to only allow PCs P2P updates and have a few "honeypot" PCs that download updates during downtime.

[PotNoodleTech]

Updates can be a problem - windows, chrome os, IoS, adobe cloud, etc it's all gigabytes and gigabytes per machine.

 

Cache me if you can!!!

[3s-gtech]

When we first rolled out Windows 10, we’d be maxing out our connection often (100Mbit then).

 

I installed a PFSense firewall to give me a better idea what was happening - it was hundreds of heavy calls to Akamai hosting telemetry, app updates etc. I thought I’d crushed all of that out of the image, obviously not.

 

I actually bandwidth limited most of the CDNs they used, which sorted it. Never gave issues after that. Since going back through our policies, WSUS config and two versions of Windows 10 I was able to turn those limits back off, but we also have 1Gbit now.