[1703, cu] Bandwidth being *slammed* by *.tlu.dl.delivery.mp.microsoft.com

[LemonEntry]

I've disabled the Store, and configured my Windows 10 clients to check our local WSUS server for updates, defer Feature Updates for 365 days, prevented deferral policies from causing scans against Windows Update, prevented connection to any WIndows Update Internet locations, and removed access to use all Windows Update features. I am still seeing 300GB of traffic per day from subdomains of tlu.dl.delivery.mp.microsoft.com.What else can I do to kill this traffic? Would blocking these domains on the Smoothwall cause problems?

[atcoates]

This help? https://blogs.technet.microsoft.com/windowsserver/2017/01/09/why-wsus-and-sccm-managed-clients-are-reaching-out-to-microsoft-online/

[jnfarmer]

We've got the same issue using Smoothwall as well, thankfully we only have a smattering of Windows 10 devices so it's not to bad. I know that doesn't help you but hoping if it is more of us, might get some attention from Smoothwall guys.

[free780]

This was an issue in 1511 if you had the c drive hidden. It was to do with spotlight search. I had to find reg keys to disable it. I thought it was resolved in 1607. The store will reach out for updates and the Ms solution is WUDO rather than WSUS. If clients are failing due to WUDO behind a proxy it may be why your seeing so much traffic. You can switch WUDO to use bits instead. Each client will download app updates from the Internet but at least they succeed. Edited November 10, 2017 by free780

[LemonEntry]

So from reading the blog post from @atcoates I shouldn't be using any upgrade deferral settings. These cause Windows 10 to operate in "Windows Update for Business" mode and check against Microsoft Update for patches. The fix is to just leave the original WSUS settings in place and ignore everything else.

I'll let you know if it works!

 

Edit: I had the policy "Select when Preview Builds and Feature Updates are received" enabled and configured which is inside the subsection Computer Policies\Administrative Templates\Windows Components\Windows Update\Windows Update for Business. Now I think about it, I'm managing availability of Feature Updates via WSUS, so I don't need to defer them.

 

If that's not how it works and I end up with a mixed 1703 / 1709 environment, I'll go become a goat herder. I've had enough of this tomfoolery.

Edited November 13, 2017 by LemonEntry
Additional information

[jnfarmer]

We don't have that policy set but we do have 'Configure Automatic Updates' set to enable and 'Do not connect to any Windows Update Internet location' not configured. I'm guessing that as we are using SCCM we shouldn't need the first one on and should enable the second.

[LemonEntry]

24 hours after the policy was applied:

 

7.tlu.dl.delivery.mp.microsoft.com 84.8 (GBytes)

3.tlu.dl.delivery.mp.microsoft.com 74.4 (GBytes)

2.tlu.dl.delivery.mp.microsoft.com 59.2 (GBytes)

 

I'm tempted to just blackhole the domains.

Edited November 14, 2017 by LemonEntry

[JRowley]

I removed the deferral settings we had in place by mistake and that solved the problem for us within an hour, I would block the domains in the mean time if I were you.

[jnfarmer]

We don't have the deferral policies set and don't like the idea of just blocking domains, would rather manage it properly.

[LemonEntry]

I'm looking at the Smoothwall Bandwidth module stats... Microsoft Update (Windows Update and BITS) is using 0KB/s, but I'm still seeing the domain filter past in the Recent Blocks list on the Dashboard, and the total bandwidth used isn't decreasing. And I still have slow internet access everywhere -_-

 

Maybe it's DNS? Isn't it always?

[jnfarmer]

Just spoken to Smoothwall and they've put bandwidth limiting on the Software Updates group, will see how that goes.

[LemonEntry]

Turns out the policy change did resolve the issue, I just had to wait a few days for the policy to apply to sufficient workstations to make a difference to the bandwidth readout.

[Warwick_Tech]

So from reading the blog post from @atcoates I shouldn't be using any upgrade deferral settings. These cause Windows 10 to operate in "Windows Update for Business" mode and check against Microsoft Update for patches. The fix is to just leave the original WSUS settings in place and ignore everything else.

I'll let you know if it works!

 

Edit: I had the policy "Select when Preview Builds and Feature Updates are received" enabled and configured which is inside the subsection Computer Policies\Administrative Templates\Windows Components\Windows Update\Windows Update for Business. Now I think about it, I'm managing availability of Feature Updates via WSUS, so I don't need to defer them.

 

If that's not how it works and I end up with a mixed 1703 / 1709 environment, I'll go become a goat herder. I've had enough of this tomfoolery.

 

Hi LemonEntry -

 

I've been tearing my hair out over this issue for a few weeks now http://www.edugeek.net/forums/windows-10/190764-windows-store-updates-stuck-cannot-install-apps-high-traffic.html

 

do you have a breakdown of what you did to resolve this? I've turned off store updates and changed 'download modes' but with no joy - Do you have special rules setup on your smoothwall?

 

thanks in advance

 

Edited November 27, 2017 by Warwick_Tech

[LemonEntry]

It was configuring "Select when Preview Builds and Feature Updates are received" in Policies\Windows Components\Windows Update\Windows Update for Business\ that caused the issue. Windows Update for Business is for networks without their own WSUS server and connects to Microsoft Update. Resolving it involved setting this back to "Not configured" and also setting the following policies as a "scorched earth" solution:

 

- Do not allow update update deferral policies to cause scans against Windows Update: Enabled

 

- Remove access to Windows Update features: Enabled

 

- Do not connect to any Windows Update Internet locations: Enabled

[Warwick_Tech]

Thanks for this -

 

We have disabled all updates and will wait the 24/48 hours (we're seeing minor fluctuations in speed)

 

On a side note we have also disabled the Windows Update Service and the MSStore

 

- Turn off Automatic Download and install of Updates

 

- Turn off the Store Application

 

the only thing is we use device manager to download the latest drivers and I'm not sure that will now work - but it's a case of going through bit by bit until a balance is found.

[ADMaster]

I can say that with that setting configured device driver updates will not happen. I just had an issue today looking for a print driver. It could not be downloaded from the manufacturer all searches pointed to using windows update.

I had to revert the policy of do not allow internet locations to not configured.

[LemonEntry]

I can say that with that setting configured device driver updates will not happen. I just had an issue today looking for a print driver. It could not be downloaded from the manufacturer all searches pointed to using windows update.

I had to revert the policy of do not allow internet locations to not configured.

 

Indeed that is the case, but I don't perform driver updates via Windows Update anyway. Updates are tested on a single workstation and deployed using MDT during the next refresh, or via PDQ Deploy if it's mid-term and can't wait (new USB device, for example).

[Warwick_Tech]

Ok, so here's what I've been informed by Smoothwall -

 

The issue seems to be caused by the lack of support for HTTPS Range Requests on Smoothwall UTM's - This is being resolved in the next update, but there is no ETA for this.

 

The 'fix' for now is to just whitelist all software updates to bypass the smoothwall, but we have yet to see if this works when you disable the policies put in place suggested by LemonEntry

 

Long story short, smoothwall are working on a fix for something Microsoft have done - But if anyone has any solutions in the meantime make sure to mention them, but it may be difficult as we all have different configurations on our boxes.

[LemonEntry]

Ok, so here's what I've been informed by Smoothwall -

 

The issue seems to be caused by the lack of support for HTTPS Range Requests on Smoothwall UTM's - This is being resolved in the next update, but there is no ETA for this.

 

The 'fix' for now is to just whitelist all software updates to bypass the smoothwall, but we have yet to see if this works when you disable the policies put in place suggested by LemonEntry

 

Long story short, smoothwall are working on a fix for something Microsoft have done - But if anyone has any solutions in the meantime make sure to mention them, but it may be difficult as we all have different configurations on our boxes.

 

So, a little update... It's not really fixed. The domain just changed. I now see traffic to download.windowsupdate.com instead. It's not funny anymore.

 

My "fix" is to whitelist Software Updates for my WSUS server only. If Microsoft can't obey it's own policies with regards to updating, I'll cut them off at the border. I have Software Assurance; I'll open a ticket with Microsoft and see what they suggest. Probably "Install 1709 yolo!!1"

Edited December 1, 2017 by LemonEntry

[jnfarmer]

So, a little update... It's not really fixed. The domain just changed. I now see traffic to download.windowsupdate.com instead. It's not funny anymore.

 

My "fix" is to whitelist Software Updates for my WSUS server only. If Microsoft can't obey it's own policies with regards to updating, I'll cut them off at the border. I have Software Assurance; I'll open a ticket with Microsoft and see what they suggest. Probably "Install 1709 yolo!!1"

 

We're on 1709, not making a difference! We're quite lucky in that we were due to upgrade our pipe early next year, we brought it forward a few months and have upgraded to 500Mb/s, so we've stuck bandwidth limiting to 100Mb/s on those URLs during school hours and then during the night it goes to town.

 

May try whitelisting for the WSUS only though as well.

[Warwick_Tech]

May try whitelisting for the WSUS only though as well.

 

The only issue with only whitelisting for WSUS is as I understand the Store doesn't get it's updates from there - Even if you turn store updates off they still download (??)

 

Our solution was to release the updates via whitelist wholesite so store updates could install and still use WSUS and 'search windows update' for PC's

 

The Store updates are now working across the site, so the theory is the pipe will slowly clear as all machines update their store apps, at which point we can turn off bandwidth limits and Smoothwall will have the HTTPS Range Requests implemented.

 

Edited December 6, 2017 by Warwick_Tech

[jnfarmer]

 

I think we've downloaded Microsoft's entire catalogue in the last 24 hours. On a more positive note our bandwidth is only hitting 80 Mbit/s this morning so hopefully we're past the worst of it.

[LemonEntry]

[ATTACH=CONFIG]46615[/ATTACH]

 

I think we've downloaded Microsoft's entire catalogue in the last 24 hours. On a more positive note our bandwidth is only hitting 80 Mbit/s this morning so hopefully we're past the worst of it.

 

It's more like you've downloaded November's patches (Windows, Office, extras?) on your entire site. That's why I blacklisted those URLs for all but the WSUS server; Clients can't bypass the GPO settings if they tried. It's a DoS, AFAIC. The threat gets mitigated.

[jnfarmer]

It's more like you've downloaded November's patches (Windows, Office, extras?) on your entire site. That's why I blacklisted those URLs for all but the WSUS server; Clients can't bypass the GPO settings if they tried. It's a DoS, AFAIC. The threat gets mitigated.

 

What was the web filter policy you created to do this? Trying to see how I could fit on in with the rest of our policies.

[3s-gtech]

Since turning off the ability of our machines to go to the internet for updates even though they use WSUS (with the handy GPO that was introduced) we have greatly cut this down. Machines still make small, occasional calls out but they're not grabbing updates now.