Exchange On-Prem Messages Delayed Delivery to Exchange Online

[Squan_JRP]

We have an on-premises Exchange server (2019) which relays messages from on-premises clients (scanners, servers, etc.) and most of our user mailboxesare in Exchange Online.

Over the past few days, some (but not all) messages relayed from them on-premises server to exchange online get delayed for up to nearly 2 hours. You can tell from the timestamps in the headers of the messages.

 

 

 

As an example, this morning, I ran the Get-Queue command on my on-premises server. This showed 32 messages in the queue to my exchange online domain (myorg.mail.onmicrosoft.com).

Some of the messages had been delayed over an hour.

 

 

When I ran Get-Queue -Identity "8137" nearly all the messages showed as being in "ready" status.

[html]

[/html]

Nothing in queue was being delivered.

I then ran Retry-Queue -Identity "8137" and within about a minute all the messages were delivered.

As I noted, this has happened a few days in a row now and not sure why. I've rebooted the server and it still persists, sometimes.

Can anyone suggest next steps to investigate why this is happening?

[chaplic]

Is your on prem boxes on the very latest patch levels? I have a memory about MS starting to deliberately delay messages from on prem if the sending servers weren't the latest version A good google showed nothing, but the whisky I had last night wasn't THAT strong..

[Squan_JRP]

Thanks, we were a couple builds behind so we'll give that a try and see what happens.

[HPlum78]

There are some setting in Exchange that govern how mail is sent to a single destination like exchange online or while using Exchange in a hybrid setup and relaying messages through on-prem exchange:

SmtpConnectorQueueMessageCountThresholdForConcurrentConnections - set in the EdgeTransport.exe.config file the default is something I cannot remember but you can set this to 2 from memory (I will let you look it up as you need to add a line to that file to override the default) changing this essentually puts less messages on the queue as Exchange will create more parallel queues.

 

MaxPerDomainOutboundConnections - again not sure of the default but you can increase this using set-transportservice cmlet and setting it to a higher value (one that I look after is set to 40 and i bet that I would have taken that from some documentation)

 

and as @chaplic says:

https://techcommunity.microsoft.com/t5/exchange-team-blog/throttling-and-blocking-email-from-persistently-vulnerable/ba-p/3815328

Timeline is here:

https://techcommunity.microsoft.com/t5/exchange-team-blog/update-on-transport-enforcement-system-in-exchange-online/ba-p/3827774

 

But unless you are running Ex2013 this should not be a thing YET!

 

Oh and you lot made me look here is a link to what I outlined above!

https://learn.microsoft.com/en-us/exchange/troubleshoot/mailflow/email-stuck-queues

Edited February 14, 2024 by HPlum78

[Squan_JRP]

Thank you for these resources, I'll work through the last article you linked to.