[Gsuite] SMTP connection fails after X emails

[timbo343]

We have various services that provide us with notifications and one of them is causing us issues when using 2FA with Gsuite (an App Password is in use at the moment with the software). This application uses port 587 and connects via smtp.gmail.com to send emails.

 

It seems to fail after 75 emails have been sent in succession, we then see an error on the application stating: tempoary system probem. We've detected an unusual rate of unsolicited mail originating from this IP address. To prevent spam, mail sent from this IP address has been temporarily blocked..

 

Later in the logs we see Username and Password not accepted by the mail server

 

This has only just started happening since we moved over to Gsuite.

 

Is there a way to omit our IP address so we are able to send bulk emails via Gsuite?

[localzuk]

G Suite have rate limiting on their addresses via SMTP - they don't really want lots of email like this sending. Their limit is currently 2000 a day, on a rolling basis. This works out around 83 an hour.

 

Your options are - set up a local SMTP server, or use a different service (something like Sendgrid). We used to handle rate limiting by having our own relay server (Postfix) which also rate limited how quickly it forwarded emails onwards to Office 365.

Edited February 24, 2020 by localzuk

[timbo343]

Thanks, i got the impression there wasn't a limit for Education GSuite mailbox users unless i read it wrong in this article > https://support.google.com/a/answer/2956491 under heading Review sending limits for the SMTP relay service.

 

Looks like we might have to review this especially for our Bromcom emails to parents.

 

Limits per user

The maximum number of messages a user can send in a 24-hour period is 10,000. However, this can vary, depending on the number of user licenses in your G Suite account.

The maximum number of unique recipients allowed per user per 24-hour period is approximately 130 times the number of user licenses in your G Suite account.

Notes:

 

If a user exceeds either of these limits, the sender receives the error "550 5.4.5 Daily SMTP relay limit exceeded for user."

 

Google servers can handle more than 100 recipients per transaction. However, RFC 5321 limits might block some transactions.

 

The message count is based on the address of the envelope sender presented during the SMTP relay transaction. If the envelope sender is not a registered user, then the per-user limits don't apply. Addresses in the From: and Reply-to: fields are not considered. Nor do we consider the address presented during SMTP authentication, which is turned on if you select the Require SMTP Authentication option, described below.

 

Any sender, whether or not they're a registered user, can also be prevented from sending messages if a customer relay limit has been reached. In this case, they see a different error message based on limits per customer.

 

Important: SMTP relay limits are lower for trial accounts. To increase SMTP relay limits for a trial account, you must pay a Google-generated bill. This is different than increasing Gmail limits, which can be initiated by ending your free trial.

 

Limits per customer

These limits are determined by the number of user licenses in your G Suite account. For small customers, these limits come into effect much earlier than the limits per user.

 

There are two per-customer limits:

 

The maximum number of total recipients allowed per customer per 24-hour period is approximately 130 times the number of user licenses in your G Suite account, with an upper bound of 4,600,000 recipients per 24-hour period for large customers. If a customer exceeds this limit, users see the error "550 5.7.1 Daily SMTP relay limit exceeded for customer."

The maximum number of total recipients allowed per customer in a 10-minute window is approximately 9 times the number of user licenses in your G Suite account, with an upper bound of 319,444 recipients per 10-minute window for large customers. If a customer exceeds this limit, users see the error "450 4.2.1 Peak SMTP relay limit exceeded for customer."

Additional notes

The per-user recipient limits are for unique recipients, while the per customer limits are for total recipients. For example, when a given user relays 1000 messages to RecipientA and another 1000 to RecipientB, this counts as 2 for the per user limit, but 2000 for the customer limit.

Customer limits apply to all messages relayed by the customer, including messages relayed from any of the customer's secondary or subdomains, as well as messages relayed from external addresses where one of the customer's domains is presented in the “helo” argument.

Customer limits are lower for customers who have not yet paid a bill: 100 total recipients per day, 50 total recipients in a 10-minute window.

There are separate per-user sending limits published here for sending email using Gmail rather than SMTP relay. The SMTP relay and Gmail user sending limits are independent and are counted separately from each other.

[jthompson]

You can create a dedicated SMTP relay service in G Suite and point you service or application to smtp-relay.gmail.com rather than to smtp.gmail.com. The usage limitations are more forgiving with an SMTP relay service.

 

Regular sending limits: https://support.google.com/a/answer/166852?hl=en

SMTP relay service: https://support.google.com/a/answer/2956491

 

Limits per user

The maximum number of messages a user can send in a 24-hour period is 10,000. However, this can vary, depending on the number of user licenses in your G Suite account.

The maximum number of unique recipients allowed per user per 24-hour period is approximately 130 times the number of user licenses in your G Suite account.

[timbo343]

You can create a dedicated SMTP relay service in G Suite and point you service or application to smtp-relay.gmail.com rather than to smtp.gmail.com. The usage limitations are more forgiving with an SMTP relay service.

 

Regular sending limits: https://support.google.com/a/answer/166852?hl=en

SMTP relay service: https://support.google.com/a/answer/2956491

 

The SMTP relay service had already been setup, i didn't know about the smtp-relay.gmail.com address.

[timbo343]

You can create a dedicated SMTP relay service in G Suite and point you service or application to smtp-relay.gmail.com rather than to smtp.gmail.com. The usage limitations are more forgiving with an SMTP relay service.

 

Regular sending limits: https://support.google.com/a/answer/166852?hl=en

SMTP relay service: https://support.google.com/a/answer/2956491

 

PS - this has now worked for application / mail account that requires mass email notification.

[jthompson]

We did run into that limitation of the relay service at one point towards the end of last Autumn term, when reports and all sorts of other bulk messages were going out to parents. It was a pain as there's no real way to know what emails have been dropped. I've argued for some degree of organisational restraint when it comes to the volume of bulk emails to parents, but as it is it's a bit of a staff free-for-all at the moment and nobody want that job. I think long term we'll need to do as @localzuk suggests and either go for an on-prem SMTP server (do I really want the security and availability headaches?) or a commercial mailing service like SendGrid or ClickSend. And then wait for parents to grumble about all the toot they get sent.

[timbo343]

Does the offer of a 3rd party outweigh the headaches of an internal SMTP server? If so for the small amount it is a month for a SendGrid account can the school refuse it even if it's to be used a couple of times a year.

[localzuk]

From what I can tell, the relay service uses its own limits - but you weren't using the relay service, you were using SMTP with app passwords. So that's probably the issue.

 

That said, I'd worry about any individual service sending 75 notification emails in such a short period as to trigger the normal SMTP limits too.

[timbo343]

From what I can tell, the relay service uses its own limits - but you weren't using the relay service, you were using SMTP with app passwords. So that's probably the issue.

 

That said, I'd worry about any individual service sending 75 notification emails in such a short period as to trigger the normal SMTP limits too.

 

The application i am using is a password reminder which is set to send an email to those users whose password is expiring in *ConfiguredNumberOfDays* so this morning there was over 600 emails being sent out from the server.

[Shaun_Dark_Lord]

Sorry to dig up an old thread, but this topic has kept me occupied for most of the last week.

 

The issue popped up for us when switching from Bromcom's limited SMTP service to Google SMTP. As a Google workspace customer, the advertised daily SMTP limit is 130x the number of users, so ~2000000 for us (we're a MAT). The maximum send rate in any 10 minute period is 9x the number of users, so ~140000 for us. SOURCE

 

But, we were consistently hitting a limit after ~100 emails. I wrote a Powershell script to determine where the issue was occurring, and discovered that;

• Powershell doesn't do a great job of displaying SMTP error codes.
  
• I could replicate the behavior seen in Bromcom - Failures after ~100 emails.
  
• The issue related to the authenticating sender account - Attempting to send an email from a different external IP using the same credentials failed instantly if the limit had been reached within the last few minutes.
  

 

I fired up a copy of Pegasus mail, and attempted to send an email whilst rate limited - This time I received an error;

454 4.7.0 Too many login attempts, please try again later. c5sm123129wrn.18 - gsmtp

 

A Google search (or many) found similar issues, but the most relevant was this one.

"OK, so this appears to be expected as Jira is connecting to our servers for every message you send and they are connecting to smtp.gmail.com. G Suite SMTP relay servers have protections in place to guard against Denial of Service (DoS) attacks. To avoid conflicts with these protections, SMTP agents that send large amounts of mail through smtp-relay.google.com and smtp.gmail.com should reuse connections, sending multiple messages per connection. This is also known as connection caching. For instructions on this process, please contact your email server software provider. "

 

I modified my test script, and was able to send 1000 emails successfully if I simply used the same initial connection for all messages.

 

Hope this helps someone.

 

Original Code;

 

Modified Code;