Exchange 2016 Certificate Problem

[smarties11]

Hi All,

 

We use a public certificate to encrypt our Exchange traffic, and we have this assigned to IIS and SMTP services. It's a wildcard cert with SANs for autodiscover etc.

 

The existing certificate expires on Friday, and I have just installed the replacement certificate.

 

The certificate imports fine, however when I then attempt to assign the IIS and SMTP services, I get an error. This happens whether I use ECP or EMC.

 

The error is below. Google research suggests that this is because the 'network service' account doesn't have full permissions to access the private key, however I've tried adding the network service account to the private key permissions (full control) in the certificates mmc and I still get the same error.

 

When I view existing certificate in both ECP or EMC, it shows the new one installed and says it is assigned to both SMTP and IIS?? It presents the new certificate correctly when I try OWA but I'm not sure how I would test SMTP.

 

Any ideas?

 

A special Rpc error occurs on server S-XXXXXX: Could not grant Network Service access to the certificate with
thumbprint xxxxxxxxxxxxxxxxxxxxxxxxxxxxx because a cryptographic exception was thrown.
   + CategoryInfo          : NotSpecified: ( [Enable-ExchangeCertificate], InvalidOperationException
   + FullyQualifiedErrorId : [server=S-XXXXXX,RequestId=0ce68699-de3e-4ac4-9c7a-094b538d302d,TimeStamp=01/06/2020 13:
  59:03] [FailureCategory=Cmdlet-InvalidOperationException] D9EA479C,Microsoft.Exchange.Management.SystemConfigurati
 onTasks.EnableExchangeCertificate
   + PSComputerName        : s-XXXXXX.xxxxxxxx.co.uk

Edited June 1, 2020 by smarties11

[Steve21]

I'm assuming you've got the cert in the right places like Trusted root? Rather than personal etc?

 

Steve

[smarties11]

I've got the cert itself in 'personal'. The intermediate in 'intermediate certification authorities' and the root in 'trusted root certification authorities'. This is how I've always done it in the past, and is where the exiting certificate is located?

 

EDIT: I've assigned services back to the old cert, removed the new and re-imported. Exchange itself places the cert in 'personal' (local computer). If I move it anywhere else, Exchange doesn't see it - so surely this is the correct place? I've never known them to be anywhere else since Exchange 2003.

 

I'm starting to wonder whether this is a non-issue. I get the same error when assigning SMTP to any cert - whether it's the old one, new one, or the self-generate one. Hmmm.

 

I also noticed that when I re-imported the new cert and assigned services, and then view certificates that SMTP is not assigned to it. As soon as I grant read access to the private key for 'NETWORK SERVICE' it shows as SMTP assigned. I don't even have to run the command to assign again.

 

I'm *hoping* that the issue is that for whatever reason, Exchange doesn't have the rights to adjust private key permissions - and so adding them manually corrects that. However, I still can't remove the old certificate, I get an error saying that the internal transport certificate cannot be removed. So Exchange is still using the old cert for internal transport and I have no idea how to change that. I'm wondering if it will automatically use an alternative certificate (i.e. new one) once the old one expires on Friday? Really hoping that we don't have SMTP issues on Friday :-/

Edited June 2, 2020 by smarties11

[3s-gtech]

How are you assigning the cert to services, Powershell? As I use Let's Encrypt for Exchange, I have to assign certs TOO OFTEN so I've been through a few quirks. Last one was having to force delete the old certs as the ECP wouldn't let me remove them. I run a PS script which takes the new LE cert and assigns it to the new services. The old one stays assigned to some services. I have the situation currently where I have two valid certs, one doing IMAP, POP and SMTP and another doing IMAP, POP and IIS. Parts of the renewal don't automate well at all.

 

I can share some parts of the renewal script and process if that helps?

[smarties11]

How are you assigning the cert to services, Powershell? As I use Let's Encrypt for Exchange, I have to assign certs TOO OFTEN so I've been through a few quirks. Last one was having to force delete the old certs as the ECP wouldn't let me remove them. I run a PS script which takes the new LE cert and assigns it to the new services. The old one stays assigned to some services. I have the situation currently where I have two valid certs, one doing IMAP, POP and SMTP and another doing IMAP, POP and IIS. Parts of the renewal don't automate well at all.

 

I can share some parts of the renewal script and process if that helps?

 

Hi,

 

Thanks, yes I'd be interested to see your scripts please! I had thought about going down the LetsEncrypt route this time around as our certificate renewal price was stupid, but managed to get them down to a sensible price in the end. We have a wildcard cert with SANs for Exchange and use it on quite a few services so didn't want the hassle on the 90 day renewal - especially when on things like Ruckus you can't automate it.

 

I used ECP initially however when I ran into issues I switched to PowerShell and both give exactly the same error. In the end I bit the bullet and deleted the old certificate from the local store, hoping that would force Exchange onto the new cert for SMTP. That seems to have done the trick. And mail flow is still working as expected. I'm pretty sure it's just some sort of bug in Exchange. Fingers crossed...

[3s-gtech]

Sounds like you basically did what I did, put two fingers up to Exchange and wiped the old cert manually! I did that from MMC, was a bit nerve wracking but so is everything with it. Will probably migrate to Exchange Online next year to save some hassle.

 

I’ll see if I can adapt the PS to post it.

[smarties11]

Yeah, that's pretty much it! I expected SMTP to fall back to non TLS but it's still running on TLS so I guess Exchange was like 'OK OK I'll use the bl00dy new certificate then' and got on with it!

 

I reckon Exchange Online will be on the cards for us in the not too distant future, as well as Teams, Stream etc. My only bugbear is the backup; I know many Schools do just rely on Microsoft's resiliency for this but I still feel we need backups of our cloud data and I haven't yet found a product that does this anywhere near as affordable as maintaining our own internal backups. There have been cases of Schools 'losing' their cloud data in malicious attacks (including one School literally down the road from us who had to start their network again from scratch - although it's debatable what measures and competencies they have within their IT team but nevertheless, it happened) and our own backups is a non-negotiable for me.

[smarties11]

SOLVED IT! Checked the event logs to see the effect of deleting the old certificate, and saw an error from the transport service saying it couldn't find the old certificate, it would use the new one in the meantime but that I should restore the old from backup. So deleting the old certificate forced the required behaviour, but there was still an issue. Happened to see an event log message for our Anti Virus and that made me wonder.....and yes, our AV (McAfee Endpoint security) was preventing Exchange from changing the security on a private key. Good thing I suppose? Temporarily disabled the AV and then ran the PowerShell command to re-assign the services on the new cert and it works. I correctly got the prompt asking me whether I wanted to replace the old certificate with the new and now the errors in the event log have disappeared.

 

Hope this helps someone in the future.

[3s-gtech]

Antivirus software - making our jobs more difficult 95% of the time for the 5% where it saves our backsides!