Suddenly getting certificate errors accessing webmail via smoothwall

[Linfit]

We have an on-site mail server (running MDaemon) which has a letsencrypt cert to allow https web access. It has two months to run.

 

Come in this morning to find machines which use the smoothwall as a proxy are getting an error on attempting the https connection which amounts to the cert is not trusted by your computer.

Access to this from outside the building or from machines not going through our smoothwall box is fine.

 

This is a big problem as most staff obviously are going through the smoothwall box and so can't access their e-mail.

 

If I switch MDaemon to accept http as well as https they can connect on http but obviously thats a reduction in security and not a long term fix.

 

Was all working OK yesterday but nothing obvious has changed overnight.

 

Is this likely something to do with the certificate chain ?

Has something changed with LetsEncrypt certs ?

 

This ares is not my strongpoint so any guidance on where to start looking would be appreciated.

[psydii]

Two intermediate certificates have expired in quick succession.

 

Causing some knock on effects.

 

https://twitter.com/gossithedog/status/1443348554131644420?s=21

Edited September 30, 2021 by psydii

[ibpalle]

Also https://kb.smoothwall.com/hc/en-us/articles/4407518085394

[Linfit]

Thanks, not seen that.

 

I have followed the instructions and installed the Lets Encrypt R3 intermediate cert.

The ISRG RootX1 cert was already installed, which I thought covered the Lets Encrypt certs.

Anyway, that now added, but doesn't seem to have fixed the issue.

 

Is there any logging on the smoothwall box that can show me exactly whats happening on this anywhere ?

 

 

UPDATE: I have looked in the certificate store on the mail server and found R3 DST Root CA X3 intermediate cert which expired yesterday. I am wondering if this is the cause of the issue ?

I have added in the same intermediate cert as I added to the smoothwall box - R3 ISRG Root CA X3 but again, this has not fixed the issue.

 

The DST Root CA x3 cert showing in Root CA's on the server does indeed expire today, but there is an ISRG Root X1

[jadzea]

We've got the same issue and have applied the certificate as described in https://kb.smoothwall.com/hc/en-us/a.../4407518085394 - we've even rebooted the appliance and tried again. Still not working.

 

Anyone have any other ideas?? @ibpalle - do you have any further suggestions?

 

Thanks

[coop1984]

Also https://kb.smoothwall.com/hc/en-us/articles/4407518085394

 

Importing the cert as per this article has not worked for us

[SpaceInvader83]

Bypassing ssl intercept for the site in question should fix it up (I think) as a quick work around

[Jonah]

As well as the intermediate R3 cert from the SW article, we've also added the X1 self signed root from https://censys.io/certificates/96bcec06264976f37460779acf28c5a7cfe8a3c0aae11a8ffcee05c0bddf08c6 and then removed the expired CA cert from the SW. That seemed to sort things for us.

[jadzea]

Ok got it fixed too!

 

So steps were:

 

1. Followed the guidance in Smoothwall's KB article (https://kb.smoothwall.com/hc/en-us/articles/4407518085394)

2. Removed the expired DST Root CA X3 certificate

3. Made sure we had the ISRG Root X1's Internet Security Research Group certificate installed (https://letsencrypt.org/certs/isrgrootx1.pem)

3. Waited a couple of minutes and everything started to work again!

 

This article gives some background as to what was going on https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

I think simply having the expired certificate in place still meant smoothwall would use this to validate certificates rather than the new ISRG Root X1 certificate.

 

:-)

Edited September 30, 2021 by jadzea

[ibpalle]

Yes, the KB was missing these steps:

 

1: In Certificates - Certificate authorities, find all the expired certs and click the tickbox to the right hand side to select them and then use the delete button at the bottom. The one CA that's specific to this case is the 'DST Root CA X3'

 

2: Clear guardian cert cache under Guardian » HTTPS inspection » Settings

Edited October 1, 2021 by ibpalle

[Linfit]

Hmm. Deleted the old cert, restarted, still not working.

Rejecting anything thats using a LetsEncrypt cert.

[jadzea]

Have you tried installing this (3. Made sure we had the ISRG Root X1's Internet Security Research Group certificate installed (https://letsencrypt.org/certs/isrgrootx1.pem))?

 

Also have you cleared the guardian certificate cache?

[Linfit]

Yep, have that installed, cleared the cache, restarted web proxy with cleared cache, still not having it.

Have to raise a ticket with them - can't do that without a logon, don't have a logon apparently.

Oh, fun days......

[flyinghaggis]

We're seeing similar issues using our Fortinet/Fortigate Firewall with anything using a Let's Encrypt certificate being blocked. We've contacted Fortinet but so far their only suggestion has been to allow expired certificates which we're apprehensive about doing due to the security risks.

 

Fortinet claim they're working on a fix/update but we haven't heard anything solid so far.

[Chuckster]

I'm glad I stumbled across this thread. Looking at Smoothwall's KB I was able to obtain the 'lets-encrypt-r3.pem' and import it into our 2 seperate firewalls - SonicWall and Sophos XG.

 

All is now well and everybody is happy.

 

Thank you Smoothwall!

[ibpalle]

We have had a few customers reporting that the KB did not solve their issue but when we looked into it, the errors were now from other devices like upstream firewalls, so if you are not using the Smoothwall as your firewall, make sure to update the certificates on other devices too.

[1an]

I followed the Smoothwall KB. It resolved the issue for site https://www.s4snextgen.org/ however still get the certificate error for https://accessitsoftware.com/updates

[jadzea]

HI @1an I think this is an issue with https://accessitsoftware.com/updates

 

Quick go on https://www.digicert.com/help/ and putting in the URL shows they have an issue with their website certificates.

[ibpalle]

Tried that as well without going via the Smoothwall HTTPS filter and browser is still giving cert errors. This works fine though https://accessitlibrary.com/

 

Also got a permission denied when accessing https://accessitsoftware.com/updates.

[Linfit]

Still not got this sorted.

Still getting "This site is not secure" messages even with both certs on the Smoothie box.

Odd thing though - Our machines almost all cannot get out onto the internet directly - we force the Smoothie as the web proxy for web browsers using GPO and the firewall generally only allows traffic out from the Smoothwall box.

If I put a firewall exception for outgoing traffic in for a specific machine, the issue goes away, even though the smoothie is still set as the proxy on that machine.

Our firewall is an old ASA that is just controlling what type of traffic can come in and go out and on which ports, no filtering or inspection, no certs installed.

This isn't an area I know much about, so suggestions welcome.

It is with Smoothwall second line support now but they don't seem convinced its a smoothwall problem and I'm not sure.

[TechMonkey]

I followed the Smoothwall KB. It resolved the issue for site https://www.s4snextgen.org/ however still get the certificate error for https://accessitsoftware.com/updates

 

It is because their certificate is only for Page Redirection, so accessitsoftware.com gives an invalid name error

[kmount]

Still not got this sorted.

Still getting "This site is not secure" messages even with both certs on the Smoothie box.

Odd thing though - Our machines almost all cannot get out onto the internet directly - we force the Smoothie as the web proxy for web browsers using GPO and the firewall generally only allows traffic out from the Smoothwall box.

If I put a firewall exception for outgoing traffic in for a specific machine, the issue goes away, even though the smoothie is still set as the proxy on that machine.

Our firewall is an old ASA that is just controlling what type of traffic can come in and go out and on which ports, no filtering or inspection, no certs installed.

This isn't an area I know much about, so suggestions welcome.

It is with Smoothwall second line support now but they don't seem convinced its a smoothwall problem and I'm not sure.

 

 

Definitely is not an issue with Smoothwall.

 

It is an issue with the cert on the site as I see the error even from home with no Smoothwall here.

[timbo343]

okay, this happened to us, and i feel so stupid.

 

We have been blocking windowsupdate.com for everyone because we didnt want machines either on the network or those brought in on a BYOD network to be pulling down updates - turns out (highlighted by Smoothwall support) that the certificates come from ctldl.windowsupdate.com. After allowing ctldl.windowsupdate.com the auth bypass list and moving the windowsupdate policy below the auth bypass, we no longer get the certificate issues!

[tom_newton]

okay, this happened to us, and i feel so stupid.

 

We have been blocking windowsupdate.com for everyone because we didnt want machines either on the network or those brought in on a BYOD network to be pulling down updates - turns out (highlighted by Smoothwall support) that the certificates come from ctldl.windowsupdate.com. After allowing ctldl.windowsupdate.com the auth bypass list and moving the windowsupdate policy below the auth bypass, we no longer get the certificate issues!

 

Definitely don't feel stupid. That's a most non-obvious cause. I've been doing this YEARS and I wouldn't have got there much quicker if at all

[FN-GM]

okay, this happened to us, and i feel so stupid.

 

We have been blocking windowsupdate.com for everyone because we didnt want machines either on the network or those brought in on a BYOD network to be pulling down updates - turns out (highlighted by Smoothwall support) that the certificates come from ctldl.windowsupdate.com. After allowing ctldl.windowsupdate.com the auth bypass list and moving the windowsupdate policy below the auth bypass, we no longer get the certificate issues!

 

Oh wow! I guess why you would want to block it.

 

Maybe you could limit the bandwidth as an alternative.