WordPress and IONOS

[lmrogers]

So I'm hoping someone can point me in the right direction.

 

We have our website hosted internally - WordPress website hosted on IIS. This is published through Azure Web App Proxy to the outside world with a DNS CNAME record in IONOS where our domain name is registered.

 

I have added a robots.txt file in the root of the website to allow Google to index the site and improve the look of the Google search results.

 

This however is not working and I just get a default robots.txt which disallows all user agents from crawling the website.

 

Currently I do not know where it is picking up this robots.txt with the disallow rule.

 

Any pointers would be appreciated.

 

Thanks

[lmrogers]

So I have found the problem, it turns out that Azure AD web app proxy serves its own robots.txt which disallows all crawling and this is something that cannot be changed.

[MicrodigitUK]

We had the same issue and after talking to Microsoft it is by design because Azure Application Proxy is designed for internal web apps but not for public facing websites, so will always block robots.txt. You have 3 options move hosting of site external, publish the web server in DMZ or onsite WAP in server 2016/19 but WAP requires ADFS infrastructure. There may be some 3rd party reverse proxy software that can achieve the same results as WAP via a firewall like Smoothwall.