On Friday evening, Updata implemented their emergency change. Previously there was a component called ‘Squid’ running on the proxies. The proxies are the servers that deal with the internet traffic. Squid has been replaced with something called ‘Authent’. Squid was causing major bottle-necks on the network and was slowing down the performance of the web browsing platform, particularly during a fail over scenario when all the traffic was going out to the internet through the one data centre. Authent is a modern HTTP/S proxy component which is compatible with Netsweeper (our filtering system that runs on the proxies) and offers significant performance advantages as well as new functionality. After Friday’s change, all appears stable and there is significant improvement in terms of CPU utilisation on the proxies. All is pointing towards the fact that the network can once again perform to the required standard, particularly when going through the one data centre. We won’t know for definite until we find ourselves in that scenario though.
Updata wish to monitor the network and are optimistic that as the Netsweeper platform is handling traffic in a more sophisticated way. There is hope that this may improve the ongoing issue of the firewall modules that keep failing.
In two weeks time the situation will be reviewed and assuming we haven’t had any further firewall failures, two things will happen:
1) Updata will intentionally fail over to one data centre one evening and we will monitor performance throughout the following day. If there are any concerns with performance, Updata will promptly manually fail back to both data centres.
2) Updata will then arrange a change being implemented whereby if the firewall module fails, like has been happening recently, then traffic will fail over to the other data centre automatically, keeping disruption to a minimum.
However, if within the next two weeks, the firewall module fails, this will confirm that the firewall module problem is still apparent, despite Friday’s change. Updata will then obviously be able to test the reliability of the service going through the one data centre and won’t need to intentionally fail over as advised above. Updata will also arrange the automatic fail over being implemented ASAP.