Information concerning last Weeks Outage
Last Weeks Outage
Last Tuesday there was a problem with one of the firewalls that was initially attributed to a memory leak and resulted in a loss of connectivity to schools. On Friday we noticed more issues with the firewalls which had to be restarted whilst patches to address the memory leaks were applied which led to the failure of the firewall.
There has been an intermittent issue which has occurred approximately every 3 months since the implementation of the Cisco ASA LGfL firewalls relating to high memory usage and service outages. These issues were raised with the external consultant engineer who converted the original Checkpoint configuration to the ASA command set. ICT services were given assurances by the external engineer that the configuration would not cause any issues and that the reported high memory issues where a known issue with the ASA. ICT services have been liaising with Cisco who have been providing fixes and new software releases to address the intermittent problems and ICT Services focus on detection of the fault to restore the service to the operational state as soon as possible. However, during the recent event of Friday 10th June, the service could not be returned to a stable operating state and ICT services engineers had no choice but to undertake a direct extensive investigation which proved to locate the root cause of the issue. Over the weekend engineers from LICTS reconfigured the firewall as the issue was identified as a configuration issue which was causing the memory leaks despite the original assurances of the external engineers and numerous attempts by Cisco to address the issue. The firewall service has now been running for a number of days with no major issues and the memory utilization has been drastically reduced, resulting in improved throughput and greater stability.
On Wednesday some schools may have noticed some disruption as an additional proxy server was added into the network to improve performance following the introduction of the new filtering. Whilst not affecting all internet connections as other proxy servers were still working, it did affect some schools, who, were presented with a logon screen for the new filtering solution.
We will be reviewing the issues that we saw last week to allow us to minimize the risk of it happening again. The LGfL server farm is designed to work in resilient way, in this case the failure of the firewall affected both firewalls simultaneously.
We apologise for the major inconvenience that the outages last week caused .