Last Weeks Outage
Last Tuesday there was a problem with one of the firewalls that was initially attributed to a memory leak and resulted in a loss of connectivity to schools. On Friday we noticed more issues with the firewalls which had to be restarted whilst patches to address the memory leaks were applied which led to the failure of the firewall.
There has been an intermittent issue which has occurred approximately every 3 months since the implementation of the Cisco ASA LGfL firewalls relating to high memory usage and service outages. These issues were raised with the external consultant engineer who converted the original Checkpoint configuration to the ASA command set. ICT services were given assurances by the external engineer that the configuration would not cause any issues and that the reported high memory issues where a known issue with the ASA. ICT services have been liaising with Cisco who have been providing fixes and new software releases to address the intermittent problems and ICT Services focus on detection of the fault to restore the service to the operational state as soon as possible. However, during the recent event of Friday 10th June, the service could not be returned to a stable operating state and ICT services engineers had no choice but to undertake a direct extensive investigation which proved to locate the root cause of the issue. Over the weekend engineers from LICTS reconfigured the firewall as the issue was identified as a configuration issue which was causing the memory leaks despite the original assurances of the external engineers and numerous attempts by Cisco to address the issue. The firewall service has now been running for a number of days with no major issues and the memory utilization has been drastically reduced, resulting in improved throughput and greater stability.
On Wednesday some schools may have noticed some disruption as an additional proxy server was added into the network to improve performance following the introduction of the new filtering. Whilst not affecting all internet connections as other proxy servers were still working, it did affect some schools, who, were presented with a logon screen for the new filtering solution.
We will be reviewing the issues that we saw last week to allow us to minimize the risk of it happening again. The LGfL server farm is designed to work in resilient way, in this case the failure of the firewall affected both firewalls simultaneously.
We apologise for the major inconvenience that the outages last week caused .
Thanks for the info Martin
The following is not directed to you but at the person responsible for our Internet access - please print it out and stick it on their desk!
As far as end users (teachers/pupils in schools) are concerned, Internet access in schools across Lancashire has been severely degraded since we returned to school after half-term. (That's 7 consecutive working days now - approx 3% of total school year)
Despite assurances that the above problems have been rectified, the continuing slowness of our internet access(to the point where on lots of sites browsers just sit there doing nothing) has severely dented any belief that action is being taken to restore the level of service that we used to receive before the half term break.
Internet access in schools is not a nice to have - losing it nowadays is pretty much the same as finding you don't have any dry wipe markers in the school and having to go back to using blackboards!
School lessons cannot be rescheduled - we run to timetables where resources have to be available for the time slots required.
And on a simple information passing note (which can go a long way to defusing resentment and anger) why can not planned work (like introducing a new proxy server) be told to us in advance?
Why can we not be told that its going to happen to try to alleviate problems caused by new filtering?
Last edited by SimpleSi; 16th June 2010 at 06:40 AM.
Just an FYI. If your ASAs aren't on 8.31 there's a chance (when you upgrade to it) that'll it'll lose your config. Someone on another forum got bitten by that recently - known issue according to Cisco.
There are currently 1 users browsing this thread. (0 members and 1 guests)