+ Post New Thread
Results 1 to 3 of 3
CLEO Thread, Information concerning last Weeks Outage in Regional Broadband Consortiums (RBC); Last Weeks Outage Last Tuesday there was a problem with one of the firewalls that was initially attributed to a ...
  1. #1

    Join Date
    Jan 2009
    Posts
    17
    Thank Post
    0
    Thanked 10 Times in 5 Posts
    Rep Power
    13

    Information concerning last Weeks Outage

    Last Weeks Outage



    Last Tuesday there was a problem with one of the firewalls that was initially attributed to a memory leak and resulted in a loss of connectivity to schools. On Friday we noticed more issues with the firewalls which had to be restarted whilst patches to address the memory leaks were applied which led to the failure of the firewall.

    There has been an intermittent issue which has occurred approximately every 3 months since the implementation of the Cisco ASA LGfL firewalls relating to high memory usage and service outages. These issues were raised with the external consultant engineer who converted the original Checkpoint configuration to the ASA command set. ICT services were given assurances by the external engineer that the configuration would not cause any issues and that the reported high memory issues where a known issue with the ASA. ICT services have been liaising with Cisco who have been providing fixes and new software releases to address the intermittent problems and ICT Services focus on detection of the fault to restore the service to the operational state as soon as possible. However, during the recent event of Friday 10th June, the service could not be returned to a stable operating state and ICT services engineers had no choice but to undertake a direct extensive investigation which proved to locate the root cause of the issue. Over the weekend engineers from LICTS reconfigured the firewall as the issue was identified as a configuration issue which was causing the memory leaks despite the original assurances of the external engineers and numerous attempts by Cisco to address the issue. The firewall service has now been running for a number of days with no major issues and the memory utilization has been drastically reduced, resulting in improved throughput and greater stability.

    On Wednesday some schools may have noticed some disruption as an additional proxy server was added into the network to improve performance following the introduction of the new filtering. Whilst not affecting all internet connections as other proxy servers were still working, it did affect some schools, who, were presented with a logon screen for the new filtering solution.

    We will be reviewing the issues that we saw last week to allow us to minimize the risk of it happening again. The LGfL server farm is designed to work in resilient way, in this case the failure of the firewall affected both firewalls simultaneously.

    We apologise for the major inconvenience that the outages last week caused .

    Thanks


    Martin

  2. 2 Thanks to schoolsupport:

    buzzard (28th June 2010), SimpleSi (16th June 2010)

  3. #2

    SimpleSi's Avatar
    Join Date
    Jun 2005
    Location
    Lancashire
    Posts
    5,824
    Thank Post
    1,476
    Thanked 593 Times in 445 Posts
    Rep Power
    168
    Thanks for the info Martin

    The following is not directed to you but at the person responsible for our Internet access - please print it out and stick it on their desk!

    As far as end users (teachers/pupils in schools) are concerned, Internet access in schools across Lancashire has been severely degraded since we returned to school after half-term. (That's 7 consecutive working days now - approx 3% of total school year)

    Despite assurances that the above problems have been rectified, the continuing slowness of our internet access(to the point where on lots of sites browsers just sit there doing nothing) has severely dented any belief that action is being taken to restore the level of service that we used to receive before the half term break.

    Internet access in schools is not a nice to have - losing it nowadays is pretty much the same as finding you don't have any dry wipe markers in the school and having to go back to using blackboards!

    School lessons cannot be rescheduled - we run to timetables where resources have to be available for the time slots required.

    And on a simple information passing note (which can go a long way to defusing resentment and anger) why can not planned work (like introducing a new proxy server) be told to us in advance?

    Why can we not be told that its going to happen to try to alleviate problems caused by new filtering?



    Simon
    Last edited by SimpleSi; 16th June 2010 at 06:40 AM.

  4. #3


    Join Date
    Dec 2005
    Location
    In the server room, with the lead pipe.
    Posts
    4,660
    Thank Post
    276
    Thanked 780 Times in 607 Posts
    Rep Power
    224
    Just an FYI. If your ASAs aren't on 8.31 there's a chance (when you upgrade to it) that'll it'll lose your config. Someone on another forum got bitten by that recently - known issue according to Cisco.

SHARE:
+ Post New Thread

Similar Threads

  1. LGFL outage
    By Face-Man in forum London Grid for Learning (LGfL)
    Replies: 0
    Last Post: 17th September 2009, 05:01 PM
  2. [News] An Unlikely Network Outage
    By mattx in forum Jokes/Interweb Things
    Replies: 0
    Last Post: 22nd April 2009, 01:51 PM
  3. Another day - another outage
    By SimpleSi in forum CLEO
    Replies: 4
    Last Post: 25th February 2009, 10:52 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •