+ Post New Thread
Page 1 of 10 12345 ... LastLast
Results 1 to 15 of 150
Birmingham Grid for Learning (BGfL) Thread, How to black out most of Birmingham....Let Link2ICT update your firewall in Regional Broadband Consortiums (RBC); Well done Link2ICT on completing your successful firewall upgrade (Their quote not mine :-S)... you managed to knock off the ...
  1. #1
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    brum
    Posts
    734
    Thank Post
    156
    Thanked 85 Times in 77 Posts
    Rep Power
    48

    How to black out most of Birmingham....Let Link2ICT update your firewall

    Well done Link2ICT on completing your successful firewall upgrade (Their quote not mine :-S)... you managed to knock off the whole city in the process.

    Seriously annoyed at their level of incompetence and unprofessional behaviour

    Who in their right mind would deploy a full city wide upgrade that hasn't been tested fully....

    One more reason to leave Link2ICT

  2. #2

    Little-Miss's Avatar
    Join Date
    Oct 2007
    Location
    London
    Posts
    5,569
    Thank Post
    2,388
    Thanked 754 Times in 462 Posts
    Blog Entries
    2
    Rep Power
    544
    We just ground to a halt, didnt lose everything. I rang and they said yes, we are aware of the issue....so i guessed it was the firewall...

  3. #3
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    brum
    Posts
    734
    Thank Post
    156
    Thanked 85 Times in 77 Posts
    Rep Power
    48
    Subject: F.A.O Head Teacher/School Technician/Network Manager - IMPORTANT: Firewall Upgrade Update
    To:



    Dear Colleagues,

    Link2ICT is pleased to announce successful completion of our Firewall
    Upgrade.

    We aware of the slow performance that you may have encountered this morning
    and have now found a solution which has enabled us to resume service.
    Please rest assured we are proactively monitoring the network to ensure
    there are no further performance issues and apologise for the inconvenience
    this may have caused.

    Benefits of the new Firewall:

    The new firewall will offer additional features including:

    The future ability to provision access to the Internet for variety of
    devices, such as Kindles, Android devices and some iPad Apps.
    Increased application control, allowing access to cloud based
    applications such as Office 365/Office 365 Hybrid Mode and Google
    Apps
    Advanced security features which will help identify suspicious
    traffic such as Malware, prevent attacks on websites, and offer
    Distributed Denial of Service (DDoS) protection

    The following improvements will also be evident:

    The maximum Internet traffic throughput will increase to 10Gb/s
    High quality logging of all Internet traffic, which will assist with
    investigations and troubleshooting
    Availability of enhanced reporting facilities
    Improved security by the automatic highlighting of any unused
    firewall rules
    Inbuilt protection against low level internet attacks

    This is the beginning of a number of investments and value added services
    that Link2ICT intends to offer customers, ultimately improving and
    enhancing the Link2ICT Network and Internet Service.


    Sent two hours after they caused chaos!

  4. #4

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    602
    Thank Post
    92
    Thanked 73 Times in 65 Posts
    Rep Power
    24
    To be fair to Link2ICT it wasn't the firewall that actually caused the issue. It was part of a filtering process that for some reason a service on that failed and this then caused the very slow internet access for a couple of hours.

    How do you stress test a change like this?

  5. #5
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    brum
    Posts
    734
    Thank Post
    156
    Thanked 85 Times in 77 Posts
    Rep Power
    48
    Easy option is to at least batch test 50 schools first and not roll out the whole thing in one go... They used to do this with CMIS, PCE and SIMS, seem to have forgotten to test things before deploying them and communication seems to have failed as nobody could get through to Service Desk via phone or email.

    As for what's failed I've spoken to four different service desk bods and all four have given me different answers so not sure who to trust one thing is clear somebody really messed up and then they sent that email to say it was successfully implemented which kind of rubs salt in the wound.

    That and the fact that it ruined our christmas play has really annoyed me.
    Not impressed today

  6. #6

    plexer's Avatar
    Join Date
    Dec 2005
    Location
    Norfolk
    Posts
    13,713
    Thank Post
    667
    Thanked 1,636 Times in 1,462 Posts
    Rep Power
    424
    What was their backout plan when they realised the issue though?

    Ben

  7. #7
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    brum
    Posts
    734
    Thank Post
    156
    Thanked 85 Times in 77 Posts
    Rep Power
    48
    it looks like they rolled it back after three hours of attempting to get it to work.... not sure if they really had a plan, or why they decided to do it during school hours but i'm not part of their brain trust i just have to pick up the pieces like most of my colleagues when they drop us in it.

    just had an email to say further updates will be carried out tonight at 9:30pm, fingers crossed for tomorrow!

  8. #8

    Michael's Avatar
    Join Date
    Dec 2005
    Location
    Birmingham
    Posts
    9,262
    Thank Post
    242
    Thanked 1,572 Times in 1,252 Posts
    Rep Power
    340
    I have to agree, I wouldn't exactly describe the firewall migration as success, but more disastrous. I think lessons have been learnt today for sure.

    Why it needed to be done all in one weekend is unclear. I think batches of 50 or 100 would have been more appropriate, unless there was a good reason it needed to be done in one go.

    I would have much preferred if it was delayed until February half-term, without Christmas round the corner adding to the pressure to get all the work completed.

  9. #9

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    602
    Thank Post
    92
    Thanked 73 Times in 65 Posts
    Rep Power
    24
    It wasn't a back out plan. They checked and found a problem service which was fixed by restarting it.

  10. #10

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    602
    Thank Post
    92
    Thanked 73 Times in 65 Posts
    Rep Power
    24
    Quote Originally Posted by Michael View Post
    I have to agree, I wouldn't exactly describe the firewall migration as success, but more disastrous. I think lessons have been learnt today for sure.

    Why it needed to be done all in one weekend is unclear. I think batches of 50 or 100 would have been more appropriate, unless there was a good reason it needed to be done in one go.

    I would have much preferred if it was delayed until February half-term, without Christmas round the corner adding to the pressure to get all the work completed.
    It should have been done over a half term when there is much less traffic and problems could have been resolved much easier.

  11. #11

    Michael's Avatar
    Join Date
    Dec 2005
    Location
    Birmingham
    Posts
    9,262
    Thank Post
    242
    Thanked 1,572 Times in 1,252 Posts
    Rep Power
    340
    Quote Originally Posted by Tricky_Dicky View Post
    To be fair to Link2ICT it wasn't the firewall that actually caused the issue. It was part of a filtering process that for some reason a service on that failed and this then caused the very slow internet access for a couple of hours.

    How do you stress test a change like this?
    I don't think a stress test was necessary, however the fact remains, if it was done in batches of 50 or 100 - the problem would (without doubt) have been resolved faster. I think everyone just presumed it was the new firewall (and rightly so), making it more difficult to isolate to a filtering type issue. As a result what probably took minutes to fix took hours to diagnose and isolate the issue.

  12. #12
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    brum
    Posts
    734
    Thank Post
    156
    Thanked 85 Times in 77 Posts
    Rep Power
    48
    [QUOTE=Tricky_Dicky;1102646]It wasn't a back out plan. They checked and found a problem service which was fixed by restarting it.[/

    Yep but why wasn't this tested first and spotted first, it was planned and implemented poorly which resulted in a failure of service.

    look at it this way 600 - 550 schools approximately use the BGFL for internet and they had no internet provision in school this morning because they messed up. We pay them over the odds for internet services, sims and cmis and support desk and the service is shocking. If a large organisation / company had a failure like this it wouldn't be accepted and I really hope schools and NM's get on the phone to Link2ICT and start complaining.

    And here comes the fix of the fix!

    "Date: Monday 16th December 2013

    Duration: From 21:30 tonight

    Affected Users: All BGfL Users

    Further to our earlier communication, essential updates to the new firewall
    will be made tonight at 9.30pm. There will be no impact to service during
    this time."

    ah well only four more sleeps ;-)

  13. #13

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    602
    Thank Post
    92
    Thanked 73 Times in 65 Posts
    Rep Power
    24
    I honestly don't think it should have been dragged out like that. Maybe a test of 20 of the more forward thinking/challenging schools would have been good and then everyone else over a weekend.
    Apart from a 2 hour reduction in speed what else has gone wrong?

    How did it ruin the school play?

  14. #14

    CHR1S's Avatar
    Join Date
    Feb 2006
    Location
    Birmingham
    Posts
    4,505
    Thank Post
    1,585
    Thanked 486 Times in 304 Posts
    Rep Power
    217
    I won't go into it but the mayhem this "almost" caused today!!

    Thankfully tethering and mobile data saved the day!

    Edit - there was no access for about two hours, 504 errors and slower than 56k speeds!

  15. #15
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    brum
    Posts
    734
    Thank Post
    156
    Thanked 85 Times in 77 Posts
    Rep Power
    48
    Quote Originally Posted by Tricky_Dicky View Post
    I honestly don't think it should have been dragged out like that. Maybe a test of 20 of the more forward thinking/challenging schools would have been good and then everyone else over a weekend.
    Apart from a 2 hour reduction in speed what else has gone wrong?

    How did it ruin the school play?
    No access to vimeo channel where the class teacher had uploaded video clips and no back up copy, ended up having to do a voice over with a microphone. just spoilt it as the kids had worked really hard to produce these videos and we couldn't show them led to several crying children....

    Lost connection to our hosted admin and finance servers so office couldn't do their work.

    Two parents workshop cancelled as no internet access meant we couldn't use resources.

    Planned video conference with a large charity and two other schools in our cohort cancelled as we couldn't get online, not good when your supposed to be showcasing IT.

    No email access for office, SLT or teachers (some thought this was great, others had planning and documents they needed urgently.) Staff never seem to understand that it isn't my fault despite explaining that the whole city has issues.

    Our wasn't a reduction in speed we had 3hrs of nothing still intermittently dropping out and seems to be affecting a few schools local to us so maybe an exchange issue as well.

SHARE:
+ Post New Thread
Page 1 of 10 12345 ... LastLast

Similar Threads

  1. How to find out which machine a student logged onto, & when
    By indiegirl in forum How do you do....it?
    Replies: 32
    Last Post: 16th March 2012, 01:17 PM
  2. How to work out domain string (LDAP I think)
    By garethedmondson in forum Windows
    Replies: 16
    Last Post: 9th September 2009, 11:15 PM
  3. How to find out who deleted what
    By Chrish5 in forum Windows
    Replies: 9
    Last Post: 17th October 2008, 10:37 AM
  4. How to structure the layout of the Kaleidos Portal Intrest Groups
    By AM_LHS in forum Virtual Learning Platforms
    Replies: 1
    Last Post: 6th June 2008, 07:32 PM
  5. How to find out a preferred bidder
    By MK-2 in forum BSF
    Replies: 26
    Last Post: 8th January 2008, 09:56 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •