+ Post New Thread
Page 1 of 10 12345 ... LastLast
Results 1 to 15 of 150
Birmingham Grid for Learning (BGfL) Thread, How to black out most of Birmingham....Let Link2ICT update your firewall in Regional Broadband Consortiums (RBC); Well done Link2ICT on completing your successful firewall upgrade (Their quote not mine :-S)... you managed to knock off the ...
  1. #1
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    Somewhere near you
    Posts
    843
    Thank Post
    210
    Thanked 136 Times in 94 Posts
    Rep Power
    60

    How to black out most of Birmingham....Let Link2ICT update your firewall

    Well done Link2ICT on completing your successful firewall upgrade (Their quote not mine :-S)... you managed to knock off the whole city in the process.

    Seriously annoyed at their level of incompetence and unprofessional behaviour

    Who in their right mind would deploy a full city wide upgrade that hasn't been tested fully....

    One more reason to leave Link2ICT

  2. #2

    Little-Miss's Avatar
    Join Date
    Oct 2007
    Location
    London
    Posts
    5,558
    Thank Post
    2,416
    Thanked 800 Times in 481 Posts
    Blog Entries
    2
    Rep Power
    552
    We just ground to a halt, didnt lose everything. I rang and they said yes, we are aware of the issue....so i guessed it was the firewall...

  3. #3
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    Somewhere near you
    Posts
    843
    Thank Post
    210
    Thanked 136 Times in 94 Posts
    Rep Power
    60
    Subject: F.A.O Head Teacher/School Technician/Network Manager - IMPORTANT: Firewall Upgrade Update
    To:



    Dear Colleagues,

    Link2ICT is pleased to announce successful completion of our Firewall
    Upgrade.

    We aware of the slow performance that you may have encountered this morning
    and have now found a solution which has enabled us to resume service.
    Please rest assured we are proactively monitoring the network to ensure
    there are no further performance issues and apologise for the inconvenience
    this may have caused.

    Benefits of the new Firewall:

    The new firewall will offer additional features including:

    The future ability to provision access to the Internet for variety of
    devices, such as Kindles, Android devices and some iPad Apps.
    Increased application control, allowing access to cloud based
    applications such as Office 365/Office 365 Hybrid Mode and Google
    Apps
    Advanced security features which will help identify suspicious
    traffic such as Malware, prevent attacks on websites, and offer
    Distributed Denial of Service (DDoS) protection

    The following improvements will also be evident:

    The maximum Internet traffic throughput will increase to 10Gb/s
    High quality logging of all Internet traffic, which will assist with
    investigations and troubleshooting
    Availability of enhanced reporting facilities
    Improved security by the automatic highlighting of any unused
    firewall rules
    Inbuilt protection against low level internet attacks

    This is the beginning of a number of investments and value added services
    that Link2ICT intends to offer customers, ultimately improving and
    enhancing the Link2ICT Network and Internet Service.


    Sent two hours after they caused chaos!

  4. #4

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    616
    Thank Post
    93
    Thanked 75 Times in 67 Posts
    Rep Power
    26
    To be fair to Link2ICT it wasn't the firewall that actually caused the issue. It was part of a filtering process that for some reason a service on that failed and this then caused the very slow internet access for a couple of hours.

    How do you stress test a change like this?

  5. #5
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    Somewhere near you
    Posts
    843
    Thank Post
    210
    Thanked 136 Times in 94 Posts
    Rep Power
    60
    Easy option is to at least batch test 50 schools first and not roll out the whole thing in one go... They used to do this with CMIS, PCE and SIMS, seem to have forgotten to test things before deploying them and communication seems to have failed as nobody could get through to Service Desk via phone or email.

    As for what's failed I've spoken to four different service desk bods and all four have given me different answers so not sure who to trust one thing is clear somebody really messed up and then they sent that email to say it was successfully implemented which kind of rubs salt in the wound.

    That and the fact that it ruined our christmas play has really annoyed me.
    Not impressed today

  6. #6

    plexer's Avatar
    Join Date
    Dec 2005
    Location
    Norfolk
    Posts
    13,689
    Thank Post
    756
    Thanked 1,715 Times in 1,526 Posts
    Rep Power
    438
    What was their backout plan when they realised the issue though?

    Ben

  7. #7
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    Somewhere near you
    Posts
    843
    Thank Post
    210
    Thanked 136 Times in 94 Posts
    Rep Power
    60
    it looks like they rolled it back after three hours of attempting to get it to work.... not sure if they really had a plan, or why they decided to do it during school hours but i'm not part of their brain trust i just have to pick up the pieces like most of my colleagues when they drop us in it.

    just had an email to say further updates will be carried out tonight at 9:30pm, fingers crossed for tomorrow!

  8. #8

    Michael's Avatar
    Join Date
    Dec 2005
    Location
    Birmingham
    Posts
    9,345
    Thank Post
    242
    Thanked 1,602 Times in 1,278 Posts
    Rep Power
    346
    I have to agree, I wouldn't exactly describe the firewall migration as success, but more disastrous. I think lessons have been learnt today for sure.

    Why it needed to be done all in one weekend is unclear. I think batches of 50 or 100 would have been more appropriate, unless there was a good reason it needed to be done in one go.

    I would have much preferred if it was delayed until February half-term, without Christmas round the corner adding to the pressure to get all the work completed.

  9. #9

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    616
    Thank Post
    93
    Thanked 75 Times in 67 Posts
    Rep Power
    26
    It wasn't a back out plan. They checked and found a problem service which was fixed by restarting it.

  10. #10

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    616
    Thank Post
    93
    Thanked 75 Times in 67 Posts
    Rep Power
    26
    Quote Originally Posted by Michael View Post
    I have to agree, I wouldn't exactly describe the firewall migration as success, but more disastrous. I think lessons have been learnt today for sure.

    Why it needed to be done all in one weekend is unclear. I think batches of 50 or 100 would have been more appropriate, unless there was a good reason it needed to be done in one go.

    I would have much preferred if it was delayed until February half-term, without Christmas round the corner adding to the pressure to get all the work completed.
    It should have been done over a half term when there is much less traffic and problems could have been resolved much easier.

  11. #11

    Michael's Avatar
    Join Date
    Dec 2005
    Location
    Birmingham
    Posts
    9,345
    Thank Post
    242
    Thanked 1,602 Times in 1,278 Posts
    Rep Power
    346
    Quote Originally Posted by Tricky_Dicky View Post
    To be fair to Link2ICT it wasn't the firewall that actually caused the issue. It was part of a filtering process that for some reason a service on that failed and this then caused the very slow internet access for a couple of hours.

    How do you stress test a change like this?
    I don't think a stress test was necessary, however the fact remains, if it was done in batches of 50 or 100 - the problem would (without doubt) have been resolved faster. I think everyone just presumed it was the new firewall (and rightly so), making it more difficult to isolate to a filtering type issue. As a result what probably took minutes to fix took hours to diagnose and isolate the issue.

  12. #12
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    Somewhere near you
    Posts
    843
    Thank Post
    210
    Thanked 136 Times in 94 Posts
    Rep Power
    60
    [QUOTE=Tricky_Dicky;1102646]It wasn't a back out plan. They checked and found a problem service which was fixed by restarting it.[/

    Yep but why wasn't this tested first and spotted first, it was planned and implemented poorly which resulted in a failure of service.

    look at it this way 600 - 550 schools approximately use the BGFL for internet and they had no internet provision in school this morning because they messed up. We pay them over the odds for internet services, sims and cmis and support desk and the service is shocking. If a large organisation / company had a failure like this it wouldn't be accepted and I really hope schools and NM's get on the phone to Link2ICT and start complaining.

    And here comes the fix of the fix!

    "Date: Monday 16th December 2013

    Duration: From 21:30 tonight

    Affected Users: All BGfL Users

    Further to our earlier communication, essential updates to the new firewall
    will be made tonight at 9.30pm. There will be no impact to service during
    this time."

    ah well only four more sleeps ;-)

  13. #13

    Join Date
    Jun 2009
    Location
    Birmingham
    Posts
    616
    Thank Post
    93
    Thanked 75 Times in 67 Posts
    Rep Power
    26
    I honestly don't think it should have been dragged out like that. Maybe a test of 20 of the more forward thinking/challenging schools would have been good and then everyone else over a weekend.
    Apart from a 2 hour reduction in speed what else has gone wrong?

    How did it ruin the school play?

  14. #14

    CHR1S's Avatar
    Join Date
    Feb 2006
    Location
    Birmingham
    Posts
    4,573
    Thank Post
    1,652
    Thanked 506 Times in 309 Posts
    Rep Power
    221
    I won't go into it but the mayhem this "almost" caused today!!

    Thankfully tethering and mobile data saved the day!

    Edit - there was no access for about two hours, 504 errors and slower than 56k speeds!

  15. #15
    DocHouse's Avatar
    Join Date
    Jul 2011
    Location
    Somewhere near you
    Posts
    843
    Thank Post
    210
    Thanked 136 Times in 94 Posts
    Rep Power
    60
    Quote Originally Posted by Tricky_Dicky View Post
    I honestly don't think it should have been dragged out like that. Maybe a test of 20 of the more forward thinking/challenging schools would have been good and then everyone else over a weekend.
    Apart from a 2 hour reduction in speed what else has gone wrong?

    How did it ruin the school play?
    No access to vimeo channel where the class teacher had uploaded video clips and no back up copy, ended up having to do a voice over with a microphone. just spoilt it as the kids had worked really hard to produce these videos and we couldn't show them led to several crying children....

    Lost connection to our hosted admin and finance servers so office couldn't do their work.

    Two parents workshop cancelled as no internet access meant we couldn't use resources.

    Planned video conference with a large charity and two other schools in our cohort cancelled as we couldn't get online, not good when your supposed to be showcasing IT.

    No email access for office, SLT or teachers (some thought this was great, others had planning and documents they needed urgently.) Staff never seem to understand that it isn't my fault despite explaining that the whole city has issues.

    Our wasn't a reduction in speed we had 3hrs of nothing still intermittently dropping out and seems to be affecting a few schools local to us so maybe an exchange issue as well.



SHARE:
+ Post New Thread
Page 1 of 10 12345 ... LastLast

Similar Threads

  1. How to find out which machine a student logged onto, & when
    By indiegirl in forum How do you do....it?
    Replies: 32
    Last Post: 16th March 2012, 02:17 PM
  2. How to work out domain string (LDAP I think)
    By garethedmondson in forum Windows
    Replies: 16
    Last Post: 10th September 2009, 12:15 AM
  3. How to find out who deleted what
    By Chrish5 in forum Windows
    Replies: 9
    Last Post: 17th October 2008, 11:37 AM
  4. How to structure the layout of the Kaleidos Portal Intrest Groups
    By AM_LHS in forum Virtual Learning Platforms
    Replies: 1
    Last Post: 6th June 2008, 08:32 PM
  5. How to find out a preferred bidder
    By MK-2 in forum BSF
    Replies: 26
    Last Post: 8th January 2008, 10:56 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •