Birmingham Grid for Learning (BGfL) Thread, How to black out most of Birmingham....Let Link2ICT update your firewall in Regional Broadband Consortiums (RBC); Well done Link2ICT on completing your successful firewall upgrade (Their quote not mine :-S)... you managed to knock off the ...
16th December 2013, 03:44 PM #1
How to black out most of Birmingham....Let Link2ICT update your firewall
Well done Link2ICT on completing your successful firewall upgrade (Their quote not mine :-S)... you managed to knock off the whole city in the process.
Seriously annoyed at their level of incompetence and unprofessional behaviour
Who in their right mind would deploy a full city wide upgrade that hasn't been tested fully....
One more reason to leave Link2ICT
16th December 2013, 03:50 PM #2
We just ground to a halt, didnt lose everything. I rang and they said yes, we are aware of the issue....so i guessed it was the firewall...
16th December 2013, 03:54 PM #3
Subject: F.A.O Head Teacher/School Technician/Network Manager - IMPORTANT: Firewall Upgrade Update
Link2ICT is pleased to announce successful completion of our Firewall
We aware of the slow performance that you may have encountered this morning
and have now found a solution which has enabled us to resume service.
Please rest assured we are proactively monitoring the network to ensure
there are no further performance issues and apologise for the inconvenience
this may have caused.
Benefits of the new Firewall:
The new firewall will offer additional features including:
The future ability to provision access to the Internet for variety of
devices, such as Kindles, Android devices and some iPad Apps.
Increased application control, allowing access to cloud based
applications such as Office 365/Office 365 Hybrid Mode and Google
Advanced security features which will help identify suspicious
traffic such as Malware, prevent attacks on websites, and offer
Distributed Denial of Service (DDoS) protection
The following improvements will also be evident:
The maximum Internet traffic throughput will increase to 10Gb/s
High quality logging of all Internet traffic, which will assist with
investigations and troubleshooting
Availability of enhanced reporting facilities
Improved security by the automatic highlighting of any unused
Inbuilt protection against low level internet attacks
This is the beginning of a number of investments and value added services
that Link2ICT intends to offer customers, ultimately improving and
enhancing the Link2ICT Network and Internet Service.
Sent two hours after they caused chaos!
16th December 2013, 03:57 PM #4
To be fair to Link2ICT it wasn't the firewall that actually caused the issue. It was part of a filtering process that for some reason a service on that failed and this then caused the very slow internet access for a couple of hours.
How do you stress test a change like this?
16th December 2013, 04:06 PM #5
Easy option is to at least batch test 50 schools first and not roll out the whole thing in one go... They used to do this with CMIS, PCE and SIMS, seem to have forgotten to test things before deploying them and communication seems to have failed as nobody could get through to Service Desk via phone or email.
As for what's failed I've spoken to four different service desk bods and all four have given me different answers so not sure who to trust one thing is clear somebody really messed up and then they sent that email to say it was successfully implemented which kind of rubs salt in the wound.
That and the fact that it ruined our christmas play has really annoyed me.
Not impressed today
16th December 2013, 04:17 PM #6
What was their backout plan when they realised the issue though?
16th December 2013, 04:31 PM #7
it looks like they rolled it back after three hours of attempting to get it to work.... not sure if they really had a plan, or why they decided to do it during school hours but i'm not part of their brain trust i just have to pick up the pieces like most of my colleagues when they drop us in it.
just had an email to say further updates will be carried out tonight at 9:30pm, fingers crossed for tomorrow!
16th December 2013, 04:34 PM #8
I have to agree, I wouldn't exactly describe the firewall migration as success, but more disastrous. I think lessons have been learnt today for sure.
Why it needed to be done all in one weekend is unclear. I think batches of 50 or 100 would have been more appropriate, unless there was a good reason it needed to be done in one go.
I would have much preferred if it was delayed until February half-term, without Christmas round the corner adding to the pressure to get all the work completed.
16th December 2013, 04:34 PM #9
It wasn't a back out plan. They checked and found a problem service which was fixed by restarting it.
16th December 2013, 04:35 PM #10
It should have been done over a half term when there is much less traffic and problems could have been resolved much easier.
Originally Posted by Michael
16th December 2013, 04:37 PM #11
I don't think a stress test was necessary, however the fact remains, if it was done in batches of 50 or 100 - the problem would (without doubt) have been resolved faster. I think everyone just presumed it was the new firewall (and rightly so), making it more difficult to isolate to a filtering type issue. As a result what probably took minutes to fix took hours to diagnose and isolate the issue.
Originally Posted by Tricky_Dicky
16th December 2013, 04:47 PM #12
[QUOTE=Tricky_Dicky;1102646]It wasn't a back out plan. They checked and found a problem service which was fixed by restarting it.[/
Yep but why wasn't this tested first and spotted first, it was planned and implemented poorly which resulted in a failure of service.
look at it this way 600 - 550 schools approximately use the BGFL for internet and they had no internet provision in school this morning because they messed up. We pay them over the odds for internet services, sims and cmis and support desk and the service is shocking. If a large organisation / company had a failure like this it wouldn't be accepted and I really hope schools and NM's get on the phone to Link2ICT and start complaining.
And here comes the fix of the fix!
"Date: Monday 16th December 2013
Duration: From 21:30 tonight
Affected Users: All BGfL Users
Further to our earlier communication, essential updates to the new firewall
will be made tonight at 9.30pm. There will be no impact to service during
ah well only four more sleeps ;-)
16th December 2013, 05:07 PM #13
I honestly don't think it should have been dragged out like that. Maybe a test of 20 of the more forward thinking/challenging schools would have been good and then everyone else over a weekend.
Apart from a 2 hour reduction in speed what else has gone wrong?
How did it ruin the school play?
16th December 2013, 05:20 PM #14
I won't go into it but the mayhem this "almost" caused today!!
Thankfully tethering and mobile data saved the day!
Edit - there was no access for about two hours, 504 errors and slower than 56k speeds!
16th December 2013, 05:45 PM #15
No access to vimeo channel where the class teacher had uploaded video clips and no back up copy, ended up having to do a voice over with a microphone. just spoilt it as the kids had worked really hard to produce these videos and we couldn't show them led to several crying children....
Originally Posted by Tricky_Dicky
Lost connection to our hosted admin and finance servers so office couldn't do their work.
Two parents workshop cancelled as no internet access meant we couldn't use resources.
Planned video conference with a large charity and two other schools in our cohort cancelled as we couldn't get online, not good when your supposed to be showcasing IT.
No email access for office, SLT or teachers (some thought this was great, others had planning and documents they needed urgently.) Staff never seem to understand that it isn't my fault despite explaining that the whole city has issues.
Our wasn't a reduction in speed we had 3hrs of nothing still intermittently dropping out and seems to be affecting a few schools local to us so maybe an exchange issue as well.
By indiegirl in forum How do you do....it?
Last Post: 16th March 2012, 02:17 PM
By garethedmondson in forum Windows
Last Post: 10th September 2009, 12:15 AM
By Chrish5 in forum Windows
Last Post: 17th October 2008, 11:37 AM
By AM_LHS in forum Virtual Learning Platforms
Last Post: 6th June 2008, 08:32 PM
Last Post: 8th January 2008, 10:56 AM
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)