Schools Broadband issues

winng · November 25, 2016

we seem to be ok on filter 17

We're on 17 and we lost connection for a while too....

nathan · November 25, 2016

We're on 17 and we lost connection for a while too....

Not sure then, our tech on site said everything was up. Unless he didn't actually check and just wanted me off the phone.

EDIT: Just VPN into the site and all is working.

Edited November 25, 2016 by nathan

fiza · November 25, 2016

Our Primary School was back online after about 25 minutes.

SchoolsBroadband · November 25, 2016

Evening all,

Fortigate have advised us to replace firewall 3 which is happening late this evening with a new unit. It's early days but they think that one of the WAN uplink ports may have become faulty and is causing the device to reboot under certain situations.

There's emergency maintenance happening tonight to replace the unit and we'll update you once complete. All customers will have received the notification this evening.

Thanks

Dave

Steve21 · November 25, 2016

I have to ask though, again why is one piece of equipment failing taking down so many schools? Isn't all of this equipment in redundant setups?

Steve

SchoolsBroadband · November 25, 2016

I have to ask though, again why is one piece of equipment failing taking down so many schools? Isn't all of this equipment in redundant setups?

Steve

in short yes they should failover and the firewall hardware is resilient. The price of a pair of them is more than a 3 bedroom house in some towns!

The Fortigates work in Active / passive mode. When Fortigate 3 stops passing traffic over one of it's WAN porst it should instigate a failover on to the other unit. We've had to physically log onto the remote reboot switch connected to Foritgate 3, reboot it and then Fortigate 4 (the failover unit) immediately springs into life and takes all traffic. When Fortigate 3 re-loads it then takes over as the master / active device.

Fortinet TAC engineers are working on why the HA isn't kicking in automatically (it's not a configuration issues) and from speaking with them they have some other customers with similar cases open relating to the v5.2.x firmware the firewalls are running. We have had failover in the past even on this firmware automatically failover on other units but it is not doing so in this specific case. We may upgrade the firmware at some point but we can't be cavalier upgrading firmware on such a major piece of kit as that could make matters worse by introducing other issues.

Very frustrating, particularly when compounding some other issues and couldn't have happened at a worse time.

As discussed although we've had quite a few customer affected the majority still haven't been.

Thanks for your patience. More comms to go out later tonight / tomorrow morning once the swop out has gone through.

Dave

SchoolsBroadband · November 26, 2016

Morning all,

The swop out of Fortigate 3 yesterday evening appears to have been successful. We will be closely monitoring over the next week.

Thanks

Dave

nathan · November 26, 2016

Thanks for the update, I suspect yet another Primus post coming up....

Wubbalubbadub · November 29, 2016

in short yes they should failover and the firewall hardware is resilient. The price of a pair of them is more than a 3 bedroom house in some towns!

I don't care if it cost the same as repairing Buckingham palace... If you're selling a service.. You need to pay!

If it was up to me, you would have a 3rd given the track record!

Edited November 29, 2016 by Wubbalubbadub
Spelling

nathan · November 29, 2016

I don't care if it cost the same as repairing Buckingham palace... If you're selling a service.. You need to pay!
If it was up to me, you would have a 3rd given the track record!

if failover didn't work then whats the point of having a 3rd?

Fazza · November 29, 2016

if failover didn't work then whats the point of having a 3rd?

3rd time lucky maybe?! lol

nathan · November 29, 2016

3rd time lucky maybe?! lol

SchoolsBroadband · November 30, 2016

Since the kit was replaced on Friday evening we've not seen the same issue happen again. The suspected faulty kit is on its way to Fortinet to diagnose exactly what issue has happened to the hardware. We're also expecting a new firmware version to be released very soon which addresses this particular set of conditions where failover did not occur when it should have done.

We'll keep you updated on here and by the usual channels too. I suspect a full incident report will not be fully available until Fortinet get back in touch with us regarding the what exactly happened to the faulty unit and why a failover condition didn't occur when it should have done.

FYI I think you can actually now have 3 active / active / active Fortigates in a cluster although we've not tested that and I suspect it'd be quite a network overhaul to implement it too.

Thanks

Dave

jertsy · December 1, 2016

Anyone else getting intermittent connection issue?

EDIT - Ihave just called up and they are experiencing a network issue.

Edited December 1, 2016 by jertsy

synaesthesia · December 1, 2016

I hope they don't mind me copy & pasting this from their Technical Director @SchoolsBroadband :

We are allowing FortiNet to investigate the issue as it happens to finally get to the bottom of the issue, so we might be down a little longer than normal to allow this investigation to occur.

Comms are going out shortly and updates will follow as normal.

FortiNet are currently doing a screen share with a senior NOC engineer in order to provide a final resolution.

nathan · December 1, 2016

Anyone else getting intermittent connection issue?

EDIT - Ihave just called up and they are experiencing a network issue.

filter 17?

winng · December 1, 2016

We're on filter 17 and also lost connection!

lou-ashtonhurst · December 1, 2016

Unfortunately we have had a recurrence of the issue this morning at 10.30.

We are currently working with senior engineers at Fortinet who are logged live into the kit - this does mean that the service will be impacted for a slightly longer period of time while we allow them to inspect what is happening.

We appreciate this is not an acceptable level of service and we are doing everything in our power to restore stability as soon as possible.

Thank you for your patience.

Lou Ashtonhurst

NOC Manager

Talk Straight/Schools Broadband

Edited December 1, 2016 by lou-ashtonhurst

jertsy · December 1, 2016

filter 17?

Filter 5

Wubbalubbadub · December 1, 2016

Anyone worked out how much downtime we have had in total? = what % of up time we are being provided!?

As nice as the name Carole is.. I'm sick to death of seeing emails from her saying stuff is down!

Filter 14 here!

nathan · December 1, 2016

just had conformation that it's all our schools (inc filter 14 and 9).

Don't mind them taking a little long so long as they get to the bottom of problem. Laughable that Fortinet are on everyones buy list on the share sites - if only they knew...

winng · December 1, 2016

Unfortunately we have had a recurrence of the issue this morning at 10.30.

We are currently working with senior engineers at Fortinet who are logged live into the kit - this does mean that the service will be impacted for a slightly longer period of time while we allow them to inspect what is happening.

We appreciate this is not an acceptable level of service and we are doing everything in our power to restore stability as soon as possible.

Thank you for your patience.

Lou Ashtonhurst
NOC Manager
Talk Straight/Schools Broadband

Having switched to SB on 1st November, we are not impressed with the level of service to date. Following initial setup issues we now seem to have loss of connection weekly.

nathan · December 1, 2016

Having switched to SB on 1st November, we are not impressed with the level of service to date. Following initial setup issues we now seem to have loss of connection weekly.

We have 10 schools with them, in the last 6 months things haven't been great but before then it was brilliant. I just hope they sort this issue out once and for all.

nathan · December 1, 2016

Just heard that one of our schools are fine, just finding out what filter they are on.

We have a school on filter5 that is working ok. strange

EDIT: Now confirmed 3 our of 10 schools are still up on filters 5, 14 and 17

Edited December 1, 2016 by nathan

Blue_Cookeh · December 1, 2016

Been down for an hour and counting so far. We'll be looking into our SLAs and contracts today, this is completely unacceptable during the school day.

I appreciate FortiGate are working on the problem, I appreciate it might not be SB's fault and that they're working on sorting their network out, and I totally appreciate how unreachable I've been when it comes to having a talk with them, but this is starting to negatively impact our school on a large scale.

What's the escalation process on complaints?

Edited December 1, 2016 by Blue_Cookeh

Schools Broadband issues

Recommended Posts

winng

nathan

fiza