Schools Broadband issues

[winng]

we seem to be ok on filter 17

 

We're on 17 and we lost connection for a while too....

[nathan]

We're on 17 and we lost connection for a while too....

 

Not sure then, our tech on site said everything was up. Unless he didn't actually check and just wanted me off the phone.

 

EDIT: Just VPN into the site and all is working.

Edited November 25, 2016 by nathan

[fiza]

Our Primary School was back online after about 25 minutes.

[SchoolsBroadband]

Evening all,

 

Fortigate have advised us to replace firewall 3 which is happening late this evening with a new unit. It's early days but they think that one of the WAN uplink ports may have become faulty and is causing the device to reboot under certain situations.

 

There's emergency maintenance happening tonight to replace the unit and we'll update you once complete. All customers will have received the notification this evening.

 

Thanks

 

Dave

[Steve21]

I have to ask though, again why is one piece of equipment failing taking down so many schools? Isn't all of this equipment in redundant setups?

 

Steve

[SchoolsBroadband]

I have to ask though, again why is one piece of equipment failing taking down so many schools? Isn't all of this equipment in redundant setups?

 

Steve

 

in short yes they should failover and the firewall hardware is resilient. The price of a pair of them is more than a 3 bedroom house in some towns!

 

The Fortigates work in Active / passive mode. When Fortigate 3 stops passing traffic over one of it's WAN porst it should instigate a failover on to the other unit. We've had to physically log onto the remote reboot switch connected to Foritgate 3, reboot it and then Fortigate 4 (the failover unit) immediately springs into life and takes all traffic. When Fortigate 3 re-loads it then takes over as the master / active device.

 

Fortinet TAC engineers are working on why the HA isn't kicking in automatically (it's not a configuration issues) and from speaking with them they have some other customers with similar cases open relating to the v5.2.x firmware the firewalls are running. We have had failover in the past even on this firmware automatically failover on other units but it is not doing so in this specific case. We may upgrade the firmware at some point but we can't be cavalier upgrading firmware on such a major piece of kit as that could make matters worse by introducing other issues.

 

Very frustrating, particularly when compounding some other issues and couldn't have happened at a worse time.

 

As discussed although we've had quite a few customer affected the majority still haven't been.

 

Thanks for your patience. More comms to go out later tonight / tomorrow morning once the swop out has gone through.

 

Dave

[SchoolsBroadband]

Morning all,

 

The swop out of Fortigate 3 yesterday evening appears to have been successful. We will be closely monitoring over the next week.

 

Thanks

 

Dave

[nathan]

Thanks for the update, I suspect yet another Primus post coming up....

[Wubbalubbadub]

in short yes they should failover and the firewall hardware is resilient. The price of a pair of them is more than a 3 bedroom house in some towns!

 

I don't care if it cost the same as repairing Buckingham palace... If you're selling a service.. You need to pay!

If it was up to me, you would have a 3rd given the track record!

Edited November 29, 2016 by Wubbalubbadub
Spelling

[nathan]

I don't care if it cost the same as repairing Buckingham palace... If you're selling a service.. You need to pay!

If it was up to me, you would have a 3rd given the track record!

 

if failover didn't work then whats the point of having a 3rd?

[Fazza]

if failover didn't work then whats the point of having a 3rd?

 

3rd time lucky maybe?! lol

[nathan]

3rd time lucky maybe?! lol

 

[SchoolsBroadband]

Since the kit was replaced on Friday evening we've not seen the same issue happen again. The suspected faulty kit is on its way to Fortinet to diagnose exactly what issue has happened to the hardware. We're also expecting a new firmware version to be released very soon which addresses this particular set of conditions where failover did not occur when it should have done.

 

We'll keep you updated on here and by the usual channels too. I suspect a full incident report will not be fully available until Fortinet get back in touch with us regarding the what exactly happened to the faulty unit and why a failover condition didn't occur when it should have done.

 

FYI I think you can actually now have 3 active / active / active Fortigates in a cluster although we've not tested that and I suspect it'd be quite a network overhaul to implement it too.

 

Thanks

 

Dave

[jertsy]

Anyone else getting intermittent connection issue?

 

EDIT - Ihave just called up and they are experiencing a network issue.

Edited December 1, 2016 by jertsy

[synaesthesia]

I hope they don't mind me copy & pasting this from their Technical Director @SchoolsBroadband :

 

We are allowing FortiNet to investigate the issue as it happens to finally get to the bottom of the issue, so we might be down a little longer than normal to allow this investigation to occur.

 

Comms are going out shortly and updates will follow as normal.

 

FortiNet are currently doing a screen share with a senior NOC engineer in order to provide a final resolution.

[nathan]

Anyone else getting intermittent connection issue?

 

EDIT - Ihave just called up and they are experiencing a network issue.

 

filter 17?

[winng]

We're on filter 17 and also lost connection!

[lou-ashtonhurst]

Unfortunately we have had a recurrence of the issue this morning at 10.30.

 

We are currently working with senior engineers at Fortinet who are logged live into the kit - this does mean that the service will be impacted for a slightly longer period of time while we allow them to inspect what is happening.

 

We appreciate this is not an acceptable level of service and we are doing everything in our power to restore stability as soon as possible.

 

Thank you for your patience.

 

Lou Ashtonhurst

NOC Manager

Talk Straight/Schools Broadband

Edited December 1, 2016 by lou-ashtonhurst

[jertsy]

filter 17?

 

Filter 5

[Wubbalubbadub]

Anyone worked out how much downtime we have had in total? = what % of up time we are being provided!?

 

As nice as the name Carole is.. I'm sick to death of seeing emails from her saying stuff is down!

 

Filter 14 here!

[nathan]

just had conformation that it's all our schools (inc filter 14 and 9).

 

Don't mind them taking a little long so long as they get to the bottom of problem. Laughable that Fortinet are on everyones buy list on the share sites - if only they knew...

[winng]

Unfortunately we have had a recurrence of the issue this morning at 10.30.

 

We are currently working with senior engineers at Fortinet who are logged live into the kit - this does mean that the service will be impacted for a slightly longer period of time while we allow them to inspect what is happening.

 

We appreciate this is not an acceptable level of service and we are doing everything in our power to restore stability as soon as possible.

 

Thank you for your patience.

 

Lou Ashtonhurst

NOC Manager

Talk Straight/Schools Broadband

Having switched to SB on 1st November, we are not impressed with the level of service to date. Following initial setup issues we now seem to have loss of connection weekly.

[nathan]

Having switched to SB on 1st November, we are not impressed with the level of service to date. Following initial setup issues we now seem to have loss of connection weekly.

 

We have 10 schools with them, in the last 6 months things haven't been great but before then it was brilliant. I just hope they sort this issue out once and for all.

[nathan]

Just heard that one of our schools are fine, just finding out what filter they are on.

 

We have a school on filter5 that is working ok. strange

 

EDIT: Now confirmed 3 our of 10 schools are still up on filters 5, 14 and 17

Edited December 1, 2016 by nathan

[Blue_Cookeh]

Been down for an hour and counting so far. We'll be looking into our SLAs and contracts today, this is completely unacceptable during the school day.

 

I appreciate FortiGate are working on the problem, I appreciate it might not be SB's fault and that they're working on sorting their network out, and I totally appreciate how unreachable I've been when it comes to having a talk with them, but this is starting to negatively impact our school on a large scale.

 

What's the escalation process on complaints?

Edited December 1, 2016 by Blue_Cookeh