Schools Broadband In-line Filter down?

[SchoolsBroadband]

VMB for us was unaffected, any idea who else was as could be useful for future procurement decisions.

 

Any carrier or ISP in that data centre on one of the 4 floors without power will have been affected (whether they have resilient power or not). I believe the following have been as I've seen notifications from them.

 

BT, Talk Talk, Entanet, Virgin, Exa Networks, FluidOne, Virtual One, Vodafone, Origin, SSE, Cogent, Level 3, Vaioni, Giganet......

 

This will only be the tip of the iceberg though and many more will have had full or partial outages.

 

Dave

[caffrey]

Well it's 9pm and it's still down - so that's 16 hours and counting. I really hope it's up in the morning as results are due in and I'll no doubt get it in the neck, I know it's not SBBs fault but still mightily frustrating... Edited August 18, 2020 by caffrey

[MrWu]

Still down for me too ... they want the GCSE results downloaded 8am in the morning ...

[caffrey]

Seems to be working now - phew

[SchoolsBroadband]

Morning folks,

 

although most carriers were back online around half three, all carriers (Virgin in particular) were restored at around 11pm last night once Equinix had fully restored power to all parts of the data centre. As of then we were just about fully back to normal.

 

Dave

Edited August 19, 2020 by SchoolsBroadband

[caffrey]

Thanks Dave, and thank you for all the regular communication yesterday

[RobFuller]

Thanks Dave, and thank you for all the regular communication yesterday

 

Ditto great comms!

[Wave9_Lee]

VMB for us was unaffected, any idea who else was as could be useful for future procurement decisions.

I wonder if it's just NNI traffic, as we take DIA from many carriers including VMB, TTB, BT etc and not a single outage yesterday.

[Olliesaurus]

Agreed with above, far better communications than we have experienced in the past, more or this please!

 

To add, I had phoned in to support yesterday and dealt with a member of the NOC team who really was great to work with, I’ve worked with her in the past for installs and config changes and every time she has always phoned back to check all is OK, she’s a real superstar and I hope you employee more like her to the Talk Straight ranks!

[snagrat]

Agreed with above, far better communications than we have experienced in the past, more or this please!

 

To add, I had phoned in to support yesterday and dealt with a member of the NOC team who really was great to work with, I’ve worked with her in the past for installs and config changes and every time she has always phoned back to check all is OK, she’s a real superstar and I hope you employee more like her to the Talk Straight ranks!

 

Is that Kulthoom by any chance?

I work with her a lot and really like her dedication and attention to detail!

[caffrey]

Not going into the full story but I still don't have a working backup line because of the support -I'd just rather not deal with it

[dhicks]

Any carrier or ISP in that data centre on one of the 4 floors without power will have been affected (whether they have resilient power or not).

 

Our main line went down (fiber line provided by Exponential-e) but our secondary line was okay (standard BT fiber-to-the-cabinet ADSL service).

[SchoolsBroadband]

Agreed with above, far better communications than we have experienced in the past, more or this please!

 

To add, I had phoned in to support yesterday and dealt with a member of the NOC team who really was great to work with, I’ve worked with her in the past for installs and config changes and every time she has always phoned back to check all is OK, she’s a real superstar and I hope you employee more like her to the Talk Straight ranks!

 

Thanks @Olliesaurus / @snagrat,

 

yes Kulthoom is a superstar, that said we do have lots of other great engineers too.

 

We've had a new head of support start as of about a month ago who brings a lot of experience from working in the ISP industry heading up support teams so this change is already having a positive effect which is good. He's called Paul Collier in case you ever need to speak with him and is an escalation point in our new customer services document which will be sent out soon i think.

 

Thanks

 

Dave

[andy_b]

Loosing my mind today.....

[DalekSec]

Ours went down today around 10:45-11:15

[sister_annex]

Wasn't just us then... Sooner we can be off this the better

[andy_b]

Up down Up down .....

 

I think I'd prefer it to just be down

[caffrey]

Ours is okay for now...

[andy_b]

Ours is okay for now...

 

Seems to have stabilised.

[andy_b]

Or not lol

[cullingsh]

Bouncing up and down more than my yo-yo. AGAIN

[SchoolsBroadband]

Hello everyone,

 

sorry for the issues today. You should all have received updates throughout the course of the day. This issue affected 3 of our current 16 proxy servers and customers connected via them (not to anyone else or any non HTTP and / or HTTPS service) and was due to a bug that was found in a patch applied yesterday. This patch had already been rolled out 3 weeks ago to another server with no issues so we rolled out to all other ones yesterday.

 

The patch was put in place to allow us to use some new reporting infrastructure we are working on which will vastly improve the reports available and the time it takes to create reports.

 

The issue appears to have been that if a reporting entry was an empty file then it would force the NS Proxy service to crash and re-start (hence the intermittency). Of the millions of entries we log for content filtering reporting I've been told today we had a number in the teens that were empty and unfortunately made NS Proxy crash and those servers thus forced closed (stopping web browsing) rather than forced open (allowing access to anything). A temporary fix has been applied and a patch is being rolled out this evening.

 

 

Thanks

 

Dave

Edited December 15, 2020 by Dos_Box