Schools Broadband - Service Credit for Outage

[paulkerton]

No redirection here either, not immediately anyway. Took 5 to 10 minutes before we got connectivity back.

This move to Netsweeper is going really well, isn't it? I left a school that went to Netsweeper with SBB (forced to, I was rather happy with LS) and it was 10 months till I joined another, and it seems to have become worse over that time, not better.

[paulkerton]

Netsweeper are investigating why it happened.

 

That's becoming quite the list now isn't it?

 

May I suggest the easiest answer as to why it happens is "Because it's Netsweeper" which is the net filter equivalent of purchasing a unicycle to take part in the Tour de France...

[caffrey]

And again.... Luckily the kids have just gone home

[cullingsh]

Based on what Dave Tindell has said, it would appear the Cisco load balancer does't work, at least it hasn't for us. A string of calls this afternoon and another lot of lessons interrupted.

[cullingsh]

In addition to my previous comment, I have had screenshots that show inlineproxy10 and inlineproxy11 as being affected. Plus it weren't just a few minutes either.

[ChrisC]

Why do I seem to remember around 2 years ago (give or take) there was a hardware failure of [one of] the load balancer(s) that meant it had to be replaced?

 

Again, this didn't affect everyone, but we were down for the X hours it took an engineer to get to the data centre and replace.

[SchoolsBroadband]

Evening folks,

 

our Cisco load balancers are absolutely fine. We have seen today a failure of the NS Proxy Daemon on numerous servers at random times. Unlike seeing a physical hardware failure this doesn't as standard pull the affected proxy server out of the pool and we have had to manually restart the daemon on it rather than the server itself.

 

We believe this is a Netsweeper software issue again and we suspect a fallout from the software upgrade which was performed a couple of weeks ago. We are also seeing an a bug affecting authentication which Netsweeper are releasing a patch for and we should have imminently. Our engineers have been speaking with Netsweeper in Canada all afternoon and we will continue to do so tonight. I also have a call with their CTO and CEO tomorrow late morning to discuss.

 

I will have some sort of update from our NOC team tonight although I suspect they will work into the very early hours with Netsweeper.

 

Sorry for the blips in service today. We're working with Netsweeper to do all we can to make sure they don't reoccur tomorrow.

 

Dave

[Edu-IT]

It appears that the netsweeper daemon died on one server. A restart fixed it. Netsweeper are investigating why it happened. This only affected one proxy server out of the many we have.

 

Now fixed and working again in a few minutes.

 

Sorry for any inconvenience caused.

 

Thanks

 

Dave

So when you said one, you meant many, really?

 

At least 9, 10 and 11.

 

From a technical point of view what stops any of the other numbered proxy servers becoming affected assuming identical hardware/config on each?

Edited March 4, 2020 by Edu-IT

[fiza]

At our Primary School we were with SBB for 3 years and had nothing but problems which were blamed on Lightspeed. At the end of the 3 years we moved away from SBB at the same time they moved to Netsweeper. We moved to LGfL who also use Netsweeper. Not had a minute of downtime in the year we have been with LGfL. Just saying.....

[eddyc]

At our Primary School we were with SBB for 3 years and had nothing but problems which were blamed on Lightspeed. At the end of the 3 years we moved away from SBB at the same time they moved to Netsweeper. We moved to LGfL who also use Netsweeper. Not had a minute of downtime in the year we have been with LGfL. Just saying.....

 

Likewise, our connection is provided by our LEA. Our LEA moved from BLOXX to Netsweeper around 18 months ago. Similar setup with multiple transparent proxies all load balanced. We’ve never had any issues with Netsweeper as a part of this provision either...

[Guest Guest]

Likewise, our connection is provided by our LEA. Our LEA moved from BLOXX to Netsweeper around 18 months ago. Similar setup with multiple transparent proxies all load balanced. We’ve never had any issues with Netsweeper as a part of this provision either...

We had a similar setup via our LA using Netsweeper and we had no issues

[SchoolsBroadband]

So when you said one, you meant many, really?

 

At least 9, 10 and 11.

 

From a technical point of view what stops any of the other numbered proxy servers becoming affected assuming identical hardware/config on each?

 

Hi @Edu-IT,

 

when I originally replied it was 1 but that changed over time.

 

All hardware for the inline proxies are exactly the same.

 

Netsweeper have some more thoughts at the moment and we're working with them this evening. I will update in due course once they suggest which option we go with.

 

To the other replies when we first brought Netsweeper on we had no issues either up until November(ish) time with nearly 18 months of perfect service (bar a few very minor niggles). We are one of the biggest users of Netsweeper in the UK. Our service is slightly different to other LA's as we can use the same internal IP address over multiple sites (Netsweeper wrote this for us) but bar that we are also heavy users of Wagent which also interacts with this and looks after authentication. We don't have an exact replica of what other LA's do. That said Netsweeper asked us to put in certain hardware (network and servers) which we have so we haven't gone against what they recommended we put in. We'll continue to do this on their advice too.

 

Watch this space.

 

Thanks

 

Dave

[SchoolsBroadband]

ok so lots of conversations with Netsweeper this evening. They're making some software changes to cache's on each inline server tonight and monitoring. Irrelevant of the outcome of that we have also been asked to potentially upgrade the hardware our loggers reside on. This is something we were going to do anyway to increase capacity and the servers just need to be brought live as they are already in our DC. If we do that it will be done on Friday evening.

 

Thanks

 

Dave

[FN-GM]

Remember a few years back it was light speed's fault... Now its Netsweeper....

[SchoolsBroadband]

As much as we've had issues with Netsweeper at least they try to fix things and engage with us. I cant knock them for their effort at all. They're also very different solutions. Our old vendor was their own hardware and closed shop for us doing or monitoring anything on it other than the web gui which was very limited from a system point of view.

 

Number 1 priority is to fix things so our customers get a consistent reliable service as they have done on netsweeper in the past. I'm sure that will be the case again with the help of Netsweeper. I do take offence at thinking our engineers are constantly at fault and we dont tell the truth as that's important to us. We have 100% uptime on our other services.

 

I met a customer that I wont name at BETT this year who I'd never met before who thanked me for our candour, openness and honesty on here whether good or bad and the fact we engage with you all. We will continue to do so and will have the issue fixed with the help of Netsweeper. Thank you to that customer in particular as you did say how much you appreciated our service and despite the occasional bump in the road have done for many years despite other more vocal people on here who are not our fans.

 

That's it for now I think. I'll be in touch with an update when I have it from our or Netsweeper engineers.

 

Dave

[caffrey]

Dave,

I appreciate your candour, it's refreshing

But we are not fans, we are customers - which is why we are vocal. End of the day it's embarrassing for everyone

 

Fingers crossed it's a better day today

[Opendium_Steve]

In @SchoolsBroadband's defence, in the past we've very occasionally seen cases where a new app suddenly becomes popular and that app's network traffic has something unusual in it that triggers an existing bug in our filters. So it can certainly be the case that a previously stable system can suddenly become unstable simply because a bunch of users have changed their behaviour.

 

I don't know if that's what happened here, but just pointing out that you can get failures that aren't related to any changes made by IT staff or filtering provider. When you do upgrades you can make plans to roll-back the upgrade if it causes problems, but if a long standing bug suddenly starts to be regularly triggered there's no way to just roll-back a change to fix the problem. Some software bugs can sit there unnoticed for years.

 

In our case we keep schools very isolated (we mostly supply on-prem systems, but our cloud filtering systems also maintain a lot of segregation between schools), so problems tend to be isolated to individual schools rather than impacting everyone. I'm always quite wary about bundling large numbers of schools onto a single system where a bug that affects one school can easily escalate to affect many.

 

Hopefully everyone in IT understands that the occasional software bug is unavoidable and what matters is how well it is handled once it is discovered - communication with stakeholders, time to fix/mitigate, good customer service in general.

[SchoolsBroadband]

Thanks @Opendium_Steve,

 

the Google authentication bug which has occurred since the software upgrade should be patched this evening. Netsweeper are finalising that through their QA procedure. They've also made a change around logging which is part of the problems we have seen of late and we have been fine today having gone through the half eleven busy period. We are working to their recommendations and we may bring some new logging servers live if they deem that necessary as they are already in our racks and have been ready to go for some time. We believe issues relating to live logging and a crash of the NS Daemon are logging related. Netsweeper are unsure at the moment if this is software or hardware related but are still working on this.

 

So far today has been good (bar Virgin having a major outage on the South Coast see https://www.ispreview.co.uk/index.php/2020/03/major-virgin-media-broadband-and-tv-outage-on-south-coast.html)

 

Thanks

 

Dave

[paulkerton]

the Google authentication bug which has occurred since the software upgrade should be patched this evening.

Google authentication bug as in the Netsweeper Workstation Agent will now work? Would be nice to see some authentication against the user!

[Griff]

I met a customer that I wont name at BETT this year who I'd never met before who thanked me for our candour, openness and honesty on here whether good or bad and the fact we engage with you all. We will continue to do so and will have the issue fixed with the help of Netsweeper. Thank you to that customer in particular as you did say how much you appreciated our service and despite the occasional bump in the road have done for many years despite other more vocal people on here who are not our fans.

 

That's it for now I think. I'll be in touch with an update when I have it from our or Netsweeper engineers.

 

Dave

 

:pound::pound::pound::pound::pound::pound::pound:

[BrotherSidious]

I appreciate openness and honesty as much as the next person. I also appreciate everyone working hard to resolve issues, escalating them and having discussions with NetSweeper but ultimately what my SLT wants is a reliable, filtered and available connection to the internet to support the delivery of the school curriculum and the school's admin functions. In all of the previous instances, there have been explanations and fixes and patches and I am sure things are moving forwards but what teachers and students see is disruption. They do not see the progress that is discussed. What they see is what was once a reliable and stable internet connection is becoming more and more unreliable. They remember every single disrupted lesson due to the sudden on the fly replanning or adjustment that may require. As support staff, we get inundated with calls from multiple classrooms when the internet goes down or slows to the point of being unusable and we frequently have to appease them and can offer no real comfort in terms of when things will be resolved, how long the outage will be and will the issue occur again. As much as Schools Broadband may be working to improve things and maybe improving their communication with us, unfortunately, the discontented voices of staff continue to rise and feedback to SLT so it surely cannot come as a surprise that we are asked to look at alternatives when our current contract comes to an end. Realistically I can offer them no reason to stay as "Netsweeper is better then lightspeed", "NetSweeper is looking to improve things", "new hardware is going in", etc is of no interest to them whatsoever. They are completely uninterested. All they want is a reliable internet connection so that they can deliver the lessons they have planned. As much as I applaud Schools Broadband honesty with regards to what has happened and what they are doing we are not looking for a broadband supplier who is "getting there" we are looking for one that has arrived at a reliable and stable solution. When staff are complaining that their home broadband is more reliable you know you're in trouble! Edited March 6, 2020 by BrotherSidious

[paulkerton]

Realistically I can offer them no reason to stay as "Netsweeper is better then lightspeed"

I know this isn't your opinion, but if anyone says that to me personally, this meme will be my response...

 

[Griff]

Lightspeed is the best out there right now.

[synaesthesia]

All I can bring to this conversation is this.

 

[mthomas08]

Reading through these posts, I'm glad we're with Orbital Net.

 

Barely any downtime in the 3 years we've been with them, only Lightspeed issues we have are down to licenses not automatically renewing (which they are trying to get LS to sort out). We get our McAfee from the usual LEA and our domain name (mainly due to the work required).

 

We don't have to fight for weeks to get ports locked/opened, got a backup connection and we save money. They are local too.