Schools Broadband In-line Filter down?

[Dos_Box]

**MOD WARNING**

This thread has been brought to our attention due to the way what was feedback has devolved into unhelpful comments.

We can appreciate some of you are frustrated, but please keep things professional in here.

No company wants problems and all will do their upmost to fix them as quickly as possible, but often things do not, or cannot, happen overnight, as much as we would want them to.

Again, please keep things civil in here as we are now keeping an eye on things.

[theo_logical]

Dos_Box, can you be a bit more precise and tell us which bit Mr Tindall was offended by?

 

None of us can see in to the future so discussion of past and present performance of a potential supplier helps us to make informed choices.

[Primus]

**MOD WARNING**

This thread has been brought to our attention due to the way what was feedback has devolved into unhelpful comments.

We can appreciate some of you are frustrated, but please keep things professional in here.

No company wants problems and all will do their upmost to fix them as quickly as possible, but often things do not, or cannot, happen overnight, as much as we would want them to.

Again, please keep things civil in here as we are now keeping an eye on things.

 

This isn't a good look - they're a forum sponsor and now their threads are receiving special treatment - where was anyone unprofessional?

[BrotherSidious]

Hello everyone,

 

Although the issue hasn't affected all of ours customers, one affected customer is one too many. We may put together a technical webinar for those of our customers that wish to understand the technical intracicies of what happened and how everything was fixed if there is demand.

 

Dave

 

May I ask if there is anything that identifies the subset of customers that were affected. Were they using a setting in common or is it just a random subset of customers.

[DJ-1701]

Dos_Box, can you be a bit more precise and tell us which bit Mr Tindall was offended by?

 

None of us can see in to the future so discussion of past and present performance of a potential supplier helps us to make informed choices.

 

Who said it was Mr Tindall that reported this?

 

This isn't a good look - they're a forum sponsor and now their threads are receiving special treatment - where was anyone unprofessional?

 

Being civil is special treatment? How?

 

This forum was created to help. Logic suggests if you are having an issue to state the issue, if you have a fix, state the fix.

 

We all face frustration with so many things, and it can be hard walking a line of getting your message across and being polite, but at the end of the day we are all human and having a go at someone that is trying to help you in a public place isn't very professional now is it?

[Dos_Box]

This isn't a good look - they're a forum sponsor and now their threads are receiving special treatment - where was anyone unprofessional?

 

They are not receiving special treatment, we have also intervened on other threads in the past where the original point about other companies products and services has turned into members frustration bubbling over.

In much the same way we do not allow profanity on the forums we, that is the royal 'we' on EduGeek, members and staff alike are, well regarded in the way in which we conduct ourselves, and this was starting to go beyond acceptable behaviour and into repeated rant territory here.

By all means provide feedback, but please keep it relevant.

[caffrey]

Depends if you're not getting anywhere with their support

[Primus]

They are not receiving special treatment, we have also intervened on other threads in the past where the original point about other companies products and services has turned into members frustration bubbling over.

In much the same way we do not allow profanity on the forums we, that is the royal 'we' on EduGeek, members and staff alike are, well regarded in the way in which we conduct ourselves, and this was starting to go beyond acceptable behaviour and into repeated rant territory here.

By all means provide feedback, but please keep it relevant.

 

There was no profanity, no unprofessional comments but what you were seeing is many members very very unhappy with a service that has lurched from issue to issue over a very extended time frame. The posts were directly relevant as they picked up on points made by a director of the firm.

 

So is it appropriate that a brand with so many issues and so many unhappy members continues to sponsor and advertise on EduGeek - you carrying their name and advertisements can be seen to be a recommendation/endorsement?

[SchoolsBroadband]

May I ask if there is anything that identifies the subset of customers that were affected. Were they using a setting in common or is it just a random subset of customers.

 

Hello @BrotherSidious,

 

the core issue of the webadmin slowing down will have affected everyone at that time when the issue happened in the specific way of slow loading times of the webadmin user interface and some user auth issues. The auth issues though were random depending on when the request came into the webadmin server and how SQL then essentially put them "in the queue" to be processed. The issue was that the queue was filling up and requests would either get queued and processed very slowly or there would be no space for further connections.

 

Authentication issues wouldn't have affected everyone when using Wagent because the Wagent has a timer where i re-sends its users credentials back to the Webadmin in a set period of time. If the issue with MY-SQL was manually resolved prior to the Wagent re-send then the end user wouldn't have seen any issue.

 

However, when SQL essentially failed and new requests couldn't be processed it had a knock on effect to the NS Proxy service which was very random. I believe that when NS Proxy tried to pull configuration from the Webadmin when it had its issues that in certain situations it would crash NS Proxy causing filtering to fail and block any new filtering connections on that specific inline proxy server (i.e. not everyone). We essentially had to play 'whack a mole" and restart that server when the issue happened. Customers would be re-routed to other in-line filters but would experience a small period of time where the load balancers would recognise the server was down and send traffic to other physical servers.

 

Netsweeper identified this and created a patch which was rolled out to NS Proxy early last week which meant that we could recover that service much quicker but it didn't fix the underlying database issue and at that time Netsweeper hadn't identified that the database was the issue. Therefore the Webadmin still suffered issues where the database would lock and not allow any new connections to it once its queue filled up.

 

Netsweeper changed a specific table within MY-SQL (and excuse me but i'm not a database expert) from one type to another which allowed row locking rather than table locking as well as a number of other tweaks to MY-SQL. This appears to have fixed the issue and since last Thursday evening we have had no recurring issues.

 

Unlike with our previous filtering vendors, Netsweeper had dedicated people on this since we first reported the issue. I know they have worked day and night to sort it in conjunction with our engineers and although it has taken much longer than either party wanted to find the issue and issue a fix this has not been on lack of effort on Netsweepers part. I personally had a teams video call with their top developer, their CEO, other execs and Netsweeper and our technical director every 48 hours to get the latest and an update on what Netsweeper were doing to fix the issue. The fixes they've made will be rolled out to future version of NS Proxy and i'm sure that any other of Netsweepers customers using a high number of Wagents will also have changes made to their databases / setup.

 

To be 100% clear we buy the software, support and setup / management of the SQL and Linux operating system from Netsweeper and make no changes to this except where instructed to by Netsweeper. We provide the server hardware, load balancing and network architecture around it.

 

Therefore to summarise, the webadmin slowdown issue will have affected all customers but this did not stop filtering working directly. Any other issues happened randomly / sporadically after the webadmin slowed and NS Proxy got into a specific state because it could not poll the webadmin directly.

 

Our official communications and being put together and will be much more in-depth than the above.

 

To be clear I am certainly not throwing Netsweeper "under the bus". They could have shied away from these issues and very easily blamed us but they took immediate responsibility and attempted to resolve the issue. Ok it took much longer than either party wanted but unfortunately we can not turn back time.

 

We are Netsweepers biggest Wagent users in the world which is why this issue had not been previously seen with other customers.

 

Going forward we are working with Netsweeper to see if they suggest a re-architecture of our platform as it continues to put on more customers although the underlying MY-SQL issue should not happen again. Currently though this looks like it may not be required.

 

I'll leave it at that for now I think. If anything slightly differs slightly from our / Netsweepers official answer then I apologise but I am not a technical architect and have interpreted what I've been told over the past couple of weeks into something most people can probably understand.

 

Thank you to all of our customers for your patience.

 

Dave

[Primus]

Who said it was Mr Tindall that reported this?

 

 

 

Being civil is special treatment? How?

 

This forum was created to help. Logic suggests if you are having an issue to state the issue, if you have a fix, state the fix.

 

We all face frustration with so many things, and it can be hard walking a line of getting your message across and being polite, but at the end of the day we are all human and having a go at someone that is trying to help you in a public place isn't very professional now is it?

 

Cleary being civil isn't special treatment and it is clear I didn't mean that it was.

 

SB have had major and prolonged issues for years now, they then went entirely silent on here on the matter. Every time they have issues they blame a third party - their line is always our network is superb, it wasn't an issue with our network etc.

 

At no point has the discussion lacked civility or professionalism, indeed if you read the thread you will see that schools are now having to take legal advice on rescission of the contract due to the significant issues.

 

I do also think it is entirely proper that their sponsorship and the associated kudos and associated recommendation/endorsement that comes of being a sponsor of EduGeek is questioned.

[caffrey]

I got blamed for a lot of things which turned out to be SBB, wasted a huge amount of time on them too

 

Got another one now which I've just about given up on, support isn't the quickest

 

Sorry if I'm whinging but I don't really have another soundboard

[Wubbalubbadub]

I do think that it's important to note that @SchoolsBroadband are using technology that not many others are. Aren't they one of the few, if not the only one that offer in-line filtering from Netsweeper?

 

Also, from someone who looks after 12 schools, having to deal with a variety of ISP's, Schools Broadband is the only one where i can "usually" phone up and get a change put through there and then, or even do it myself. This is a god-send when i get a phone call at 8:00am from a phone engineer wanting me to unblock some ports for SIP, which no one told me about!

 

RM - takes on average 3 weeks per change - I genuinely have to tell the engineer to head home and wait for us to get back to them!

EMPSN - 3-5 days, sometimes quicker if you get the right guy.

 

They are also the only ISP that offers service credit for outages. So just make sure you are submitting them when you do have issues. Although i don't think it's really what motivates the teams at Schools Broadband (having met them), the service credit requests will be some more motive to make sure it remains stable.

 

Matt

[BrotherSidious]

When your the one on site, everyone looks to you for resolution. Fortunately our staff were supportive of the fact that we couldn't do any more than we were doing but it was frustrating for both us and them. The service is now stable for us and I appreciated the post above from @SchoolsBroadband as it does shed some light on the situation and does give me information to pass up the chain so to speak. Hopefully things are learn't from this experience such as the importance of a daily update, the acknowledgement of support tickets and followup of those tickets, this forum should not be the place where customers feel they have to go to get support. I appreciate that Dave Tindall takes the time to monitor and reply to posts here but it must be a concern that people feel this is now their routine place to have to come rather than a place of last resort which indicates some overhaul of the SBB support process is needed. I would encourage others to post up constructive comments in relation to improvements they would recommend so that as SBB look into ways to improve after this incident they can not only take into analysis of their own procedures and policies but can also take not of concerns and suggestions from customers. Ultimately you may choose to stay with SBB, look to leave early or choose not to renew as that is your choice and you are entitled to your opinion on it but for anyone staying with them for any period of time, there is no harm in putting suggestions up here in the hope that things can be improved going forwards. Personally, I would like on online status page where we can see whats happening and daily updates could be posted when issues such as this occur as I think that would fundamentally improve communication and even when there are outages, we would all be less in the dark about them which for me would be a good start.

[caffrey]

Luckily they are quite understanding here, but I don't like T&L being disrupted and it's frustrating for me not being able to fix anything myself. With the previous solution we used I could fix things in minutes

[SchoolsBroadband]

When your the one on site, everyone looks to you for resolution. Fortunately our staff were supportive of the fact that we couldn't do any more than we were doing but it was frustrating for both us and them. The service is now stable for us and I appreciated the post above from @SchoolsBroadband as it does shed some light on the situation and does give me information to pass up the chain so to speak. Hopefully things are learn't from this experience such as the importance of a daily update, the acknowledgement of support tickets and followup of those tickets, this forum should not be the place where customers feel they have to go to get support. I appreciate that Dave Tindall takes the time to monitor and reply to posts here but it must be a concern that people feel this is now their routine place to have to come rather than a place of last resort which indicates some overhaul of the SBB support process is needed. I would encourage others to post up constructive comments in relation to improvements they would recommend so that as SBB look into ways to improve after this incident they can not only take into analysis of their own procedures and policies but can also take not of concerns and suggestions from customers. Ultimately you may choose to stay with SBB, look to leave early or choose not to renew as that is your choice and you are entitled to your opinion on it but for anyone staying with them for any period of time, there is no harm in putting suggestions up here in the hope that things can be improved going forwards. Personally, I would like on online status page where we can see whats happening and daily updates could be posted when issues such as this occur as I think that would fundamentally improve communication and even when there are outages, we would all be less in the dark about them which for me would be a good start.

 

Hi @BrotherSidious,

 

we do actually have a new portal on the way that will do just that. ETA 4 months i believe.

 

@MJTayloronline, we are behind on cases at the moment due to the backlog from the issues we had the other week. The NOC team are mucking in and I expect us to be back to normal in 2 weeks.

 

Thanks

 

Dave

[snagrat]

Although I started this post, I still have positive things to say to our customers who use the Schools Broadband service.

 

With the Fortinet and Netsweeper solution I feel we have a very good control over the whole network.

We have had some Shared Lists created so can easily allow/deny sites for either individual or groups of schools very very quickly.

 

The support at times could be quicker but today I had an issue with clients not using the correct group and turns out it was an incorrect client name on my part. Within minutes they fixed it and everything instantly worked as expected.

Hopefully with the Netsweeper issue resolved the amount of support tickets will reduce and the team can concentrate on new tickets.

[andy_b]

"Request Log Files" useless again.

[caffrey]

Oh is that why I'm struggling to troubleshoot something...

[Wubbalubbadub]

I have just logged a call and been told they are currently looking at the issue. Hopefully wont be long!

[andy_b]

I have just logged a call and been told they are currently looking at the issue. Hopefully wont be long!

 

since at least yesterday. Can't recall when I last looked before then.

[RobMason]

Good Afternoon All,

 

Hopefully you all read the roadmap that was released a couple of weeks back, resolving the outstanding issue with the log feed is now priority one after a January without the issues from last year. We initially planned to get this resolved in September when it initially raised its head but the webadmin problems ended up taking precedence. Please rest assured that logs are not being lost, it is just the live feed that is delayed and can timeout.

 

My engineers are currently working on the problem with Netsweeper and we will have both a fix and increase in report performance soon.

 

Thanks,

[Edu-IT]

I never saw the RFO from the last Netsweeper big issue, where they got involved to create patches. Was this posted?

[caffrey]

Why do I feel like I'm taking part in a beta ?

[Edu-IT]

What's the current service issue affecting DNS?

 

Edit: Power surge / Power issues at DC.

Edited January 29, 2020 by Edu-IT

[SchoolsBroadband]

hi @Edu-IT,

 

please see below. We have an issue in a little used data centre although it used to house our main DNS servers in part. If you haven't changed your DNS servers (which we asked everyone to numerous times about a year ago and repeated this weekend) then your DNS may be broken.

 

Please following the below steps which we've just sent to all of our customers again.

 

Thanks

 

Dave

 

 

Potential Outage 29th January 2020

 

Good Afternoon,

Please be advised that one of the datacentres used by Talk Straight is experiencing power issues. We have just dispatched an engineer to the datacentre to repair a switch that has failed following a power surge.

Some customers may be experiencing DNS resolution issues if they have not followed the steps provided in previous customer advisory notifications about the retirement of one of our old DNS forwarders. If this is the case can customers please ensure that they are no longer using 85.92.168.103.

To confirm the correct two servers to use are 85.92.188.226 and 85.92.168.104

Edited January 29, 2020 by SchoolsBroadband