SchoolsBroadband

Hi Tracey, that is something we generally do. I'm not sure why it wasn't in this instance though as the team do have a process to follow. We have spoken with Angela who is head of support who has assured me that won't happen again. Thanks Dave

Hi @caffrey, as per our recent notification we are having some intermittency regarding the live logs and immediate on demand reports. We believe this to be Netsweeper software related and we're working with them on fixing the issue. It's likely this is also related to the underlying issue we have had of late regarding stability. That said Netsweeper have stabilised things but to the detriment of the live logs. The reason is rather complex to say the least and Netsweeper are still investigating. I have another call with their CTO and CEO on Friday although they are engaged with our team daily at an engineering level. Thanks Dave

Afternoon everyone, one of our engineers has been speaking with Tesco today regarding why pages have been denied when using our service. They basically blacklist IP addresses which send a lot of requests and as we use some specific IP address for our proxy servers then they assumed we were attacking them and automatically blocked us. We of course weren't attacking them more people just trying to do some shopping We've now had our IP's whitelisted. We don't anticipate any further issues but if we do we've the contact details and a specific reference number to our service with an engineer at Tesco security. This may also be helpful to other people who have similar issues using another ISP. Thanks Dave

A very simplistic test can be done below. Does your ISP install a managed router that sits in front of the Smoothwall box? If so why not do some constant pings from inside your network to the external IP from your router and then also from another Internet connection send constant pings to your ISP's router. That way you can tell which bit of the kit (i.e. smoothwall or router) could be dropping the packets. Good luck Dave

I'll check with the team on monday @snagrat. New loggers have gone it last night so should fix delayed logs issue people have been seeing and hopefully all issues we've had. We will be monitoring closely with Netsweeper on Monday. Dave

nowt like a bit of relevant advertising and useful advice@Wave9_Lee that said all of our customers have SSL VPN access FOC as part of their packages with Schools Broadband. If anyone needs it setting up please let the support team know. Dave

Yes new loggers are going in tonight. We believe it should fix the delayed logging issue. Also we deployed a patch for the google dir sync temporary issue since the platform was upgraded to Netsweeprs latest release. Theres also a new version of wagent which allows local directory sync rather than copying from your DC to the webadmins. This gives a better user experience with live lookups. We'll send comms out on that soon but you can ask our support team for the latest Wagent version now. Thanks Dave

That's a very good point you make @DrCheese. We do actually have that on the potential development road map. It wasn't an option when we started using Netsweeper for various reasons but could be in the future. Dave

Thanks @Opendium_Steve, the Google authentication bug which has occurred since the software upgrade should be patched this evening. Netsweeper are finalising that through their QA procedure. They've also made a change around logging which is part of the problems we have seen of late and we have been fine today having gone through the half eleven busy period. We are working to their recommendations and we may bring some new logging servers live if they deem that necessary as they are already in our racks and have been ready to go for some time. We believe issues relating to live logging and a crash of the NS Daemon are logging related. Netsweeper are unsure at the moment if this is software or hardware related but are still working on this. So far today has been good (bar Virgin having a major outage on the South Coast see https://www.ispreview.co.uk/index.php/2020/03/major-virgin-media-broadband-and-tv-outage-on-south-coast.html) Thanks Dave

As much as we've had issues with Netsweeper at least they try to fix things and engage with us. I cant knock them for their effort at all. They're also very different solutions. Our old vendor was their own hardware and closed shop for us doing or monitoring anything on it other than the web gui which was very limited from a system point of view. Number 1 priority is to fix things so our customers get a consistent reliable service as they have done on netsweeper in the past. I'm sure that will be the case again with the help of Netsweeper. I do take offence at thinking our engineers are constantly at fault and we dont tell the truth as that's important to us. We have 100% uptime on our other services. I met a customer that I wont name at BETT this year who I'd never met before who thanked me for our candour, openness and honesty on here whether good or bad and the fact we engage with you all. We will continue to do so and will have the issue fixed with the help of Netsweeper. Thank you to that customer in particular as you did say how much you appreciated our service and despite the occasional bump in the road have done for many years despite other more vocal people on here who are not our fans. That's it for now I think. I'll be in touch with an update when I have it from our or Netsweeper engineers. Dave

ok so lots of conversations with Netsweeper this evening. They're making some software changes to cache's on each inline server tonight and monitoring. Irrelevant of the outcome of that we have also been asked to potentially upgrade the hardware our loggers reside on. This is something we were going to do anyway to increase capacity and the servers just need to be brought live as they are already in our DC. If we do that it will be done on Friday evening. Thanks Dave

Hi @Edu-IT, when I originally replied it was 1 but that changed over time. All hardware for the inline proxies are exactly the same. Netsweeper have some more thoughts at the moment and we're working with them this evening. I will update in due course once they suggest which option we go with. To the other replies when we first brought Netsweeper on we had no issues either up until November(ish) time with nearly 18 months of perfect service (bar a few very minor niggles). We are one of the biggest users of Netsweeper in the UK. Our service is slightly different to other LA's as we can use the same internal IP address over multiple sites (Netsweeper wrote this for us) but bar that we are also heavy users of Wagent which also interacts with this and looks after authentication. We don't have an exact replica of what other LA's do. That said Netsweeper asked us to put in certain hardware (network and servers) which we have so we haven't gone against what they recommended we put in. We'll continue to do this on their advice too. Watch this space. Thanks Dave

Evening folks, our Cisco load balancers are absolutely fine. We have seen today a failure of the NS Proxy Daemon on numerous servers at random times. Unlike seeing a physical hardware failure this doesn't as standard pull the affected proxy server out of the pool and we have had to manually restart the daemon on it rather than the server itself. We believe this is a Netsweeper software issue again and we suspect a fallout from the software upgrade which was performed a couple of weeks ago. We are also seeing an a bug affecting authentication which Netsweeper are releasing a patch for and we should have imminently. Our engineers have been speaking with Netsweeper in Canada all afternoon and we will continue to do so tonight. I also have a call with their CTO and CEO tomorrow late morning to discuss. I will have some sort of update from our NOC team tonight although I suspect they will work into the very early hours with Netsweeper. Sorry for the blips in service today. We're working with Netsweeper to do all we can to make sure they don't reoccur tomorrow. Dave

We use a Cisco load balancing devices and protocol to send schools traffic to a specific transparent proxy server out of the pool of servers we have. In the event of sever failure schools connected to the broken server are then re-routed to another server from the pool and the original one automatically taken out. Dave

It appears that the netsweeper daemon died on one server. A restart fixed it. Netsweeper are investigating why it happened. This only affected one proxy server out of the many we have. Now fixed and working again in a few minutes. Sorry for any inconvenience caused. Thanks Dave

Evening, possibly is the answer to that. If you can ask for a claim by going through the correct process we can investigate if it did affect you or not and in what capacity. Certainly we never try to "worm out" of paying SLA compensation claims as we pay what our SLA says and always will. Dave

We can try contact Yahoo's postmaster. We often find though that these emails go into a big void and we don't get a response.... if you can send me a PM with your IP and school name we can try contact them. Can I suggest you also do the same? Thanks Dave

Hi all, there wasn't an outage for all of our customers. From our initial investigations it affected some customers, some of the time. Even if it affected a customer we believe it didn't affect all users on site. We have our theories on this but we're waiting on an official response from Netsweeper. As soon as we have it we'll create the RFO and send out to everyone. Our SLA (where compensation can be claimed) is on a complete outage for a specific site. Don't get me wrong, we don't want anyone to have a partial outage but this is what our definition is and our SLA compensation goes over and above the industry norm. Dave

Hi @caffrey, an update will be out later today. I suspect this will be holding update as we are still waiting to hear back from Netsweeper ourselves. Thanks Dave

Morning all, the changes we made in conjunction with Netsweeper appears to have fixed the issue. We are not seeing slow speeds over our estate in general. if you do have any speed issues please raise with our support team. That said I'll PM @caffrey and @PrimaryNetMan as if you are having a common issue I can take that to the NOC team to see if there's anything wider but we're certainly not seeing that at the moment. Thanks Dave

Hi @Edu-IT, Quite a few different things and I meant that as a generic term. So change in config, change in underlying code etc Thanks Dave

Good evening everyone, Sorry for my tardy response. We appear to have had an intermittent issue for some customers following a Netsweeper software upgrade performed early last week which initially appeared to go fine but has only become apparent today following heavy usage after the holidays. We have worked with Netsweeper all day and are continuing to do this evening as we make some recommended changes as advised by Netsweeper. We will continue to monitor throughout the evening and early morning when traffic volumes return to normal daily use. We have a live call with Netsweeper senior engineers at 08:30 who will be monitoring the platform to check if the changes have fixed the underlying issue. It's too early yet to fully comment on the fault but we belive it's down to a software alternation different from our previous build. We will update everyone in due course following monitoring tomorrow. Thanks Dave

No, LGFL use Netsweeper (same as us (Schools Broadband) RM use their own in house solution which is Safetynet or sometimes Smoothwall I think. Dave

@DalekSec, that is indeed frustrating. I've emailed my contacts at Netsweeper to ask for an update. If you don't auth by Wagent then we can go to a backup Captive Portal which will mean you can still be filtered on a per user basis just users have to put in their username and password. We could also filter by IP address if we gave those MACs a static DHCP lease too. I'll wait for Netsweepers response and will get back in touch. Thanks Dave

Big fortinet fans and partners here. Take fortianalyzer as well and you get a bucket load of reports as well as the ability to customise your own Dave