SchoolsBroadband

hi @Edu-IT, please see below. We have an issue in a little used data centre although it used to house our main DNS servers in part. If you haven't changed your DNS servers (which we asked everyone to numerous times about a year ago and repeated this weekend) then your DNS may be broken. Please following the below steps which we've just sent to all of our customers again. Thanks Dave Potential Outage 29th January 2020 Good Afternoon, Please be advised that one of the datacentres used by Talk Straight is experiencing power issues. We have just dispatched an engineer to the datacentre to repair a switch that has failed following a power surge. Some customers may be experiencing DNS resolution issues if they have not followed the steps provided in previous customer advisory notifications about the retirement of one of our old DNS forwarders. If this is the case can customers please ensure that they are no longer using 85.92.168.103. To confirm the correct two servers to use are 85.92.188.226 and 85.92.168.104

Morning @sadams1980 we've been looking after Gloucestershire and Somerset schools for around 5 years. More than happy to do you a quote for connectivity, with our hosted Fortigate UTM security and Netsweeper filtering offering. It's all cloud based and virtualised so there's no need for any kit on site bar a router and you can easily grow into more processing power as your line increases over time without having to change any kit. We also do MPLS / VPLS WAN as standard as well so if you need to connect to others schools then there's no additional charge or hardware required for that. Finally we should be very competitive on price, particularly with Virgin as we're one of their largest wholesale partners. Feel free to PM me and we can get your local account manager Rob to pop over from the Forest of Dean to see you. Thanks Dave Thanks Dave

We use virtualised Fortinet and Netsweeper combo hosted in our data centres. The good thing over onsite boxes is that there's multiple of them and more importantly you can expand over 1Gbps without having to buy a new box. We take care of all the licensing etc and give you your own dedicated virtualised instance. Our largest inidivdual customer pumps about 2.5Gbit through it each day. Thanks Dave

Hi @MrWu sounds like a plan. Remember if you're a secondary school you can also logon to your firewall and make this change yourself too although we're still happy to do it

Filter I believe and yes all customers. @RobMason sent me a WhatsApp after I messaged him tonight. He will be on to explain shortly. Every now and then google add more URLs to their services which often filtering providers then need to white list for Google services to continue to function correctly. I believe its been changed for all customers. Thanks Dave

Hi folks. Google changed something again. All should be working now. If not please send me or @RobMason a pm. Thanks Dave

Shouldn't be. Things are generally quite quiet and case load is back to normal. Suggest you give the team a call to chase and if need be ask foe team leader which for you is Michelle Miller who will escalate for you. Alternatively pm me your case reference and I will speak to her for you. Thanks Dave

Fortinet charge for it. It's the same price whether a physical fob or a software app. Dave

Yep. You can lock it down as much as you want. I'd suggest 1. Use remote ssl vpn software 2. Tie authentication into windows ad for user username and password for simplicity purposes 3. Create a vpn use group in ad which then only allows users in that group to be allowed to authenticate via vpn 4. Use 2 factor authentication via a mobile software app (note there is a charge per physical fob or software app deployment for two factor authentication. 5. In the policy only allow vpn use access to the required ip addresses and ports. 6. Consider use of a DMZ although that may be going a little ott if you do the rest above as could complicate your rule sets but is ultimately the best thing to do. 7. Really go to town and setup Ips/ids and inline anti virus scanning on that profile to check for malicious traffic which could infect rdp or any other service/ port which is accessible via vpn Theres more things on top of that you can do but doing the above is pretty strong. Dave

it's better to put it in a DMZ so that if it were ever compromised then it either can't access your LAN or the ports / IP's it can do are restricted. Your Fortinet firewall comes with IPSEC and SSL VPN. I'd recommend you use that rather than allow RDP straight through from any IP. You can also do 2-factor authentication for even better security. If you're with us then do send me a PM and I'll get our pre-sales tech to advise. Thanks Dave

@Pyroman I'm surprise we've come out as expensive. Perhaps you're on one of our old prices as they've reduced quite a bit from 3-4 years ago. We've already a number of schools in and around Rotherham. I believe we can also do Dark Fibre from CityFibre in that area too as well as GPON services in the future. Thanks Dave

Hi @BrotherSidious, we do actually have a new portal on the way that will do just that. ETA 4 months i believe. @MJTayloronline, we are behind on cases at the moment due to the backlog from the issues we had the other week. The NOC team are mucking in and I expect us to be back to normal in 2 weeks. Thanks Dave

Hello @BrotherSidious, the core issue of the webadmin slowing down will have affected everyone at that time when the issue happened in the specific way of slow loading times of the webadmin user interface and some user auth issues. The auth issues though were random depending on when the request came into the webadmin server and how SQL then essentially put them "in the queue" to be processed. The issue was that the queue was filling up and requests would either get queued and processed very slowly or there would be no space for further connections. Authentication issues wouldn't have affected everyone when using Wagent because the Wagent has a timer where i re-sends its users credentials back to the Webadmin in a set period of time. If the issue with MY-SQL was manually resolved prior to the Wagent re-send then the end user wouldn't have seen any issue. However, when SQL essentially failed and new requests couldn't be processed it had a knock on effect to the NS Proxy service which was very random. I believe that when NS Proxy tried to pull configuration from the Webadmin when it had its issues that in certain situations it would crash NS Proxy causing filtering to fail and block any new filtering connections on that specific inline proxy server (i.e. not everyone). We essentially had to play 'whack a mole" and restart that server when the issue happened. Customers would be re-routed to other in-line filters but would experience a small period of time where the load balancers would recognise the server was down and send traffic to other physical servers. Netsweeper identified this and created a patch which was rolled out to NS Proxy early last week which meant that we could recover that service much quicker but it didn't fix the underlying database issue and at that time Netsweeper hadn't identified that the database was the issue. Therefore the Webadmin still suffered issues where the database would lock and not allow any new connections to it once its queue filled up. Netsweeper changed a specific table within MY-SQL (and excuse me but i'm not a database expert) from one type to another which allowed row locking rather than table locking as well as a number of other tweaks to MY-SQL. This appears to have fixed the issue and since last Thursday evening we have had no recurring issues. Unlike with our previous filtering vendors, Netsweeper had dedicated people on this since we first reported the issue. I know they have worked day and night to sort it in conjunction with our engineers and although it has taken much longer than either party wanted to find the issue and issue a fix this has not been on lack of effort on Netsweepers part. I personally had a teams video call with their top developer, their CEO, other execs and Netsweeper and our technical director every 48 hours to get the latest and an update on what Netsweeper were doing to fix the issue. The fixes they've made will be rolled out to future version of NS Proxy and i'm sure that any other of Netsweepers customers using a high number of Wagents will also have changes made to their databases / setup. To be 100% clear we buy the software, support and setup / management of the SQL and Linux operating system from Netsweeper and make no changes to this except where instructed to by Netsweeper. We provide the server hardware, load balancing and network architecture around it. Therefore to summarise, the webadmin slowdown issue will have affected all customers but this did not stop filtering working directly. Any other issues happened randomly / sporadically after the webadmin slowed and NS Proxy got into a specific state because it could not poll the webadmin directly. Our official communications and being put together and will be much more in-depth than the above. To be clear I am certainly not throwing Netsweeper "under the bus". They could have shied away from these issues and very easily blamed us but they took immediate responsibility and attempted to resolve the issue. Ok it took much longer than either party wanted but unfortunately we can not turn back time. We are Netsweepers biggest Wagent users in the world which is why this issue had not been previously seen with other customers. Going forward we are working with Netsweeper to see if they suggest a re-architecture of our platform as it continues to put on more customers although the underlying MY-SQL issue should not happen again. Currently though this looks like it may not be required. I'll leave it at that for now I think. If anything slightly differs slightly from our / Netsweepers official answer then I apologise but I am not a technical architect and have interpreted what I've been told over the past couple of weeks into something most people can probably understand. Thank you to all of our customers for your patience. Dave

Their UK sales director was on here but he has recently left ans gone back to Bermuda of all places where he was originally from. I'll ask our new account manager or some of the tops guys in Canada. Dave

Hello everyone, official communications will be sent via the normal channels soon. On Thursday evening Netsweeper installed a significant patch which has brought back stability to the Webadmin. This was mainly a major reconfiguraton of MY-SQL although numerous patches have been made to NS Proxy as well as a new design for Wagent off loading to alternate servers implemented. It has taken Netsweeper a significant amount of time and effort to find the underlying issuing but they have been working very hard and have finally done so. We certainly thank them for working as hard as they have trying to get back to where we were previously. To be clear, this has not been an issue with Schools Broadbands hardware or our load balanced platform. This has a huge amount of capacity available in hardware and throughput as we are nowhere near touching as far as this can go. Nobody (including Netsweeper) have been happy at the amount of time it's taken to get to the bottom of the issue and there have been lessons learnt as well as an apology from both of our companies. Although the issue hasn't affected all of ours customers, one affected customer is one too many. We may put together a technical webinar for those of our customers that wish to understand the technical intracicies of what happened and how everything was fixed if there is demand. The issues raised were not total outages of our solution but generally small periods of instability or features not working in full due to a knock on effect of the primary issue with the Wedamin. We are currently continuing to monitor the Netsweeper platform before both Netsweeper and Schools Broadband sign off that the fix has 100% worked and will update our customers in due course. Thank you all for your patience. If you've any specific questions please send me a PM. Thanks Dave

Fortinet with Forticoud is good and complies fully with DFE guidelines. Happy to quote if you want to send me a PM? See https://login.forticloud.com/samlsplash?redirect_uri=https%3A%2F%2Fwww.forticloud.com%2Floginmgrlogin&response_type=code&service_id=forticloudglobal-4TCPXs1dLST1bRp2UuTmkffruyUBvpSS.forticloud.com&access_type=offline and https://www.fortinet.com/products/next-generation-firewall.html Dave

I'm not sticking the boot in. I said I like their product just not their support from the experience we had, others may of course differ. I wish them well in the future. Dave

Hi @RichardWhitelaw, yes we include the Chrome agent FOC in our package and we've quite a few clients using it. Send me a PM if you need some case studies. Dave

Netsweeper have a Chrome agent which can sit on Chromebooks and do all the auth and filtering you need. I know we use this as standard for our customers but I'm not sure about LGFL. Best ask their helpdesk but the functionality definitely exists in Netsweeper. Dave

I'll PM you both. Dave

I may be wrong but dont you also have to use their uso and other auth tools and run it their way rather than how you may want to? Someone from lgfl can confirm I think. Dave

PM'd Tracey. Dave

Hi Steve, as per my PM please send me through the case reference. We are not aware of any global filtering issues affecting all customers, just the Web Admins. I'll then get a NOC engineer to advise. Thanks Dave

Hi folks, firstly thanks for your patience and we're sorry for the impact it has on changing configuration We have an issue affecting the web configuration portal (not filtering) which Netsweeper are working on but they're currently struggling to create a permanent fix. I have another video call with their CEO, top tech and our Technical Director today to get an update on what the resolution is and why the issues are randomly happening. Netsweeper are doing all they can to fix the issue but it hasn't been resolved yet. We are very much in their hands at the moment but are doing all we can to assist. As soon as I have a positive update I will let you know. Thanks Dave

correct, we haven't we used their Rocket product Dave