SchoolsBroadband

Hello everyone, sorry for the issues today. You should all have received updates throughout the course of the day. This issue affected 3 of our current 16 proxy servers and customers connected via them (not to anyone else or any non HTTP and / or HTTPS service) and was due to a bug that was found in a patch applied yesterday. This patch had already been rolled out 3 weeks ago to another server with no issues so we rolled out to all other ones yesterday. The patch was put in place to allow us to use some new reporting infrastructure we are working on which will vastly improve the reports available and the time it takes to create reports. The issue appears to have been that if a reporting entry was an empty file then it would force the NS Proxy service to crash and re-start (hence the intermittency). Of the millions of entries we log for content filtering reporting I've been told today we had a number in the teens that were empty and unfortunately made NS Proxy crash and those servers thus forced closed (stopping web browsing) rather than forced open (allowing access to anything). A temporary fix has been applied and a patch is being rolled out this evening. Thanks Dave

Hi @TMBS, there are some specific settings you need to put in for our remote proxy. Our support team will be able to give you the correct settings. in answer to @caffrey, yes we do offer offsite filtering free of charge to our customers via this method. Thanks Dave

Hi @j_mcg we look after a couple of thousand schools Internet, content filtering and security. Netsweeper is our filtering provider of choice we use in our cloud hosted infrastructure so no need for numerous onsite boxes. You have full access to change things yourself if you wish or you can take a fully managed service. You can have filtering and reporting via multiple authentication sources which can be done on a group or user as well as an IP, IP range or VLAN. We also fully support BYOD and offer full captive portal integration as well as radius. On top of that we also use Fortinet firewalls which we can also give you full control off. The only device you need onsite is one of our routers, everything is in the cloud. Happy to give you a quote demo. Dave

Just to add on to Mark's and others comments but another thought you may or may not be aware of is that in January 2021 BT Openreach are launching a symmetrical FTTP service. i.e up to 1Gbps down and 1Gbps upload. Furthermore, we have started to deliver EoFTTP (Ethernet over FTTP) services via BT (we've done over 1,000 EoFTTC in the past). EoFTTP gives a certain guaranteed bandwidth on FTTP based circuits. They are delivered over different cablelinks at the BT Exchange so your bandwidth isn't mixed with normal residential backhaul traffic (i.e. a normal FTTP circuit). It runs over the same BT Ethernet network (i.e. leased line network) so speeds can be guaranteed. In January, you can buy for example 200Mbps guaranteed with 1Gbps burst. You can even go the full hog and buy a full 1Gbps guaranteed bandwidth. Although there's some small technical differences between that and a BT Openreach EAD based leased line these circuits are perfect for primary schools in particular where available. A full 1Gbps EoFTTP circuit will be less than the equivalent 1Gbps EAD circuit which is good news for everyone. As of today the maximum speed on BT Openreach based FTTP is 1Gbps download and 220Mbps upload but come January that's all going to change. Looking into my crystal ball I also see significant reductions in 10Gbps bearers on EAD based circuits, the death of 100Mbps circuits and 1Gbps bearers being the norm for all. BT Openreach have for the past couple of years installed 1Gbps capable kit whether you take a 100Mbps bearer or 1Gbps bearer. To upgrade the carrier asks Openreach to perform a software upgrade and wahey the link goes from 100Mbps to 1Gbps. Feel free to comment or PM me if you've any questions. Boom enjoy Dave

Hi @neonetman as a Fortinet house we think the Fortigates are the best when it comes to Firewalling and UTM. There filtering is pretty damn good too but for proper reporting purposes we'd suggest adding Fortianalyzer on top of it too. We generally advise customers to use our hosted solution. You still get access to the interface of the firewall just as you would do if you had a box onsite but you also get 2 firewalls in HA mode rather than having one on site. Do send me a PM if you'd like to discuss further. @synaesthesia, there's been a lot added to the v6 of Fortigates operating system over the old v5 you used to use with us. We've now upgraded everyone to v6.2.x There's a lot more functionality and a much cleaner interface too suggest you take a look with your primary schools if they have their own VDOM. Thanks Dave

We had this problem with BBC a long time ago when we bought some IP addresses from Romania and then re-registered them as UK. BBC uses (and i forget the exact name of the service) a geolocation IP service by a third party which takes an age to update itself to recognise the IP addresses have moved location. I'll ask my NOC team what the service was they used. I think we also got in touch with the BBC NOC team. It took about 4 months to fix which was very frustrating. Ultimately your ISP needs to sort it out with BBC and track down the geolocation IP service provider they use and hound them to fix it. Unfortunately there's little you can do yourself. Sorry I can't be more helpful. Dave

Hello, we can do this for you with varying data limits and a real world IP. Do send me a PM if you're interested. Regards David

Hello. We do this either via our 4g connections at a network level or we can do it using any isp via proxy at captive portal but yes there is a per user charge for none schools broadband customers. Dave

Hi @Sonic007, there's a few things you can do to improve speeds. 1. take 2 x FTTC connections and use load balancing / SD-WAN from an onsite device. This will effectively double your speed 2. Check for FTTP (do send me your school details and i'll check for you). Please note you may be able to access a DFE grant to get FTTP installed free of charge. We've taken orders from over 50 schools around the country with installs fee's of up to £50,000 paid for by the DFE 3. Use an altnet such as Gigaclear or Virgin Media (we can use them and many others) 4. Use a leased line (£17,000 is a lot to install). We can check and see what the indicated install costs (known as ECC's) will be by doing a desktop survey and interfacing with BT Openreachs systems. 5. Look at a 4G / 5G solution. This can be done on its own or in conjunction with load balancing / SD-WAN (we have an unlimited filtered / firewalled 4G solution) Do send me a PM if you'd like me to look into it for you. Thanks Dave

Evening folks. Try http://www.speedtest.net and use our server (Talk Straight). We can also do iperf tests for you if you think your line is running slow. Remember when you do tests there is often background traffic of other things using your connection. That said, quite often some speed test sites simply won't work when testing a full gigabit speed slthough our speediest server can Dave

we're a Netsweeper house having moved from Lightspeed. We've moved a lot of customers successfully from Smoothwal. We also have Radius integrated and working fine. Netsweeper takes a bit of getting used to but once you know what you're doing it very capable. It integrates into Windows AD, Azure AD, Google etc for all the major auth sources you need. We can also do offsite filtering and even put into into 4g filtering via private APN so students can have exactly the same kind of access as they do in school no matter where they are. i've plenty of people you can speak to from single schools to large multi academy trusts who are successfully using it. send me a PM if interested. I think some LA's restrict what you can do with Netsweeper. We don't though. Thanks Dave

Thanks @Olliesaurus / @snagrat, yes Kulthoom is a superstar, that said we do have lots of other great engineers too. We've had a new head of support start as of about a month ago who brings a lot of experience from working in the ISP industry heading up support teams so this change is already having a positive effect which is good. He's called Paul Collier in case you ever need to speak with him and is an escalation point in our new customer services document which will be sent out soon i think. Thanks Dave

Morning folks, although most carriers were back online around half three, all carriers (Virgin in particular) were restored at around 11pm last night once Equinix had fully restored power to all parts of the data centre. As of then we were just about fully back to normal. Dave

Any carrier or ISP in that data centre on one of the 4 floors without power will have been affected (whether they have resilient power or not). I believe the following have been as I've seen notifications from them. BT, Talk Talk, Entanet, Virgin, Exa Networks, FluidOne, Virtual One, Vodafone, Origin, SSE, Cogent, Level 3, Vaioni, Giganet...... This will only be the tip of the iceberg though and many more will have had full or partial outages. Dave

we are seeing some carriers as down still (Virgin in particular) so it's likely their equipment is not yet back on. All of our kit is up and now powered but if the carrier we interconnect to doesn't have power then obviously their circuit won't be working. THis is happening for other ISP's too. I've seen comms about everything should be fixed from a power point of view by 9 o'clock this evening from the data centre. Regards David

Hi all, it looks like all of our services are coming back up and running. We need to check all kit has come back safely but we can now see connections back online. Apparently 2 of the 3 floors (floors 1 and 2) which had no power are now back online too. Hopefully none of our carriers are on floor 3...... We'll update everyone through the normal channels.... Dave

Morning all, Equinix LD8 has a major power issue that is affecting numerous ISP's including our normal competitors. Please see a copy of the comms we were sent this morning from Equinix. Equinix is one of the UK's major Internet hubs. IBX(s): LD8 IBX Address: 6/7/8/9 Harbour Exchange Square London E14 9GE United Kingdom Ticket#: 5-200094218580 Date and Time of Occurrence: 18-AUG-2020 04:40 Site Local Time Date and Time Update Reported: 18-AUG-2020 09:01 Site Local Time INCIDENT SUMMARY: Fire Alarm - IBX was Evacuated UPDATE: Equinix IBX Site Staff reports that IBX Engineers are currently migrating customers' supplies into new infrastructure. IBX Engineers will start to investigate the cause of the failure on UPS. The next update will be provided in approximately 30 mins. IBX(s): LD8 IBX Address: 6/7/8/9 Harbour Exchange Square London E14 9GE United Kingdom Ticket#: 5-200094218580 Date and Time of Occurrence: 18-AUG-2020 04:40 Site Local Time Date and Time Update Reported: 18-AUG-2020 08:34 Site Local Time INCIDENT SUMMARY: Fire Alarm - IBX was Evacuated UPDATE: Equinix IBX Site Staff reports that IBX Engineers and the specialist vendor have begun restoring services to customers by migrating to newly installed and commissioned infrastructure. IBX Engineers continue to work towards restoring services to all customers and further updates will follow when more information is available. The next update will be provided in approximately 30 mins. IBX(s): LD8 IBX Address: 6/7/8/9 Harbour Exchange Square London E14 9GE United Kingdom Ticket#: 5-200094218580 Date and Time of Occurrence: 18-AUG-2020 04:40 Site Local Time Date and Time Update Reported: 18-AUG-2020 06:44 Site Local Time INCIDENT SUMMARY: Fire Alarm - IBX was Evacuated UPDATE: Equinix IBX Site Staff reports that fire alarm was triggered by the failure of output static switch from Galaxy UPS system supporting levels 1, 2, 3, 4 in building 8/9 at LD8. This has resulted in a loss of power for multiple customers and IBX Engineers are working to resolve the issue. The next update will be provided in approximately 30 mins. IBX(s): LD8 IBX Address: 6/7/8/9 Harbour Exchange Square London E14 9GE United Kingdom Ticket#: 5-200094218580 Date and Time of Occurrence: 18-AUG-2020 04:40 Site Local Time INCIDENT SUMMARY: Fire Alarm - IBX was Evacuated INCIDENT DESCRIPTION: Equinix IBX Site Staff reports that a fire alarm has been triggered and the IBX was evacuated. IBX Engineers are currently investigating the issue. The next update will be provided in approximately 60 mins or earlier if available. Latest update is they have a new UPS on site and are installing that to bring customers back online. As to why this has happened we will find out in due course. Needless to say we take resilient power feeds to all of our cabinets so as you can imagine we're not best impressed. We've put out comms as usual and have been calling as many customers as possible. We don't have a resolution time as yet but we believe some services are starting to be restored within the data centre. Thanks Dave

Hi @Jonah, The we offer this by all the major carriers. Virgin and BT Openreach insist on putting their own boxes on the end of some point to point fibre and present either 100mbit, 1gbit or 10gbit and prices vary accordingly. Cityfibre offer the same and dark fibre so you can put your own optics on the end and run at whatever speed you want. If you send me a PM I'll happily give you a quote and make sure you use the most cost effective carrier. We've installed this type of solution for well over hundred sites already so have a lot of knowledge and installing and worming with the carriers. Other than that also check out installing your own fibre or a point to point wireless solution. Thanks Dave

Hi @sarge is this part of the Rural Gigabit Connectivity scheme funded by DCMS and the DFE? If it is then expectations will be very different to a leased line as there is normally a lot of installation work to do. The reason I ask is that we've just signed up a few schools via this scheme in Cheshire. Thanks Dave

Good afternoon everyone, I've just struck a very good deal with a specific carrier for 10Gbit leased line bearers and different bandwidths going through them. e.g. 1Gbit / 10Gbit, 2Gbit / 10Gbit 5Gbit / 10Gbit etc I'm looking for a customer requiring big bandwidths so we can get specific analytics from the usage and how the carrier performs. You can be an existing customer whose contract is coming to an end or a brand new customer we've worked with before. Whoever is kind enough to beta test / contract with us will get an absolutely whopping discount. Please PM me if interested. Thanks Dave

Hi @ITGURU, Most ISP's should be able to offer you a redundant connection over 2 routers via use of a floating gateway IP using HSRP or VRRP. If you contact your current provider I'm sure they can assist. Thanks Dave

I'd say that's now a bit high. We're under £7k for a full 1Gbit to most locations with no install costs whether you take our filtering or not. Dave

Morning all, the other Fortinet which hadn't had an issue in that pair was rebooted in the early hours of the morning and has successfully re-joined the HA cluster. We will continue to monitor the service and will update with an RFO to those affected shortly. Thanks Dave

Afternoon all, i'm on holiday today but from a couple of whatsapp messages i've had from one of our chief engineers it looks as though a software bug in the FortiOS caused HA to fail on a specific cluster pair of Fortigates. We run them in Active / Passive, Passive / Active mode (a hybrid of Active / Active) and those that were connected to one of them were not automatically failed over to the other firewall. We are investigating with Fortinet TAC but this looks like a software bug. We performed an emergency reboot of the device which forced HA to kick in and move affected clients onto the other firewall. Once the device rebooted it rejoined the HA cluster and took back half of the traffic of that pair of Fortigates. We are rebooting the other Fortinet device in that pair as a precaution this evening. The Fortigate that the issue occurred on had 630 days of uptime since it was last rebooted :-/ We'll wait to see what Fortinet TAC says and then issue the RFO via our normal channels. The Fortinets are normally pretty solid so this is surprising. The customers on the other side of the pair continued to function as normal. Our other cluster pairs of Fortinets were also not affected. In the background prior to this, we've already engaged with Fortinet professional services to upgrade all of our core Fortinet estate over the summer to a version of V6 from v5 so they'll be a host of features enabled as well as permanently fixing some bugs in the current FortiOS software revision. Thanks Dave

@DrCheese We've lots of metrics we look at other than traffic graphs. Libre is one of the tools we use to achieve this although we use other products as well. @snagrat We also do some HTTP traffic request monitoring to each server too so we can see if HTTP and HTTPS traffic is being passed specifically for the reason you mention. Again I'm not going to say exactly what we do and don't monitor on a public forum but it is extensive. You can see for yourself what Libre is capable of as a minimum. Thanks Dave