SchoolsBroadband

Yes we've sold Dark Fibre and Wave connections connections although it's currently outside the budget of most schools however whether we use Cityfibre or not it will be available and affordable nationwide as of December 2017 as Ofcom has forced BT to offer a wholesale Dark Fibre product. This will cost ISP's the same price as a current 1Gbit EAD or EAD LA circuit so things will change radically and we will be fully harnessing this product as and when available. Thanks Dave

If you're a secondary school i'd recommend having an FTTC backup connection. Alternatively if you're a primary school then get a backup ADSL / FTTC. We can put via a different carrier so it gives some diversity in the event of a primary link failure and it still does the same content filtering and security. Thanks Dave

Thanks @snagrat Sure the more the merrier. I can organise another event and invite another 20 or so. There's never quite such a thing as a free lunch but I'd be happy to offer it Thanks for your support. FYI The company @snagrat works for have used us for years and moved all their connections from another ISP over to us and have hardly had a problem in years. I certainly haven't paid or asked him to say the above! Dave

I certainly do see as we see the number of tickets we have for certain incidents, thats a rather sweeping and incorrect statement. We also speak to our customers and visit them too whenever we can. Hence my offer to show you just what we are doing and how we run our operation. if you do not wish to use our services then I have no issue with that whatsoever and wish you well with whoever you do use. i do have issues with people saying sweeping and incorrect statements when all they read is what has been posted on here by people not knowing what is going on. We deeply care about our customers and always try to go that extra mile. Sometimes things do go wrong and we are doing everything we can to fix those issues when they do. I think I'll leave it at that @Primus as I do not wish to get in an argument....

ok well the offer still stands if you'd like to re-consider? Again though although some customers have some issues at the moment the huge majority do not and have not been affected by any of the issues reported on this thread as some people have said. There is a lot going on in the background too which you'll hear more about in the not so distant future. Dave

@Primus I think I'd like to invite you to meet our teams in our offices and see what we are doing to solve some of the current issues and also how we cope with the many other ones which occur that nobody ever see's. We had about 20 network managers visit us over the summer to meet the people on the front line and in the NOC departments and really see how we work, the tools we use and how we pro-actively mitigate many issues. We are certainly not a whack-a-mole company and do a great deal of work which unfortunately most people just never see (as it should be). I'd happily ask anybody else that wanted to visit us to be our guests for the day too if you wish to send me a PM please do and I can get this organised as well as give some vendor accredited training for free too. Thanks Dave

Hi all, I'm getting some info second hand today as I'm off work as its my birthday. An issue has been identified on Talk Talks network at a certain location where they are seeing unexpected high throughput at certain times (likely a collector node). They are putting measures in place to mitigate this. I expect this to be resolved tomorrow. The issue is happening sporadically and not all the time so it's been harder for them to pin point. It's not happening to all customers using Talk Talk around the country only in a certain area. We'll keep you updated. David

Hello everyone. We definitely don't have an issue with our network that's for sure. We are investigating a potential issue with one of our carriers. This is not effecting all customers using that carrier but may be some in certain locations. We'll keep you informed here and by the usual channels. This is with the carriers escalations team and we are waiting on their input. Dave

generally yes. That said though from one particularly large MAT our SD department received a huge bunch of flowers and the 1st line team a huge box of sweeties for doing a good job. You know who you are so thank you very much

I think that will take some time to rebuild trust with those effected. We've plenty which are still happy of course that haven't had any issues whatsoever but those people generally don't jump up and down and say what a great service we give where as the unhappy ones do for different reasons. Another Fortigate cluster has been upgraded to the new FortiOS this week and the remaining ones will be done shortly too which remove the bugs which were in v5.2.8 Thanks Dave

Bob was meaning if there was a complete sustained failure over a very long period of time as per the other posts here our SLA's are very similar to the carriers as we back to back it with them regarding the connectivity piece at least. The SLA's we set for the Fortigate and Lightspeed are our own and we get no service credits from our suppliers should they fail for whatever reason although we do pay up for customers if their are any outages as per the SLA which is above and beyond all other providers I know of. I'd obviously prefer to never have to invoke an SLA credit as we'd be hitting our SLA or better (as we have for most customers) but we do offer the service credits in our SLA as I think it's only fair carriers should do in the event the SLA isn't lived up to. Dave

Fortinets are great firewalls and very cost effective whether you go cloud based, onsite or VM. I've no experience of using Bloxx with them but we have with other filters (Smoothwall, Lightspeed, Iboss) and they all work well together so I'm sure Bloxx wouldn't be too different. Thanks Dave

Good evening @MJTayloronline and @Fazza I'll send you both a pm shortly. Sorry I haven't responded sooner on here but I've been away for a few days. Just checking our nni graphs into all the carriers, we are nowhere near maxing out capacity into any of our multiple interconnects so I'm surprised you're having issues. I'll still take a look though for you. I'm presuming you've checked your individual usage and had our support department run a throughout test whilst you've been on the phone to them as we can see live what's being used? Thanks Dave

Evening, we do support BGP on our network (you need to if you are an ISP and have your own AS number). You can put basic filtering on the Fortigate and also use Lightspeed, however this can cause issues particularly with HTTPS and we've also seen other problems when using some safe search functionality on certain search engines if you try to do it on two at the same time. Backup lines come in via an alternate carrier and terminate on a different Cisco core router so this gives carrier resilience in the event one of them fail. The Fortigates should fail over correctly in the event one fails and stops passing traffic (Although one of them wasn't in this particular issue due to a bug in FortiOS). We can in certain situations segregate traffic so it doesn't even hit the Lightspeed boxes. Applications such as SIP are commonplace to do that with here as they don't need filtering and you can normally segregate your VoIP traffic by putting it on a separate VLAN on your LAN and extend this into the hosted Fortigates. The Fortigate filtering is actually pretty good as a filter, just very complex for primary schools to use and setup and also without the additional of a Fortianalyzer or Forticloud service doesn't give you the reporting that Lightspeed does and in our opinion schools need. Thank you for the supporting posts. We do a heck of a lot of testing as do our vendors and we generally follow their recommendations for FortiOS updates too. Normally applying a patch can take 3 months due to the knock on effects it could have. We're all more than annoyed with the issues we've just had in particular with Fortigate 3 and v5.2.8 of FortiOS. So far it looks like the upgrade to v5.2.10 has removed the crashes that were happening and not then instigating a failover scenario. We will continue to monitor though as it's not been a week yet since we've upgraded. Dave

Thanks for the support @Edu-IT I highly suspect we are going to be rolling out an emergency patch from Fortinet later tonight. Watch this space though we'll keep you updated. Dave

90 days

Correct, we are signed into the same term as the school is with our carriers so can't let people out of contract early. Dave

Right, not a good day today for all of you affected. Again it's not been everyone by any means but we do unreservedly apologise. We have had the same issue hit Fortigate 3 that happened last week. Fortigate 3 was swopped out on Friday evening for brand new hardware and up to lunchtime today had not displayed the same symptoms as the old unit. As new hardware has not fixed the issue we've had to take a very hard decision of not immediately rebooting the device to bring customers back up and running as quickly as possible. The reason for this is that we've had to have Fortinet TAC get access to the box remotely to find out what is actually happening in the underlying FortiOS (software version) the Fortigates are running on. If we were to simply reboot the box again then we would lose what the underlying semi-crash condition is and it would likely reoccur in the future. Our NOC engineers are in constant contact with Fortinet TAC in Europe and this has been escalated as high as it possibly can be and is classed as a P1 ticket. We have our theory as to what maybe causing the issue but nevertheless the box should not get into the state it is doing. I highly suspect we will be advised to upgrade to a new version of FortiOS which we will do once Fortinet officially advise which minor upgrade patch that should be. I'll keep you all updated as soon as we have some more news from Fortinet. FYI your SLA is sent to you when you originally signed up to our service and includes how to claim service credits. If you need another copy please do contact your account manager who will send it over to you. Thanks Dave

Since the kit was replaced on Friday evening we've not seen the same issue happen again. The suspected faulty kit is on its way to Fortinet to diagnose exactly what issue has happened to the hardware. We're also expecting a new firmware version to be released very soon which addresses this particular set of conditions where failover did not occur when it should have done. We'll keep you updated on here and by the usual channels too. I suspect a full incident report will not be fully available until Fortinet get back in touch with us regarding the what exactly happened to the faulty unit and why a failover condition didn't occur when it should have done. FYI I think you can actually now have 3 active / active / active Fortigates in a cluster although we've not tested that and I suspect it'd be quite a network overhaul to implement it too. Thanks Dave

Of course it's not. The hardware issues with one of the Fortigates have only occurred this week. Good news is that the replacement unit has been installed yesterday evening so as per my other post we'll be monitoring closely over the next week. Dave.

Morning all, The swop out of Fortigate 3 yesterday evening appears to have been successful. We will be closely monitoring over the next week. Thanks Dave

We are. Nothing is more important than to fix things when they go wrong and its all our engineers are doing at the moment to get this issue resolved. We've quite a few of them camped in Telecity over the weekend. New customers aren't on this affected hardware and as much as I like and respect our sales department I really wouldn't want them fixing core Fortigate firewalls anyway Dave

in short yes they should failover and the firewall hardware is resilient. The price of a pair of them is more than a 3 bedroom house in some towns! The Fortigates work in Active / passive mode. When Fortigate 3 stops passing traffic over one of it's WAN porst it should instigate a failover on to the other unit. We've had to physically log onto the remote reboot switch connected to Foritgate 3, reboot it and then Fortigate 4 (the failover unit) immediately springs into life and takes all traffic. When Fortigate 3 re-loads it then takes over as the master / active device. Fortinet TAC engineers are working on why the HA isn't kicking in automatically (it's not a configuration issues) and from speaking with them they have some other customers with similar cases open relating to the v5.2.x firmware the firewalls are running. We have had failover in the past even on this firmware automatically failover on other units but it is not doing so in this specific case. We may upgrade the firmware at some point but we can't be cavalier upgrading firmware on such a major piece of kit as that could make matters worse by introducing other issues. Very frustrating, particularly when compounding some other issues and couldn't have happened at a worse time. As discussed although we've had quite a few customer affected the majority still haven't been. Thanks for your patience. More comms to go out later tonight / tomorrow morning once the swop out has gone through. Dave

We use Draytek 130 modems as standard on VDSL / FTTC installs. Very good boxes and we've had few problems with them. Dave

I suspect a few. Ironically the large ones I spoke about haven't had any issues and we've actually been praised by their technical team of late for providing a rock solid service and good support. More people always say bad things than good things. I can say that as we have so many customers then 5% of 2,000 is a lot more than 5% of 100 so more will be vocal..... Still we must always do better. Thank you for those that have been continually supporting us. Dave