SchoolsBroadband

I know numerous LA's have left / are leaving their service and we are also doing so. Shame as originally the product was great but they were a funny company to do business with and at the end of the day we just didn't get the support we needed to run our business when their service had issues. Dave

We're here. 1500 schools and counting. We've been installing a lot of full 1Gbit circuits or late and even a few on 10Gbit bearers and dark fibre. Thanks Dave

Gamma offer private peering although I think they charge for it which is annoying and you then have to pay the DC for the cross connect where the prices have just gone up massively in Equinix data centres. That said we have a private interconnect with our wholesale voip suppliers for the reason you mention and the ability to do end to end qos. I'm not sure if Gamma offer peering over LINX or Lonap.....

Thanks @Clansman. We've already been approached by the NCA and are meeting with you next month. All ISPs get hit by DDoS's and I personally welcome your approach. Particularly around educating. Dave

ha! I must admit it is annoying when Talk Talk do some re-zoning. It's normally to the benefit of schools and prices reduce, not the other way around but can be in specific circumstances. Can you send me a PM and I'll take a look for your personally? Thanks Dave

We now mostly sell full 1Gbit circuits to secondary schools as the price difference is now very minimal between a 300Mbit and a 1Gbit. If you'd like a quote do let me know Thanks Dave

Hi again. The escalation matrix will have been sent by our delivery team. Can you pm me your details please and I'll get an engineer on it for you asap? Thanks Dave

Evening all, I've a meeting with our Technical Director tomorrow morning. We have had an issue with the Web admins although that on the whole was resolved. However we've a case open with Netsweeper around the use of agents and some of the issues you mentioned. The issue has been very adhoc and doesnt affect everyone. We believe we know why and it's with Netsweeper to diagnose. We are Netsweepers biggest Wagent user in the world which is why we and they havent since this issue elsewhere before. I'll put an update up as soon as I have it. Just so others understand there are no outages here, just some auth issues for a subset of users in certain scenarios when using wagent. I'll be in touch. Dave

Of course. I'll PM you by monday at the latest. Dave

We are here and can do all you require. Certainly we dont have a £3k setup for VoIP for a small school!

Hi @MrWu A) I'm not sure on that one myself B) This could be to do with the timer from the Wagent or the Webadmin issue we saw which did affect some users authentication for a short while. The portal issue is now definitely resolved and may also have affected the issues you've said here. We've since fixed the Webadmin issue so these should be resolved. If they are not then please do get in touch with the support team. I know cases are still quite high at the moment. If you're still struggling please send me a PM. Thanks Dave

I'd always recommend looking for individual users searching for this such as "how to ddos myself" / "DDoS stress tester" etc. Sometimes catching them before they've done it will mean less pain in the long run. Outward bound from your network also look for machines sending high volumes of traffic and what type of traffic. They could be compromised machines flooding data out of your network attempting to attack somewhere else . DNS and ICMP are common protocols used in DDoS flood attacks. Dave

Thanks @andy_b and @SVM for bringing that to my attention. I'll speak with the NOC guys today. Dave

I do love Edugeek

I do feel for Mark and co as it appears they have been hit by some pretty big DDoS's which can be a real pain to manage. There are lots of things ISP's can do to mitigate them. I believe Exa uses ExaBGP to help mitigate DDoS attacks which is their own BGP protocol they've designed and I hear is pretty good. We get hit by DDoS's near enough everyday. A while ago we invested a large six figure sum (we haven't even announced it to our customers) into a specific anti-ddos mitigation platform. So far its never broken sweat but time will tell. What I can say is that it has got rid of some attacks which would have caused us problems in the past so from my point of view despite costing more than a Lamborghini Aventador it was money well spent. I know a number of other ISPs which have bought the same platform as us and are also very happy with it. Dave

Hi all, we've made a number of changes to the Webadmin. if anyone is getting slow load issues please do let me know. Thanks Dave

Afternoon all, with the issue we had the other day we've been very careful with putting some of the servers we took out of our load balancing pool back in as the last thing we want to do is to effect peoples Internet connections which have previously been stable using Netsweeper. The Google Captcha issue is likely happening because we have removed about 50% of our capacity as a temporary measure. We're adding in another 25% tonight and the final 25% the day after which will fix the Google Captch issue. The issue relates to Google blacklisting IP addresses when it see's in its eyes too many connections coming from a single IP address. Our emails to them to get them whitelisted goes constantly unanswered and they have been extremley unhelpful. I suspect thats why Updata / Link2ICT / Entrust also have the issue. Once more servers are added back into our cluster this issue will be resolved. The slow Webadmin issue is rather complex to say the least. We are going to make a change to the underlying Linux operating system this evening. We do not want to do this in the day time in case in has a negative effect on the platform which I'm sure you'll all appreciate is the right thing to do as we don't want to break anything. It may also have a positive effect on the change in filtering policy from staff to pupil we are occasionally seeing. We have engaged with Netsweeper support throughout this issue and I'm sure we will have it resolved shortly. Our primary focus is to ensure customers have filtered Internet access and we don't want to jeopordise that for the 1,500 or so customers which are currently live on the platform. I'll post an update when I have more information to tell you. Thanks for your patience. Dave

The affected customers should be back up and running now. If not please call the help desk. Ok so we had a partial outage on some specific netsweeper clusters which may have been caused by a software upgrade / configuration change over the weekend. We've pulled some servers from our load balancer cluster as a precaution and will re add them once we've found where the issue 100% is. Thanks for your patience Dave

Morning all. We've a small subset of customers affected. The vast majority aren't. NOC are working on it. I'm away from the office today so csnt give detailed information until later which I will do as soon as I have it. Thank you for your patience for those affected. Dave

Evening folks. We are behind compared to where we normally are with a high work stack. This is due to numerous reasons but what I can say is that weve taken on two new first line guys in the short term and have also enlisted the noc team to help with more complicated cases. We expect to be back to normal next week. If you've a critical issue please do escalate to Angela (operations manager) or send me a PM and I will do not for you.

Does it do it for all users on that machine? If so could it be the switch port?

PM'd.....

Nope what happened today was definitely very different from some of the stats and things we've initially seen. I suspect that was an adhoc thought with someone speaking out loud. We'll let you know in our report what happened as soon as its ready and we've done a full root cause analysis. Thanks Dave

Hi all, we're very sorry for the issue this morning. We're working with Netsweeper (which is where the issue was) to ascertain how this happened as it's not done before. This is our first major issue in 2 years since we've installed it which is very frustrating. We will send through an outage report as normal. Ref the webadmins and them running slow; we are aware of some issues at the moment and already have some new hardware ordered and ready to be installed soon which should see a marked improvement in response times. Please watch this space. Thanks Dave

Hi @caffrey, by default we use a transparent proxy. All you need to do is install the Wagent and certificate on any devices you want to do SSL decryption on. We do have a full proxy solution if you'd prefer to use that although we recommend transparent proxy as it's much easier to use and setup. On the day of changeover our engineer will help set everything up for you although it does help if you've the Wagent and SSL cert installed prior to the setup. Regards David