EXA_Mark

It already has pushed them down consistently for the past twenty years since DSL was first introduced. The very first leased line i was involved within installing for ourselves as a business in the 90's (not Exa) was a 256Kbps line at £18500 a year! You have to bear in mind in comparing FTTP (OR) to Leased Lines or DarkLight that those services are congested and do have contention and speeds do vary (sometimes massively), and sometimes to specific connections (such as MS, Google or Apple) on update or patch days. Leased Lines or DarkLight (Dark Fibre) should not, as long as your ISP has enough capacity in their core.

Just a quick word of warning to all about Openreach FTTP, although I am sure many of you are aware of this. From a technical perspective, all fibre to the premises (Leased Lines, Dark Fibre and such) are FTTP. However FTTP as talked about a product replacement for DSL/EoFTTC/FTTC/Gfast from Openreach uses GPON technology, and is an asymmetrical service, meaning the download is faster than the upload (how Openreach DSL type products have worked since 2000, not using GPON mind, but asymmetrical). Dark Fibre services, such as DarkLight or Leased Lines are symmetrical, meaning you have the same speed up and down. Also, FTTP (Openreach) is not a guaranteed speed service, it is subject to peak time congestion and slow down, (when Peak time actually is now that half the people are working from home all the time I am not sure though). It is in the terms and conditions of OR and others such as Virgin broadband services. The biggest thing to take out of this though is the limits of the upload on OR FTTP services, on 300Mbps FTTP you reference the upload is only 50Mbps. If you can get the 1Gbps one it is just over 200Mbps. Now you might say that is more than sufficient, but remember the quote at the start of this thread, more teachers and students are using cloud service than ever before, and that means sending data back out to the web MUCH more than when you are simply web browsing or streaming. In the past few years, we have seen upload speeds increase on our networks four times faster than downloads speed. Now do not get me wrong, people use the download element much more, but a few years ago it was a ratio of about 10:1, so if you had a 100Mbps pipe we would see peak uploads of around 10Mbps. It is now closers to 45% at peak. Meaning if you had a 100Mbps line we would see peak uploads of around 45Mbps. Bear than in mind. That is now. It will continue to increase. So if you are thinking an upload speed 50Mbps will be ample for us for years to come, it probably won't it almost isn't now. And just to finish, please don't misunderstand me, FTTP is a huge jump compared to current mass available technologies such as DSL and FTTC, and it is a service we provide ourselves too, but if you are on a 1Gbps leased line and seriously considering this as an option then do get your upload stats checked first. And this is not us trying to push more expensive or lucrative products, the actual margins on most connectivity services are very similar, it is about having something that works.

Unfortunately, it is the old adage, Steve, it depends where you are. For instance, if you are in one of our DarkLight areas we install 10Gbps bearers on our Dark Fibre and these come in under £6K a year with only £500 install, with 300Mbps on 10Gbps coming in at £3,500. For 1Gbps on 1Gbps, at the moment though you are generally seeing this in the region of £6k-£9K per year. I'm not surprised on the current pricing you have on your 300Mbps line (I assume it was a three-year contract signed in 2018) as the wholesale tail on Gigabit circuits (especially those provided into ISPs directly through Openreach has come down massively in the past three years. The price being added on for the actual internet bandwidth (not included from Openreach) has remained fairly constant for the last few years though, and whilst it has come down, it is the tails that have made the biggest impact. If you'd like me to send you over a quote just PM me your details, and don't worry, we are not a pushy sales type company, we send you a quote and then are there if you have questions or want to proceed.

Hi, You are right we are not great. We are awesome Seriously though I do not know who has said what, and that is not for me to say their opinion is wrong, even if it is. We take what we do very seriously, and have done since we started supplying schools with connectivity and filtering 17 years ago. There are plenty of people on here (and thanks to those who have said so in this thread) that like what we do, and our customer retention rate of over 95% would seem to back that up. Utlimately, do we have issues from time to time. Sure. In fact every provider on here has threads (some many more than others) on their outages or issues. But if we do have them, we try and make sure we communicate, and learn from them if we do. At present we are one of the largest non-grid for learning ISPs in the UK, and serve thousands of customers for connectivity and filtering and thousands more for our Exa Foundation. We have been very fortunate to win many awards (not the sort you sponsor and get one), including ISPA (our equivalent of BAFTA) for Best Business ISP, Best Customer Service and Best Broadband multiple times. In fact we have had over 60 finalist nominations in 16 years from ISPA. And we are shortlisted for Five next month, including Best Business ISP. As an aside to that we also write our own software in house, such as SurfProtect, so we tend to be amongst the very first in the world to fix problems such as Google or Captcha updates, as we are not waiting for update patches to push across consumer equipment in the same way or for third parties to write fixes, we do it ourselves. And our ExaBGP service is used by many of the worlds largest tech companies, including Microsoft, Twitter, Facebook, BT and the BBC. So are we perfect? No, but we are having a bloody good go at getting there. Let us put you in touch with ANY of our customers (not just testimonial ones) or others that have moved from competitors you are considering, and let them speak for us. Hope you give us a go.

Hi, Going through one of our partners or resellers does not mean you can't contact our support team. Give them a call on 0345 1451234 or email [email protected] and they will help you.

You are absolutely correct. Whilst we can issue IPv6 to any customer on our connectivity when it is something you want to run through SurfProtect Quantum, it adds in MUCH more complexity around configuring profiles, analytics, diagnosing route paths and such, which is why at present we don't. It is not to do with network routing, but rather specific requirements for the service. However, IF you would like to work with us directly, we will develop this, working with you, to get all the bits in place. We could start work on this in Q1/Q2 next year, depending on your availability. PM me with your contact details if you are interested and I'll pass it onto the team and they'll be in touch and go through what would be involved.

Just a quickie from a suppliers perspective. We have been running dual-stack across our network infrastructure on IPv4 & v6 since 2007 and a huge amount of network traffic coming back into us, in particular sites such as Google and Facebook by default is via IPv6 nowadays. BUT very few customers across any sector, Education included have made the move, but the rapid deployment and exhaustion of v4 across the world (not us specifically) will mean people will have to do at some point in the not too distant future. If any of our customers on here want an IPv6 allocation, just give your account manager a call/email, or PM me. It should go without saying, but I will in case anyone asks, we do not charge for this, it is part of the standard service.

Thanks @cdCache @TJ-Diggers if you need anything from us or have any questions at all please let me know. If you like to speak to any of our schools, local to you or further afield again just let me know. We've been providing internet connectivity and filtering to thousands of schools since 2003, so plenty for you to speak to, schools who've been with us for 15+ years or others who have just joined. And it is not just connectivity, take a look at http://www.exa.foundation to see what else we provide to schools (Foundation stuff is all free of charge, and available nationwide). Enough sales pitching, I try not to do that on here, it always feels awkward, PM me if you need anything else.

Hi @Clansman, Firstly welcome to Edugeek and thank you for the post, as you can tell it is something we feel very strongly about. We are developing some courses (for free) as part of our Exa Foundation, http://www.exa.foundation where each year we run hundreds of events for schools throughout the UK, if you'd like to get involved, or chat with me/Exa on anything we can jointly do to educate, please PM me or give me a ring on 0345 1451234

We are putting together a top tips guide just for this very reason Peter, as soon as it is done, I'll let you have it, and post it on here too. Probably late next week.

Part 2 of 2 One of the other posts on the thread commented about how they’d been on a local authority connection previously and were down for two weeks due to an attack. This is not that uncommon, unfortunately. Those sorts of providers typically simply do not have the type of resources, people, experience, different data centre, peering and therefore they often simply have to let the attack run its course. Another big issue is dealing with an attack that you block out of your network, but it still hits the upstream before your edge network, which no “Lamborghini” is going to stop. The major peering points in the UK do not have DDOS hardware or solutions installed. We also have DDOS protection service from our upstreams, and again this helps mitigate it away from us and our customers quickly, but again, only if it the traffic comes this way, and you have to bear in mind that even when our upstream providers block it, it is still hitting somewhere! So unless you decide to push all your traffic through an external source first (which in itself can cause other issues such as latency) such as Cloudflare (which works for a website but not for an internet connection), then you will have that issue, as you would with private peering/interconnects and many other issues. And even those type of companies only promises to mitigate “most attacks”. So again, no one simple solution. What you need to have is many different solutions, and we are looking at even more, and ability to write your own to meet specific requirements, and experience, which fortunately we have in our engineering and R&D side, but passing down the answers from those firefighters in the heat of an attack to the support team (and bear in mind how many different sorts of attacks they can be, its not simply a case of training), is not always easy, and so sometimes a mixed message gets out. For which I am really sorry. We are looking at ways of improving how we can get this out better in the future, without slowing down those sorting the issues. We also try and put out the updates on status.exa.net.uk but on one occasion the message was not put out quick enough, simply, the people who normally do the updates were not in the office, and it got missed until one of our customers pointed it out. One of the other challenges is the sheer volume of traffic that networks such as Google and Amazon have going to and from their networks. Simply for cost reasons, they will always prefer peering, or private peering, to transit. So if you have an attack and try and re-route one of those, often their own internal automated systems may/will override your mitigation attempts and still send it down a path you don’t want it to use. I am very aware this even though I said this is a long post, it is now a VERY long post, so I will just try to end it by saying a couple of final things. To those who were affected by the DDOS indirectly, our please accept our apologies, we always try to do better, even if it is not us causing the problems. To everyone who expressed their understanding of our dilemma, thank you. When the teams are under pressure, it is always really appreciated to hear and read these posts. Yesterday we implemented an update internally which has allowed for even faster detection and mitigation today. We had another attack this morning on a different school (who has never been attacked before). From start to end it was mitigated it in under two minutes. There will, of course, be others in the future that take us longer I am positive, but be assured we are also going to be working and implementing more solutions, internal and external to reduce the impact as much as possible. Finally, let me say that Exa has a zero-tolerance policy on DDOS, and all these attacks will be reported to the appropriate authorities. A DDOS is a criminal offence, and we will treat it, and act on it as one. We believe ISP, The Government and the schools can play a massive part in educating people and make it clear that this sort of criminal behaviour will simply not be tolerated. We hope that should a DDOS ever happens to your school, no matter who your service provider is, you will work with them to identify the individual and make sure they and the rest of the school know that you will not tolerate it either. If anyone has any other questions, please do feel free to ring me or PM me.

OK, let me start off by saying, this is going to be a long, pretty technically detailed post. I feel I cannot give a short answer to the many questions. It is so long in fact, that I have gone over Edugeeks 15000 character limit, so this thread post 1 of 2. Let me start by giving a few facts as there seems to be some misinformation on here, or lack of clarity, or maybe even confusion that we may have caused. We had four separate days (not entire days) of noticeable very large DDOS attacks, this has not been going on for months. The first one was last week, then three others this week (I'll come back to this later in the post). Over the past few days, the attacks have been against three specific schools (not SME or Corporate customers). Two each for two of them, and one for another. The schools are not connected from an area or Academy trust perspective, but we believe the attacks on the three schools are related, probably by individuals within the schools communicating via forums or chat rooms. One of those individuals initiating one of the DDOS one has already been personally identified by the school and has been dealt with very severely. A DDOS attack is a criminal offence, and we treat them as such when we can identify the individuals. The other two individuals are still being sought and we are working directly with the schools to do so. Where we manage the firewall and filtering, it is often quite easy for us to follow the breadcrumb trails, you'd be surprised how big a trail people leave, and find the perpetrators which were the case, with the first school, Two other schools were using their own in house firewalls and filtering solutions, and as such we are supporting them and doing all we can to help them discover the identities of the students performing the DDOS. So onto some facts and figures. On a normal school day, we see peak traffic around the network of between 12Gbps for a normal day & 20Gbps for a busy one peak traffic across our network. All of our core Data Centres /PoPs connect at up to 100Gbps. So you maximum throughput on any one port can only be 100Gbps. We have three upstream providers (for what is referred to within the industry as commercial transit) from our DCs, with more than 4 x our overall network peak traffic of available throughput/transit. Your average internet service provider may have 1.5 times peak (so very little overhead)! On top of that, we have multiple 10Gbps peering sessions, including a direct PNI (Private Interconnect) with Google (not public peering). As the vast majority of traffic in the UK goes through peering this means we currently have more than 9 x our peak throughput on our edge and we can increase this when we need to While this may seem excessive traffic flows can change and therefore some headroom is required to not face surprises. Any customer going through our network would be able to traceroute to different locations and see the different routes. For instance, try google, and you’ll go through the PNI, try BBC and you’ll go through peering and try BT and you’d most likely through transit. We can also prioritise how traffic leaves our network if we needed to for maintenance or emergency reasons (such as DDOS) and in some measure affect how it comes in. We publish our peering points on https://www.peeringdb.com/net/524 If you are interested in this you can also look at our competitors to see what they have in comparison. It does not, however, show private interconnects (as peeringdb is to help you find peers). It also shows you things such as our average traffic levels too. This information is however self-published and should, therefore, should be taken with a pinch of salt. On the specifics over the 4 DDOS days (not concurrent full days). The first attack took place on the 26th September at around 8 am, a status update was put on the status.exa.net.uk page within a few minutes of the attack commencing. It was targeted against a single school (the one that we have identified the individual since, and there has been no recurrence). The attack was substantial and saturated nearly all of the peering points. Our engineers' saw this on our mitigation platform some change happened to improve the situation in less than ten minutes, but not stop entirely. By 08:20 we had cleared all DDOS traffic from transit but peering points we still saturated (I’ll come back to this later on). By 08:25 this was limited down to specifically the peering points in London, LONAP and LINX (the biggest exchange point in the UK). All traffic that was going through transit or to Google was now not affected. Then at 09:25, the attackers started using Google for the attack. Google was the largest source of all traffic, which meant the PNI to Google was then affected, but all other traffic was then fine. At 09:45 Google changed how the traffic was reaching us due to the attack. Google has an automated system to optimise its traffic. We, therefore, had to perform some network configuration changes to protect our peering links. However, the traffic was lower so customers were not seeing the same level of impact. This continued to reduce down to zero over the coming hours with our mitigation services in place. By 2 pm there was no sign of any DDOS activity on the network. The next series of DDOS attacks happened on the 2nd October starting around 1:25. This was fully mitigated within five minutes. About 15 minutes later a different school (the third targeted) this was fully mitigated within a few minutes And then yesterday (4th October) the same school (the second one) was targeted again. This time the attack came through our transit, not peering locations. This meant over 80% of traffic was going through as normal, so the likes of Google and BBC were mostly unaffected, but services with some VoIP providers (as some mentioned in the thread) which do not have peering with us at the exchange points or privately, were affected. This time the traffic was identified as coming Amazon’s cloud services. For very obvious reasons we cannot simply block all of AWS traffic, as they provide cloud-based hosting. So we had to mitigate in a different way, which unfortunately ended up taking a few hours, although there were improvements to many sites/locations within a few minutes of it starting. So that is a quick(!) summary of the four separate worth of attacks. Sometimes at overlapping times. It is always much harder to deal with this sort of thing when they are coming from lots of different routes to different customers. When our customers are under a DDOS which is affecting more than the school, our operational practice is to first restore service, and then get back the affected school back online. We knew all of the attacks had commenced within the schools, or from someone who had been in the schools, as following an attack we change the IP of the connection. When following this IP change, a new DDOS occurs again the school, it is generally down to someone from the school looking up the IP address of the firewall or gateway. It could also be a compromised machine, but we are still to see this. Generally speaking, with 20 years experience of this, it is nearly always a kid. I will now try and answer some of the other comments or concerns. @GTX and a few others kind of said similar, “A network that big needs DDOS protection. It's not cheap but it the world we live in now”. I absolutely agree. Which is why we do have DDOS protection in place and have done for over 15 years; I’ll cover @SchoolsBroadband “Lamborghini” reference later on specifically. We have different DDOS protection within our network, some inside, some using our suppliers. Unfortunately, despite what some others might want you to believe, there is no single box solution, that will cover every eventuality. One of the bits that were mentioned was ExaBGP. The software we wrote in house, that to be fair is used by many large technology companies, for different reasons, including DDOS mitigation. Most DDOS ISP mitigation services use FlowSpec to stop traffic from a particular protocol for or to an IP (for instance all DNS going to the IP of a school), whilst keeping the school up. In a lot of cases, this alone is enough to end an attack. The main issue is what do you do when an attack is bigger than your upstream. No equipment within your own network can then mitigate the issue. It must be coordinated using your upstream network. This problem is made worse as there is little check from the sender that the traffic they are sending should come from their network. Industry efforts exist to attempt to sort this issue but it is not going to be an overnight thing. https://www.manrs.org/isps/guide/antispoofing/ So to mitigate an attack, you have to have many different solutions, which to an outsider, looks like one solution to one problem when it is many different ones, badged under the same umbrella. The D in DDOS is for Distributed (Denial of Service), therefore there is more than once source of the attack to handle. There are other things we have inside our network, like Dave, and these all help with mitigation. And all you can do is mitigate. You cannot stop someone attacking a network, no matter what devices or solutions you have in, what you have to do is mitigate the impact as quickly as possible. The reality is we have attacks on a regular basis, as do most ISPs but the majority of them are invisible to everyone other than the person/organisation that is targeted when we can not “simply” mitigate the issue in a way invisible to everyone. If you google for “ExaBGP FlowSpec filetype:pdf” you will see many industry-leading experts talking about it, and how they integrate it. A few open-source and commercial products are using it. As I do not expect you to just take my word for it, here are a few references Cisco speaks of it: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2019/pdf/BRKSPG-3012.pdf Juniper: https://conference.apnic.net/data/41/apricot-ddos-mitigation-using-flowspec_1456208439.pdf T-Mobile: https://ripe74.ripe.net/presentations/93-20170512-ripe74-flowspec-interop.pdf We work within the Internet Engineering Task Force (IETF) to make sure ExaBGP is a good as it can be. You can google: "ExaBGP implementation site:tools.ietf.org" to check. Several commercial DDOS companies are using ExaBGP too, so when @SchoolsBroadband said he’d “heard it was good” it is perhaps a bit of an understatement but ExaBGP is not a panacea. But it is just one piece of a very complicated issue. Which is why major companies such as Twitter, Microsoft and Facebook have had services taken down by DDOS attacks. If it was just a case of throwing unlimited money at a problem to fix all variants then those companies would do that. It is not a simple thing at all. Part 2 (due to character length limit) is directly below this post.

Could you PM me your contact details so I can get our support team to look into this. There is nothing happening on our network anywhere now that should be causing this. It is often easy to assume it is one thing (such as a DDOS) when it could be something completely unrelated and I want them to check for you. Thanks - - - Updated - - - Statement coming in the next couple of minutes. It has taken me far longer than I thought it would when I started writing it hours ago!

I guess Dave @SchoolsBroadband has a lot more spare time to be on the forums than I do I also tend not to post into other supplier-specific issues threads such as SB has had over the past week with their filtering or in the past with DDOS too, as quite simply, as someone else in this thread wrote, all suppliers will have an issue from time to time, it is how they handle them, and how they respond to them. Seriously though, I do try and keep an eye on the forums, but as you can all appreciate with your busy day jobs too, I don't always get to check them as often as I would like. The post was brought to my attention yesterday evening as I was out of the office yesterday most of the day, and I am writing a fairly detailed long post now, which will be up on here very shortly. So please don't think I am ignoring my fellow edugeekers.

There is no issue that is affecting this from our side, so if I can ask you to give our support team a call on 0345 1451234 they will be able to take a look at this for you, or if you PM me your contact details I will get them to call you.