EXA Networks - The DDOS Fiasco

[k-strider]

All seems good now this morning

[E_G_R2]

Nearly 2 years with exa and the connection has been rock solid so im sure exa will figure this out but with everything online now it does hit us hard in schools.

 

What i would like is a statement from exa to give to the senior team so they know the issue is a isp level and not the school.

 

Not had an official statement but the service desk confirmed that my issues (not VOIP) may be attributed to the DDOS attack so I have sent copy to SLT and Staff.

[Tefters]

Blipped this morning, briefly I will add but still blipped (5 minutes) complete outage!

Pings still not settled.

[DJ-1701]

Taken from https://status.exa.net.uk/incidents/223#update-82

 

Network Stability - DDoS Attack Friday 4th October 2019 10:11:45

One of our customers has been targeted by a DDoS attack. Our engineers are working on mitigating this issue. Apologies for any inconvenience caused.

 

Our engineers have now mitigated the DDoS attack.

[mikeyw]

Being a MAT and have had nothing but issues with our sites that use EXA Networks, it is causing a nightmare with our VPNs, Central services and our 3rd Party SIP Lines.

Still keep seeing our response times between sites dropping.

 

We also have a backup connection with EXA however in this situation it is unusable.

Edited October 4, 2019 by mikeyw

[GTX]

Yeah, noticed the post today @DJ-1701

 

Still waiting for this statement and how they are going to stop it happening in the future.

[EXA_Mark]

Being a MAT and have had nothing but issues with our sites that use EXA Networks, it is causing a nightmare with our VPNs, Central services and our 3rd Party SIP Lines.

Still keep seeing our response times between sites dropping.

 

We have a backup connection with EXA however it is unusable in this situation.

 

Could you PM me your contact details so I can get our support team to look into this. There is nothing happening on our network anywhere now that should be causing this. It is often easy to assume it is one thing (such as a DDOS) when it could be something completely unrelated and I want them to check for you.

 

Thanks

 

- - - Updated - - -

 

Yeah, noticed the post today @DJ-1701

 

Still waiting for this statement and how they are going to stop it happening in the future.

 

Statement coming in the next couple of minutes. It has taken me far longer than I thought it would when I started writing it hours ago!

[mikeyw]

Thanks mark, i have now sent a PM.

[EXA_Mark]

OK, let me start off by saying, this is going to be a long, pretty technically detailed post. I feel I cannot give a short answer to the many questions. It is so long in fact, that I have gone over Edugeeks 15000 character limit, so this thread post 1 of 2.

 

Let me start by giving a few facts as there seems to be some misinformation on here, or lack of clarity, or maybe even confusion that we may have caused.

 

We had four separate days (not entire days) of noticeable very large DDOS attacks, this has not been going on for months. The first one was last week, then three others this week (I'll come back to this later in the post).

 

Over the past few days, the attacks have been against three specific schools (not SME or Corporate customers). Two each for two of them, and one for another. The schools are not connected from an area or Academy trust perspective, but we believe the attacks on the three schools are related, probably by individuals within the schools communicating via forums or chat rooms.

 

One of those individuals initiating one of the DDOS one has already been personally identified by the school and has been dealt with very severely. A DDOS attack is a criminal offence, and we treat them as such when we can identify the individuals. The other two individuals are still being sought and we are working directly with the schools to do so.

 

Where we manage the firewall and filtering, it is often quite easy for us to follow the breadcrumb trails, you'd be surprised how big a trail people leave, and find the perpetrators which were the case, with the first school, Two other schools were using their own in house firewalls and filtering solutions, and as such we are supporting them and doing all we can to help them discover the identities of the students performing the DDOS.

 

So onto some facts and figures.

 

On a normal school day, we see peak traffic around the network of between 12Gbps for a normal day & 20Gbps for a busy one peak traffic across our network.

 

All of our core Data Centres /PoPs connect at up to 100Gbps. So you maximum throughput on any one port can only be 100Gbps.

 

We have three upstream providers (for what is referred to within the industry as commercial transit) from our DCs, with more than 4 x our overall network peak traffic of available throughput/transit. Your average internet service provider may have 1.5 times peak (so very little overhead)!

 

On top of that, we have multiple 10Gbps peering sessions, including a direct PNI (Private Interconnect) with Google (not public peering). As the vast majority of traffic in the UK goes through peering this means we currently have more than 9 x our peak throughput on our edge and we can increase this when we need to While this may seem excessive traffic flows can change and therefore some headroom is required to not face surprises.

 

Any customer going through our network would be able to traceroute to different locations and see the different routes. For instance, try google, and you’ll go through the PNI, try BBC and you’ll go through peering and try BT and you’d most likely through transit.

 

We can also prioritise how traffic leaves our network if we needed to for maintenance or emergency reasons (such as DDOS) and in some measure affect how it comes in.

 

We publish our peering points on https://www.peeringdb.com/net/524 If you are interested in this you can also look at our competitors to see what they have in comparison. It does not, however, show private interconnects (as peeringdb is to help you find peers). It also shows you things such as our average traffic levels too. This information is however self-published and should, therefore, should be taken with a pinch of salt.

 

On the specifics over the 4 DDOS days (not concurrent full days).

 

The first attack took place on the 26th September at around 8 am, a status update was put on the status.exa.net.uk page within a few minutes of the attack commencing.

 

It was targeted against a single school (the one that we have identified the individual since, and there has been no recurrence). The attack was substantial and saturated nearly all of the peering points. Our engineers' saw this on our mitigation platform some change happened to improve the situation in less than ten minutes, but not stop entirely. By 08:20 we had cleared all DDOS traffic from transit but peering points we still saturated (I’ll come back to this later on).

 

By 08:25 this was limited down to specifically the peering points in London, LONAP and LINX (the biggest exchange point in the UK).

 

All traffic that was going through transit or to Google was now not affected.

 

Then at 09:25, the attackers started using Google for the attack. Google was the largest source of all traffic, which meant the PNI to Google was then affected, but all other traffic was then fine.

 

At 09:45 Google changed how the traffic was reaching us due to the attack. Google has an automated system to optimise its traffic. We, therefore, had to perform some network configuration changes to protect our peering links. However, the traffic was lower so customers were not seeing the same level of impact. This continued to reduce down to zero over the coming hours with our mitigation services in place. By 2 pm there was no sign of any DDOS activity on the network.

 

The next series of DDOS attacks happened on the 2nd October starting around 1:25. This was fully mitigated within five minutes. About 15 minutes later a different school (the third targeted) this was fully mitigated within a few minutes

 

And then yesterday (4th October) the same school (the second one) was targeted again. This time the attack came through our transit, not peering locations. This meant over 80% of traffic was going through as normal, so the likes of Google and BBC were mostly unaffected, but services with some VoIP providers (as some mentioned in the thread) which do not have peering with us at the exchange points or privately, were affected.

 

This time the traffic was identified as coming Amazon’s cloud services. For very obvious reasons we cannot simply block all of AWS traffic, as they provide cloud-based hosting. So we had to mitigate in a different way, which unfortunately ended up taking a few hours, although there were improvements to many sites/locations within a few minutes of it starting.

 

So that is a quick(!) summary of the four separate worth of attacks. Sometimes at overlapping times. It is always much harder to deal with this sort of thing when they are coming from lots of different routes to different customers.

 

When our customers are under a DDOS which is affecting more than the school, our operational practice is to first restore service, and then get back the affected school back online.

 

We knew all of the attacks had commenced within the schools, or from someone who had been in the schools, as following an attack we change the IP of the connection. When following this IP change, a new DDOS occurs again the school, it is generally down to someone from the school looking up the IP address of the firewall or gateway. It could also be a compromised machine, but we are still to see this. Generally speaking, with 20 years experience of this, it is nearly always a kid.

 

I will now try and answer some of the other comments or concerns.

 

@GTX and a few others kind of said similar, “A network that big needs DDOS protection. It's not cheap but it the world we live in now”.

 

I absolutely agree. Which is why we do have DDOS protection in place and have done for over 15 years; I’ll cover @SchoolsBroadband “Lamborghini” reference later on specifically.

 

We have different DDOS protection within our network, some inside, some using our suppliers. Unfortunately, despite what some others might want you to believe, there is no single box solution, that will cover every eventuality.

 

One of the bits that were mentioned was ExaBGP. The software we wrote in house, that to be fair is used by many large technology companies, for different reasons, including DDOS mitigation.

Most DDOS ISP mitigation services use FlowSpec to stop traffic from a particular protocol for or to an IP (for instance all DNS going to the IP of a school), whilst keeping the school up. In a lot of cases, this alone is enough to end an attack.

 

The main issue is what do you do when an attack is bigger than your upstream. No equipment within your own network can then mitigate the issue. It must be coordinated using your upstream network. This problem is made worse as there is little check from the sender that the traffic they are sending should come from their network. Industry efforts exist to attempt to sort this issue but it is not going to be an overnight thing.

https://www.manrs.org/isps/guide/antispoofing/

 

So to mitigate an attack, you have to have many different solutions, which to an outsider, looks like one solution to one problem when it is many different ones, badged under the same umbrella. The D in DDOS is for Distributed (Denial of Service), therefore there is more than once source of the attack to handle.

 

There are other things we have inside our network, like Dave, and these all help with mitigation. And all you can do is mitigate. You cannot stop someone attacking a network, no matter what devices or solutions you have in, what you have to do is mitigate the impact as quickly as possible. The reality is we have attacks on a regular basis, as do most ISPs but the majority of them are invisible to everyone other than the person/organisation that is targeted when we can not “simply” mitigate the issue in a way invisible to everyone.

 

If you google for “ExaBGP FlowSpec filetype:pdf” you will see many industry-leading experts talking about it, and how they integrate it. A few open-source and commercial products are using it.

 

As I do not expect you to just take my word for it, here are a few references

Cisco speaks of it: https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2019/pdf/BRKSPG-3012.pdf

Juniper: https://conference.apnic.net/data/41/apricot-ddos-mitigation-using-flowspec_1456208439.pdf

T-Mobile:

https://ripe74.ripe.net/presentations/93-20170512-ripe74-flowspec-interop.pdf

 

We work within the Internet Engineering Task Force (IETF) to make sure ExaBGP is a good as it can be. You can google: "ExaBGP implementation site:tools.ietf.org" to check.

 

Several commercial DDOS companies are using ExaBGP too, so when @SchoolsBroadband said he’d “heard it was good” it is perhaps a bit of an understatement but ExaBGP is not a panacea.

 

But it is just one piece of a very complicated issue. Which is why major companies such as Twitter, Microsoft and Facebook have had services taken down by DDOS attacks. If it was just a case of throwing unlimited money at a problem to fix all variants then those companies would do that. It is not a simple thing at all.

 

Part 2 (due to character length limit) is directly below this post.

Edited October 4, 2019 by EXA_Mark

[EXA_Mark]

Part 2 of 2

 

One of the other posts on the thread commented about how they’d been on a local authority connection previously and were down for two weeks due to an attack. This is not that uncommon, unfortunately. Those sorts of providers typically simply do not have the type of resources, people, experience, different data centre, peering and therefore they often simply have to let the attack run its course.

 

Another big issue is dealing with an attack that you block out of your network, but it still hits the upstream before your edge network, which no “Lamborghini” is going to stop. The major peering points in the UK do not have DDOS hardware or solutions installed.

 

We also have DDOS protection service from our upstreams, and again this helps mitigate it away from us and our customers quickly, but again, only if it the traffic comes this way, and you have to bear in mind that even when our upstream providers block it, it is still hitting somewhere!

 

So unless you decide to push all your traffic through an external source first (which in itself can cause other issues such as latency) such as Cloudflare (which works for a website but not for an internet connection), then you will have that issue, as you would with private peering/interconnects and many other issues. And even those type of companies only promises to mitigate “most attacks”. So again, no one simple solution.

 

What you need to have is many different solutions, and we are looking at even more, and ability to write your own to meet specific requirements, and experience, which fortunately we have in our engineering and R&D side, but passing down the answers from those firefighters in the heat of an attack to the support team (and bear in mind how many different sorts of attacks they can be, its not simply a case of training), is not always easy, and so sometimes a mixed message gets out. For which I am really sorry. We are looking at ways of improving how we can get this out better in the future, without slowing down those sorting the issues.

 

We also try and put out the updates on status.exa.net.uk but on one occasion the message was not put out quick enough, simply, the people who normally do the updates were not in the office, and it got missed until one of our customers pointed it out.

 

One of the other challenges is the sheer volume of traffic that networks such as Google and Amazon have going to and from their networks. Simply for cost reasons, they will always prefer peering, or private peering, to transit. So if you have an attack and try and re-route one of those, often their own internal automated systems may/will override your mitigation attempts and still send it down a path you don’t want it to use.

 

I am very aware this even though I said this is a long post, it is now a VERY long post, so I will just try to end it by saying a couple of final things.

 

To those who were affected by the DDOS indirectly, our please accept our apologies, we always try to do better, even if it is not us causing the problems. To everyone who expressed their understanding of our dilemma, thank you. When the teams are under pressure, it is always really appreciated to hear and read these posts.

 

Yesterday we implemented an update internally which has allowed for even faster detection and mitigation today. We had another attack this morning on a different school (who has never been attacked before). From start to end it was mitigated it in under two minutes. There will, of course, be others in the future that take us longer I am positive, but be assured we are also going to be working and implementing more solutions, internal and external to reduce the impact as much as possible.

 

Finally, let me say that Exa has a zero-tolerance policy on DDOS, and all these attacks will be reported to the appropriate authorities. A DDOS is a criminal offence, and we will treat it, and act on it as one. We believe ISP, The Government and the schools can play a massive part in educating people and make it clear that this sort of criminal behaviour will simply not be tolerated.

We hope that should a DDOS ever happens to your school, no matter who your service provider is, you will work with them to identify the individual and make sure they and the rest of the school know that you will not tolerate it either.

 

If anyone has any other questions, please do feel free to ring me or PM me.

[sbrade47]

@exa_mark Thanks for taking the time to explain what's been happening so comprehensively.

[howartp]

An excellent report, thank you. I've passed it to SLT as a whole report, but copied the first few and last few paragraphs directly into the email for a non-technical explanation.

 

Is there anything we (as network managers) need to be doing/watching/checking on our own firewalls - we're one of those schools who have EXA connection but our own firewall/filtering?

 

Peter

[EXA_Mark]

We are putting together a top tips guide just for this very reason Peter, as soon as it is done, I'll let you have it, and post it on here too. Probably late next week.

[SchoolsBroadband]

An excellent report, thank you. I've passed it to SLT as a whole report, but copied the first few and last few paragraphs directly into the email for a non-technical explanation.

 

Is there anything we (as network managers) need to be doing/watching/checking on our own firewalls - we're one of those schools who have EXA connection but our own firewall/filtering?

 

Peter

 

I'd always recommend looking for individual users searching for this such as "how to ddos myself" / "DDoS stress tester" etc. Sometimes catching them before they've done it will mean less pain in the long run.

 

Outward bound from your network also look for machines sending high volumes of traffic and what type of traffic. They could be compromised machines flooding data out of your network attempting to attack somewhere else . DNS and ICMP are common protocols used in DDoS flood attacks.

 

Dave

[rob_coles]

With the help of exa & smoothwall we hid our ip external ip from what’s our up etc which nipped our students in the bud after suffering numerous stress tests.

 

Thanks for the update.

[Clansman]

Part 2 of 2

Finally, let me say that Exa has a zero-tolerance policy on DDOS, and all these attacks will be reported to the appropriate authorities. A DDOS is a criminal offence, and we will treat it, and act on it as one. We believe ISP, The Government and the schools can play a massive part in educating people and make it clear that this sort of criminal behaviour will simply not be tolerated.

 

Hi All,

 

This is my first post here and was directed to it from a colleague in the education IT sector. I'm from the NCA's National Cyber Crime Unit (Cyber Prevent Team) and one of the my roles is the education liaison officer. We are well aware of the issue of young students flexing their cyber muscles and often the schools are becoming victims. I've been working with PSHE for the last 12 months and launched the below in September:

 

https://www.pshe-association.org.uk/curriculum-and-resources/resources/exploring-cybercrime-ks3-lesson-plans-national

 

The resource is also on the http://www.NEN.gov.uk site. There is further work planned through the NCCE, NCSC and TES with assistance from ROCU cyber officers.

 

DDoS shouldn't be ignored and is in fact a s3 CMA offence which attracts a 10 year sentence or fine or both in the most severe circumstances. However through Cyber Prevent we aim to divert and re-educate suspected offenders as it's not in anyone's interests to criminalise young students when an alternative approach may be more appropriate and productive. But of course it's on a case by case basis. I'm already linking in with a school firewall provider to establish what further data analysis is possible from their DDoS attacks and my aim will be to reach out to others in this sector to try and establish the national picture.

 

If anyone's attending EGX in London this weekend we have a Prevent stand under "Cyber Choices".

[EXA_Mark]

Hi @Clansman,

 

Firstly welcome to Edugeek and thank you for the post, as you can tell it is something we feel very strongly about.

 

We are developing some courses (for free) as part of our Exa Foundation, http://www.exa.foundation where each year we run hundreds of events for schools throughout the UK, if you'd like to get involved, or chat with me/Exa on anything we can jointly do to educate, please PM me or give me a ring on 0345 1451234

[SchoolsBroadband]

Thanks @Clansman.

 

We've already been approached by the NCA and are meeting with you next month.

 

All ISPs get hit by DDoS's and I personally welcome your approach. Particularly around educating.

 

Dave

[Clansman]

Also my hunch is that schools are experiencing DoS attacks rather than DDoS as per the tile of this thread. Most of you will probably know the difference but briefly the former is an attack from one sole computer and should be relatively easy to defend. The latter requires a network of computers infected by malware so they can be controlled by the attackers command and control server to launch the DDoS attack ie a Botnet. So a DDoS will be more impactive as can be imagined. I know it may sound a bit like semantics but there is a difference which I thought was worth pointing out

[mavhc]

Seems like the main issue people had was with phones, as real time audio is where you really notice packet loss, and specifically Gamma (seems like a lot of schools use them), is it possible to work with them to get a peering connection?

[SchoolsBroadband]

Gamma offer private peering although I think they charge for it which is annoying and you then have to pay the DC for the cross connect where the prices have just gone up massively in Equinix data centres.

 

That said we have a private interconnect with our wholesale voip suppliers for the reason you mention and the ability to do end to end qos.

 

I'm not sure if Gamma offer peering over LINX or Lonap.....

[howartp]

We are putting together a top tips guide just for this very reason Peter, as soon as it is done, I'll let you have it, and post it on here too. Probably late next week.

 

Good morning Mark

 

Firstly, did these tips ever get written or disseminated?

 

Secondly, please could you advise the appropriate name and email address (pm or publicly) for sending a formal query to Exa? We’ve been DDoS victims for several days and management want more answers than I can give.

 

Many thanks

Peter

South Craven