EXA Networks - The DDOS Fiasco

[Tefters]

EXA Networks have been under DDOS attacks on and off for a few months now and as of late have been hit pretty hard by them, yesterday and today are great examples as our WAN is pretty much crippled on a voice and data through web filtering front!

 

Trying to speak with anyone technical to get answers is proving a headache and we are just being advised to "wait" while they shutdown the customers being targeted.

 

What i'm trying to get answers on is whether it is the customers themselves or whether the attacks are aimed at EXA through the customer as entry holes.

[gmonks]

We're supplied by Exa too and have been affected by this and its pretty frustrating.

 

I did email yesterday asking if they were doing anything to prevent the attack and received bit of a non-answer.

[DJ-1701]

Recalling previous updates on the Exa status page, it appears to be certain customers being targeted. @exa_mark would most likely know more.

[GTX]

I rang them this morning asking for an update. Today's DDOS wasnt even on the status page until i told them its not.

 

A network that big needs DDOS protection. Its not cheap but its the world we live in now.

[CyBeRkId2002]

Also desperate for an update on this! We are part of a MAT and have 4 schools on Exa, with another 2 due to join in the next fortnight. Have received great service up until this point but this needs resolving as our MIS and voice (in some circumstances) relies on having a solid connection!

[GTX]

Also desperate for an update on this! We are part of a MAT and have 4 schools on Exa, with another 2 due to join in the next fortnight. Have received great service up until this point but this needs resolving as our MIS and voice (in some circumstances) relies on having a solid connection!

 

Couldnt agree more. We are in our second year with Exa and nothing but positives.

 

Our internet and other services are fine but its really hurting our Gamma SIP's

[GTX]

New update:

 

We are experiencing a resurgence of traffic from the DDoS attack, our engineers are working on mitigating the issue.

 

[localzuk]

Can I just point out that mitigating an actual DDoS (a distributed one, not the misnamed ones) is pretty tough. It isn't something you can flip a switch with and resolve.

 

It doesn't matter which ISP you go with, if you are a school you will face the risk of internet disruption caused by DDoS's. Schools get targeted all the time, and this affects their ISPs, which affects other schools. Most schools are clustered on a fairly short list of ISPs these days - probably a couple of dozen ISPs across the country, so you can be sure they've all had this same problem at one point or other. I know we used to see similar issues with our RBC connection before we moved to Exa. Only, we now experience it far less. In fact, we only noticed something wrong on the 26th, and haven't had any issues since. Today's one hasn't affected us at all.

[GTX]

But there are systems on the market that prevent these things. As i stated before, this day in age DDOS is an everyday thing. CityFibre/Exa should take a step back and look at the pro/cons of the system. Its like not having a firewall, Its fine until it all goes wrong.

[localzuk]

But there are systems on the market that prevent these things. As i stated before, this day in age DDOS is an everyday thing. CityFibre/Exa should take a step back and look at the pro/cons of the system. Its like not having a firewall, Its fine until it all goes wrong.

There is no such system that "prevents" DDoS's. There are systems that mitigate them, that's all - with varying degrees of success. In reality, big DDoS attacks can only be mitigated upstream. So, if Exa is having a connection hit, then whoever they connect to would be the ones having to do the mitigation - as it is too late if the data is on Exa's network already.

[atcoates]

Our whole LA was down for two weeks at the start of the year with a similar batch of attacks.

[GTX]

There is no such system that "prevents" DDoS's. There are systems that mitigate them, that's all - with varying degrees of success. In reality, big DDoS attacks can only be mitigated upstream. So, if Exa is having a connection hit, then whoever they connect to would be the ones having to do the mitigation - as it is too late if the data is on Exa's network already.

 

I think your just trying to split hairs now. End of the day with Exa supplying schools and private sectors and there will be a good chance of a DDOS attack. All im trying to say is now is the time to look at how to "Prevent" these hitting the customers.

[crc-ict]

Unfortunately disruptions like this seem to be getting more and more common. It is frustrating on two counts, the main one being the disruption to operations, but also on a personal professional level because people are starting to lose faith in 'the system' and are starting to think the explanation of 'problems at the Internet provider' are just an excuse.

[Tefters]

Well the stability of the connection is good enough for web surfing and streaming but for external VOIP calls to our SIP provider (Gamma) we are pretty much crippled.

Majority of calls failing both incoming and outgoing, if any do succeed they are so robotic.

 

Can anyone at Exa from a high level @exa_mark give us any glimmer of hope on this as the communications coming out from yourselves is near enough NIL. Just putting "we are working on mitigating" is not really an update when I have just got off the phone grilling the poor 1st line person as to what I am seeing affected and having confirmation that VOIP is now the identified traffic being specifically affected.

[SchoolsBroadband]

I do feel for Mark and co as it appears they have been hit by some pretty big DDoS's which can be a real pain to manage.

 

There are lots of things ISP's can do to mitigate them. I believe Exa uses ExaBGP to help mitigate DDoS attacks which is their own BGP protocol they've designed and I hear is pretty good.

 

We get hit by DDoS's near enough everyday. A while ago we invested a large six figure sum (we haven't even announced it to our customers) into a specific anti-ddos mitigation platform. So far its never broken sweat but time will tell. What I can say is that it has got rid of some attacks which would have caused us problems in the past so from my point of view despite costing more than a Lamborghini Aventador it was money well spent.

 

I know a number of other ISPs which have bought the same platform as us and are also very happy with it.

 

Dave

[GTX]

Well the stability of the connection is good enough for web surfing and streaming but for external VOIP calls to our SIP provider (Gamma) we are pretty much crippled.

Majority of calls failing both incoming and outgoing, if any do succeed they are so robotic.

 

Can anyone at Exa from a high level @exa_mark give us any glimmer of hope on this as the communications coming out from yourselves is near enough NIL. Just putting "we are working on mitigating" is not really an update when I have just got off the phone grilling the poor 1st line person as to what I am seeing affected and having confirmation that VOIP is now the identified traffic being specifically affected.

Same

 

[E_G_R2]

So could this account for our very poor internet over the last couple of days as an EXA customer or is it just VOIP?

[GTX]

E_G_R2,

 

Its been up and down all day. Internet usage is fine for us. Just VoIP from Gamma.

[round2it]

Nearly 2 years with exa and the connection has been rock solid so im sure exa will figure this out but with everything online now it does hit us hard in schools.

 

What i would like is a statement from exa to give to the senior team so they know the issue is a isp level and not the school.

[Wubbalubbadub]

Love how Schools Broadband comment on the post before the supplier its actually about does! ha!

[round2it]

They might have been busy sorting the ddos out and not sat at a desk looking at edugeek posts.

[SchoolsBroadband]

I do love Edugeek

[k-strider]

We are still experiencing problems I can’t log into anything remotely and when I left work 1730 it was hit and miss if anything loaded

[free780]

JISC/Janet had similar a few years ago. They increased security and mitigations. They have increased costs due to the government pulling some funding.

Is it a case of you get what you pay for?

I used to work at a school and we never had any issues with Exa apart from someone cutting through some fibre and Exa called us downtime was only 2-3 hours.

[EXA_Mark]

Love how Schools Broadband comment on the post before the supplier its actually about does! ha!

 

I guess Dave @SchoolsBroadband has a lot more spare time to be on the forums than I do I also tend not to post into other supplier-specific issues threads such as SB has had over the past week with their filtering or in the past with DDOS too, as quite simply, as someone else in this thread wrote, all suppliers will have an issue from time to time, it is how they handle them, and how they respond to them.

 

Seriously though, I do try and keep an eye on the forums, but as you can all appreciate with your busy day jobs too, I don't always get to check them as often as I would like.

 

The post was brought to my attention yesterday evening as I was out of the office yesterday most of the day, and I am writing a fairly detailed long post now, which will be up on here very shortly. So please don't think I am ignoring my fellow edugeekers.