Jump to content

Consistently poor experience with EXA Networks

epoch_roots
 Share

Recommended Posts

Fazza

Fazza

Posted
Posted
Sorry for the delayed reply on this, it has just been brought to my attention in the past hour, and I'd not been on the forum for a few days.

 

Let me try and answer this, and a few other messages within this thread. But before I do that, let me firstly apologise to any of our customers who have not had the level of service you have come to expect from Exa, or we expect ourselves to deliver. For those customers who have had disruption with SurfProtect over the past couple of weeks I am deeply sorry for that.

 

Before I get into this in depth, I just want to clarify a point that has been made a few times on here, the network at Exa has not been down, at all, the internet connectivity has remained up at all times (I am not saying individual customers have not had unrelated outages such as a fibre cut or an Openreach fault), but that the "network fault" mentioned is specifically relating to either SurfProtect or related DNS issues.

 

With that out of the way, let me give you a (I apologise again), long answer.

 

As many of you know or can appreciate, software development is never simple, especially when we are working on a living, growing system such as SurfProtect, where a bug or outage is immediately impactful.

 

I wanted to give you a bit of an update about where we are with SurfProtect, what we have already done and are doing to make sure we have the stability back quickly, and what we are doing going forward.

 

The last quarter of 2018 has seen a massive increase in customers using SurfProtect Quantum. With more than 500 new schools moving on the new Quantum platform. 2019 will see our team contacting existing SurfProtect Cloud customer and migrating them and offering to do the same for our Fusion customers who are still using the Stormshield firewalls. It is therefore not surprising to hear that the load on our infrastructure has more than doubled in around 2 months, and with the migration of SurfProtect Cloud customers, plus the usual Jan-April school moves, the expectation is that we will see the load a least quadruple before the end of Q1 2019.

 

And we have to be ready for this, which is what has been causing some of the issues over the past couple of weeks.

 

About half of the SurfProtect development team is busy adding further resilience and new features to the product, the others have been hard at work implementing a solution which will allow us to take such an increase in traffic. SurfProtect Quantum is currently filtering nearly 5 Gbps of traffic every day at peak time, with this traffic expected to reach over 10 Gbps before the end of Q1 and over 20 Gbps by the end of Q2.

 

This substantial growth, while very great from a business perspective, is not without its challenges. Our engineers are currently working hard adding new servers our customers can use, to make sure we have enough capacity way ahead of time to make sure this growth can be painless. This work should complete for early January (prior to the schools' return from the Christmas break), with some servers already installed in Manchester and London.

 

Also before the end of the year, our team will be able to provide per-customer Surfprotect AD and Proxy servers, in order to better spread the load across the SurfProtect infrastructure

 

Without going into very technical details, our proxy customers are currently using `ad.quantum.exa-networks.co.uk` and `proxy.quantum.exa-networks.co.uk` to get their traffic filtered. The browsers are then connecting to these destinations to get the traffic filtered.

 

In order to apply a divide and conquer approach to the issue, our engineers have developed a solution which allows us to provide different customers personalised answers.

 

The way the internet works is by resolving these name to unique computer IP. This process uses a system called DNS (which many of you will be familiar with) and is similar to looking up a phone number in the yellow pages.

 

The solution is quite similar to a busy call centre in say London, which to better help its customers open some new office in Bradford, and get the local directory (yellow pages) updated for all northern customer with the Bradford office number. In this analogy, we have to make sure that the issue of out-of-date copies of the yellow pages ringing the wrong number does not apply. That’s why the answer we give has to have a very short lifetime, to make sure the answer is always the right one not out of date.

 

Since early this week, when your customers are asking for the location of the SurfProtect proxies, our backend systems have been able to direct them to the nearest and most capable machine to deal with the traffic.

 

We are looking forward to using this feature to redirect customers with problematic applications toward some dedicated servers configured to help us diagnose the issue(s) without requiring any intervention from the customer themselves. Hopefully allowing for faster issue resolution.

 

Our analytics database, held at its peak this month well over 100 billion records (we are inserting 10s of thousands of records per second during the day ), which all our customer can now search in real-time. Because of the huge growth in those logs one of our main projects for this last quarter and next is to work on changing some of our underlying technologies.

 

This can occasionally cause some interruptions as we are deploying the new software or hardware to cope with this, and again, my sincerest apologies for this. We do our utmost to keep disruption to a minimum.

 

HTTPS now count for over 2/3 of all connections.

It continues to cause challenges for all filtering providers, especially when Google and now others, including CloudFlare which host around 10 million domains, are taking technical measure to actively prevent any form of traffic inspection, including for filtering. The team is therefore busy to pro-actively work on solving the challenge that future mass ESNI deployment will undoubtedly cause. https://blog.cloudflare.com/esni/

 

Unlike some filtering vendors who may decide to simply open cloud service providers such as Amazon Web Service, Microsoft Azure, CloudFlare or Google Cloud, SurfProtect identifies the actual service to make sure any accessed URL can be filtered logged and is available in the analytics for review. This is in line with requirement laid out by the government on Counter-Terrorism and the DfE but can cause massive headaches.

 

Over the last year, as an example, these were some of the challenges we had to figure out a way around.

• Microsoft Outlook relying on HTTPS connection failing to fall back to HTTP as was previously the case.

• TLS key negotiation issue which was required to be able to change the key negotiation cypher for certain releases of Windows 10.

• For security, many phone/tablet applications do verify the certificate authority used for signing the certificate, preventing HTTPS filtering!

• Many applications not expecting HTTP filtering and using port 443 (SSL/HTTPS) for custom protocol, each of them requiring a unique and tailored solution, and even different version of the same product, such as Twitter works completely differently at a network level on an App to the web.

 

While most of this work has been performed in stealth mode with little or no impact, it also tends to go unnoticed and can sometimes look like there is no development on SurfProtect. A statement that could not be further away from the truth. The development team continues to expand and we recently brought in new operations and project manager to oversee the deployment of the many new features and updates.

 

The team is hard at work to make sure the new version of TLS (3.0) the protocol securing HTTPS connection can be analysed as some applications are now switching to only support TLS 3.0. As well working toward other new features which I hope to announce very soon.

 

To those on the thread who had put some heart warming comments, thank you, I think many people forget that the developers are real people, who have all now seen this thread, and critiscm is hard to take sometimes for all of us, but it is important for people to be aware of it, and for them to see the positive comments is most welcome. We will continue to do our best to make sure that those who have not had the best experience recently, get to see the improvement quickly.

 

We have to send our Teachers home to access the Microsoft Store. All new Windows 10 1809 installations are partially in the US language as we do not have access to the Microsoft Store and some UK settings are downloaded from that shortly after the first log on - as we cannot access the Microsoft Store it constantly comes up interrupting lessons.

 

The above quote is what we have been emailed in response to why are we still not able to access the Microsoft Store - please can you explain which part of this quote explains why we can not access the Microsoft Store as we cannot see any reason mentioned in it?

 

Thank you.

DJ-1701

DJ-1701

Posted
Posted (edited)
We have to send our Teachers home to access the Microsoft Store. All new Windows 10 1809 installations are partially in the US language as we do not have access to the Microsoft Store and some UK settings are downloaded from that shortly after the first log on - as we cannot access the Microsoft Store it constantly comes up interrupting lessons.

 

@Fazza, I haven't had any issue with the Microsoft Store at all, we are using the Education version. Is that the one you are using as well?

 

@Fazza, there appears to be a 'new' tool on my Surfportect Portal when logging in under 'Customer Tools' called 'Application Controls', which has an option for 'Microsoftstore'. Have you tried enabling that?

@Fazza, have you tried the option above?

 

Also if you are using Windows 10 from VLSC, you can select English International in the language list for download, this sets UK English as default, might be helpful for future installs.

 

 

 

Additional:

You're not the only one with a Microsoft Store issue, the issue has also been raised here.

http://www.edugeek.net/forums/internet-related-filtering-firewall/203314-surfprotect-quantum-accessing-microsoft-store-windows-10-a.html

Edited by DJ-1701
EXA_Mark

EXA_Mark

Posted
Posted
Anyone experiencing a google verification error? Been having this since friday with no support update despite having it logged.

 

A quick update on this, which as you said was raised on Friday to our support team.

 

Google has made changes on how they look at the pool of IP addresses coming from requesters, such as SurfProtect, and this has caused the solution that was in working on Recaptcha/verification, to have issues. This was passed to our SP Dev team and we expect a solution to be in place within 24 hours.

 

Sorry for any inconvenience this is causing anyone, and we will have a fix in place for Googles changes very soon.

 

I will keep you posted.

howartp

howartp

Posted
Posted
@exa_mark - our web access just went offline today around 3pm for 15 minutes or so. Our circuit was active as we could still ping the internet so it seems specific to Exa's filtering. What was the issue?

 

(We're with EXA but no filtering - our connection didn't drop so that agrees with your prognosis)

mavhc

mavhc

Posted
Posted (edited)

Anyone else on exa have a problem posting to edugeek, I click post quick reply and often I get back a chrome error message, pressing back and post a few more times fixes it

 

"This site can’t be reached The connection was reset."

Edited by mavhc
  • 2 weeks later...
epoch_roots

epoch_roots

Posted
  • Author
Posted

Nope, but we did just have another outage caused by SurfProtect - seems to have cleared up now.

Props to Exa for actually announcing it on their status page this time!

  • Thanks 1
round2it

round2it

Posted
Posted

yea 10 min outage but we seem to be having one outage a week, i am a fan of exas services but is this a case of over subscription for surf protect.

victim of own sucess.

  • Thanks 1
jonbd

jonbd

Posted
Posted
We were affected by the outage as well yesterday. Also had the Google verification problem just before it all went down, but seems to have been OK since then.
  • Thanks 1
hallb15

hallb15

Posted
Posted
I got the status email warning that Surfprotect was experiencing problems but ours was OK this time. We haven't had the google verification issue since last year either. :cool2:
EXA_Mark

EXA_Mark

Posted
Posted
We were affected by the outage as well yesterday. Also had the Google verification problem just before it all went down, but seems to have been OK since then.

 

I am really sorry you were impacted yesterday morning, on this. Only customers on SurfProtect Fusion were affected, so I am going to assume you are on this. If you would like to move over to SurfProtect Quantum, just PM me and I'll get your account manager to get in touch and sort it out.

 

The outage on Fusion yesterday was a total of 6 and a half minutes, followed very shortly after by 3 minutes. The root cause of this was identified and a fix is being pushed out today.

 

You may have also seen a very short outage this morning (again Fusion only), this was human error and was resolved in a couple of minutes.

 

Once again, really sorry for any disruption this may have caused.

 

One thing that is a puzzle though is you mentioned about Google Verification problems. This was not a symptom of yesterdays problem, and we had no other reports before or since. Can you give our support team a call on 0345 1451234 and they'll have a chat with you about it. Problems on Google verification can be caused by numerous issues, but primarily it is the IP or IP range that the request is coming from that Google has the biggest issue with. On Fusion, Google see you direct IP addresses, not a pooled or proxy range.

EXA_Mark

EXA_Mark

Posted
Posted
I got the status email warning that Surfprotect was experiencing problems but ours was OK this time. We haven't had the google verification issue since last year either. :cool2:

 

Thanks for the comment. If you have signed up to our status notifications you will get notifications on anything SurfProtect related, however, the brief problem yesterday only affected SurfProtect Fusion, so if you are on Cloud, Proxy or Quantum, you would not have been impacted. I also suspect the Google issue mentioned by the other person, was specific to that school as we had not other support tickets or calls about it before the issue yesterday or since.

 

We are going to update the status notification system in the future, so you can just have notifications on the variant you have, ie Quantum, as opposed to getting something that may not be related or affect you.

  • Thanks 1
EXA_Mark

EXA_Mark

Posted
Posted
Sorry about the short outage yesterday, which was just under ten minutes, which is still about ten minutes too much. This only affected customers on SurfProtect Fusion, and we do try and get any issues on the Status page asap. We know what caused the fusion issue yesterday and a fix will be pushed out tonight to make sure this can't happen again.
EXA_Mark

EXA_Mark

Posted
Posted
yea 10 min outage but we seem to be having one outage a week, i am a fan of exas services but is this a case of over subscription for surf protect.

victim of own sucess.

 

Thank you for the comment, and I just want to say, any outage for us is unacceptable even ten minutes. A fix was identified for this yesterday, which is for a very obscure bug, that hadn't raised its head before, and this was only on SurfProtect Fusion, so Quantum, Proxy and Cloud were unaffected. The fix is being pushed out tonight.

 

Obviously, there is for any company (and I have commented on it on this thread before), a chance that as you grow and are successful you encounter problems or challenges you've not seen before. That's why we have spent so much additional time and resources on SurfProtect recently to make sure we are in a position where this doesn't affect our customers.

 

As I have said to others, I'm really sorry for any inconvenience, we will always strive to do better, at everything we do.

  • Thanks 1
jonbd

jonbd

Posted
Posted
I am really sorry you were impacted yesterday morning, on this. Only customers on SurfProtect Fusion were affected, so I am going to assume you are on this. If you would like to move over to SurfProtect Quantum, just PM me and I'll get your account manager to get in touch and sort it out.

 

The outage on Fusion yesterday was a total of 6 and a half minutes, followed very shortly after by 3 minutes. The root cause of this was identified and a fix is being pushed out today.

 

You may have also seen a very short outage this morning (again Fusion only), this was human error and was resolved in a couple of minutes.

 

Once again, really sorry for any disruption this may have caused.

 

One thing that is a puzzle though is you mentioned about Google Verification problems. This was not a symptom of yesterdays problem, and we had no other reports before or since. Can you give our support team a call on 0345 1451234 and they'll have a chat with you about it. Problems on Google verification can be caused by numerous issues, but primarily it is the IP or IP range that the request is coming from that Google has the biggest issue with. On Fusion, Google see you direct IP addresses, not a pooled or proxy range.

 

I raised a ticket with support about it yesterday (who were really good as always!) but as the problem seemed to disappear once the outage was over and done with, we decided to leave it for now and see what happens. Apart from these little outages, we've been very happy with Exa.

  • Thanks 1
Theldron

Theldron

Posted
Posted
I know not everyone will agree but have been using SWGFL for over 10 years and cannot complain about their internet connection, stability or support. Not a fan of their filtering, particularly their UBF as have issues with that, the school is planning on moving to a Sophos UTM soon.
steve_forbes

steve_forbes

Posted
Posted
I know not everyone will agree but have been using SWGFL for over 10 years and cannot complain about their internet connection, stability or support. Not a fan of their filtering, particularly their UBF as have issues with that, the school is planning on moving to a Sophos UTM soon.
Would love to hear your feedback on UBF, we are currently doing some work to make this easier to deploy so any additional feedback would really help.

 

Thanks

Steve (Product Manager)

  • 4 weeks later...
epoch_roots

epoch_roots

Posted
  • Author
Posted

Anyone else having surfprotect issues today?

It's blocking some sites with the "we're on it, site is currently being classified" message which just sits there refreshing itself indefinitely without ever classifying.

It's not happening for all sites but randomly. It's just locked me out of admin.gsuite.com now yet it was working earlier?

mavhc

mavhc

Posted
Posted
Anyone else having surfprotect issues today?

It's blocking some sites with the "we're on it, site is currently being classified" message which just sits there refreshing itself indefinitely without ever classifying.

It's not happening for all sites but randomly. It's just locked me out of admin.gsuite.com now yet it was working earlier?

 

I've had that before, make sure you've emailed them

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing


×
×
  • Create New...