Consistently poor experience with EXA Networks

[Katy]

I concur, however will add it won't hurt to get a bundle from someone who knows what they're doing. For instance, with Wave9 they provide us a Sophos XG and on installation give us the option of having it basically fully managed by them, leaving it entirely to us to fiddle with/break, or a mixture. As it's a hardware box on site, we're free to do as we wish with it and are also safe in the knowledge that support from the guys at W9 is good enough to get a quick & reliable answer regarding the filtering if we need it; upshot of this is we have a bundle with support for both but don't have to put up with issues regarding cloud filtering which seems to be more of a factor than bundled filtering.

From my experience as one school on a shared filtering platform, vs having our own, all the problems come down to the fact multiple schools are sharing a filter. Doesn't matter how much you spend on hardware, the filtering company will go "Yeah it can support 1Gbps and 10,000 users if you use $hardware" only to discover, no it can't. So by all means use a bundle so long as it's a filtering appliance dedicated to you.

[DJ-1701]

Since switching over to them we are not able to use the Microsoft Store and they cant fix it - we have to tell Teachers to take their laptops home to use the Microsoft Store which is nuts!

 

@Fazza, I haven't had any issue with the Microsoft Store at all, we are using the Education version. Is that the one you are using as well?

@Fazza, there appears to be a 'new' tool on my Surfportect Portal when logging in under 'Customer Tools' called 'Application Controls', which has an option for 'Microsoftstore'. Have you tried enabling that?

[mavhc]

> > The Surfprotect web internet is pretty good

> How can it be 'pretty good' when you also said this in the same thread:

 

that should have said Interface, not Internet

 

> SurfProtect Quantum gives me a "something has gone wrong" message maybe once or twice a day

 

This is the main issue, it's been rather bad today, apparently they're fixing it.

 

> sometimes it just says "error" when you don't have permission, instead of saying why

 

Admin UI problem, not filtering problem

 

> the database of categories of websites isn't great, have to reclassify quite often.

 

Well, don't have much data on how bad every other classifier is.

 

> Personally I think it's pretty awful.

 

> Our users are used to getting a message saying there is a problem with the backend that most dont report it to us any more, not getting on to a website 2 or 3 times a day has become the norm.

 

How often does pressing reload not fix it?

 

We specifically went with Exa to avoid paying twice for web filtering boxes/subscriptions for 2 sites. With just one site might not have; with a main secondary and multiple primary might not have.

 

If you're using an unfiltered IP are there any problems? Is it only SurfProtect?

[fiza]

I'm a huge supporter of LGfL/TRUSTnet. In my opinion, LGfL/TRUSTnet bring the most value added product for schools. If anything, these days, robust and stable broadband is their most basic offering! You now get some amazing products bundled in to the package for free:

 

Free Meraki MDM licenses (300 for primaries, 700 for for secondaries)

Free Sophos Anti-Virus with Intercept X

Free Sophos Server Advanced Anti-Virus with Intercept X

Free User Provisioning for G Suite and Office 365

Free Neverware Cloud Ready licenses (30 for primaries, 100 for secondaries) with discounts for further licenses

Free Malwarebytes licenses

Free Egress Switch licenses (5 for primaries, 15 for secondaries)

Built in fully cloud based web filtering

Free training on lots of key curriculum areas in schools (delivered at LGfL HQ or at your school by one of the LGfL curriculum experts)

Free training on Office 365, G Suite (delivered at LGfL HQ or at your school by one of the LGfL Digital Cloud Champions)

100s of free curriculum content via the LGfL Content platform

Free entry to the LGfL Conference - Annual curriculum conference

Free entry to the LGfL Digisafe Conference - Annual Safeguarding conference

 

And these are just the ones I can recall off the top of my head!

 

Under the leadership of @JohnJackson, LGfL have grown and improved leaps and bounds and some of the prospective new features, upgrades and packages coming are gonna make it, what I think, is one of the best all round broadband package to schools!

Totally agree with this. We are with LGfL and our Primary are moving over to them next year. Solid connection and good customer service. Since @JohnJackson took over the price has come down but the number of included services has gone up. Over 6 years that we have been with LGfL I cannot remember any significant times when we lost internet connectivity.

[FN-GM]

I'm a huge supporter of LGfL/TRUSTnet. In my opinion, LGfL/TRUSTnet bring the most value added product for schools. If anything, these days, robust and stable broadband is their most basic offering! You now get some amazing products bundled in to the package for free:

 

Free Meraki MDM licenses (300 for primaries, 700 for for secondaries)

Free Sophos Anti-Virus with Intercept X

Free Sophos Server Advanced Anti-Virus with Intercept X

Free User Provisioning for G Suite and Office 365

Free Neverware Cloud Ready licenses (30 for primaries, 100 for secondaries) with discounts for further licenses

Free Malwarebytes licenses

Free Egress Switch licenses (5 for primaries, 15 for secondaries)

Built in fully cloud based web filtering

Free training on lots of key curriculum areas in schools (delivered at LGfL HQ or at your school by one of the LGfL curriculum experts)

Free training on Office 365, G Suite (delivered at LGfL HQ or at your school by one of the LGfL Digital Cloud Champions)

100s of free curriculum content via the LGfL Content platform

Free entry to the LGfL Conference - Annual curriculum conference

Free entry to the LGfL Digisafe Conference - Annual Safeguarding conference

 

And these are just the ones I can recall off the top of my head!

 

Under the leadership of @JohnJackson, LGfL have grown and improved leaps and bounds and some of the prospective new features, upgrades and packages coming are gonna make it, what I think, is one of the best all round broadband package to schools!

 

 

It is not free. Its just part of the service package.

[fiza]

It is not free. Its just part of the service package.

When you work out the price of all those services if bought separately and then add that cost to another providers basic service of fibre and firewall then you see what a good deal it is.

[FN-GM]

When you work out the price of all those services if bought separately and then add that cost to another providers basic service of fibre and firewall then you see what a good deal it is.

 

I didn't say it was a bad deal. On the contrary it is a good deal. Plus you have to factor in the time saved in purchasing everything separate. My point is it isn't free.

[Fazza]

How often does pressing reload not fix it?

 

This is not normal behaviour and is not how an internet connection should work. Just because it happens so often it has become the norm does not make it normal to have to do this! It's their filtering system they they created themselves that is the issue and to say just press reload is absurd. You dont have to do that with Virgin Media, Plusnet, SchoolsBroadband, RM Internet - you didnt even have to do that on Freeserve or AOL!!

[mikkydoos]

FreeServe.... is it still 1996 ??

[mikkydoos]

Funny that Exa aren't all over this thread like a rash..... They usually are when anyone mentions them.

[mavhc]

This is not normal behaviour and is not how an internet connection should work. Just because it happens so often it has become the norm does not make it normal to have to do this! It's their filtering system they they created themselves that is the issue and to say just press reload is absurd. You dont have to do that with Virgin Media, Plusnet, SchoolsBroadband, RM Internet - you didnt even have to do that on Freeserve or AOL!!

 

Of course it's not, but does it fix it?

 

Which proxy are you using? AD, SSL or NoSSL?

[Fazza]

FreeServe.... is it still 1996 ??

 

Yep! I've just purchased an i486DX2-50 laptop with 8MB of RAM and a massive 350MB hard disk! Just installed MS-DOS 6.22 and Windows 3.11 for Workgroups

 

Edited December 20, 2018 by Fazza

[epoch_roots]

Funny that Exa aren't all over this thread like a rash..... They usually are when anyone mentions them.

 

Most of the issues raised in this thread are indefensible so they're probably just laying-low. A response would require an admittance of their issues and a plan for addressing them...

 

I'm just glad that there is a recent thread where people can share their real-world experiences using their service to counteract the shilling that seems to happen all the time.

[mavhc]

Can we actually gather some data then? What service are you using, what proxy, what fails?

[EXA_Mark]

Our internet is up and down today but it's been like this for a few months now.

It's been multiple issues from the stormshield they supplied not being able to handle our load to their R&D team deploying updates that bring their network down or their filtering product deciding to stop working. Today apparently its a hardware failure as I'm guessing they have zero redundancy in their network.

We've also looked for opinions on schoolsbroadband but they have similar issues from what I'm told. The issues aren't localised to us as we have another school around the corner that gets hit at the same time.

 

I don't get it - they seem to get a lot love here which is probably the most perplexing part.

 

I couldn't see them functioning as an organisation for very long if they were offering services to corporate entities. They'd be bankrupted by paying out for SLA breaches. Where does the love come from and why does everyone tolerate the service they provide? We're stuck with them for the next 9 months and I can't see it improving at all.

 

Sorry if this is looks like a rant but I've come from Industry and had maybe two or three instances of downtime in 10 years compared to the 2 or 3 this week alone with Exa.

 

Sorry for the delayed reply on this, it has just been brought to my attention in the past hour, and I'd not been on the forum for a few days.

 

Let me try and answer this, and a few other messages within this thread. But before I do that, let me firstly apologise to any of our customers who have not had the level of service you have come to expect from Exa, or we expect ourselves to deliver. For those customers who have had disruption with SurfProtect over the past couple of weeks I am deeply sorry for that.

 

Before I get into this in depth, I just want to clarify a point that has been made a few times on here, the network at Exa has not been down, at all, the internet connectivity has remained up at all times (I am not saying individual customers have not had unrelated outages such as a fibre cut or an Openreach fault), but that the "network fault" mentioned is specifically relating to either SurfProtect or related DNS issues.

 

With that out of the way, let me give you a (I apologise again), long answer.

 

As many of you know or can appreciate, software development is never simple, especially when we are working on a living, growing system such as SurfProtect, where a bug or outage is immediately impactful.

 

I wanted to give you a bit of an update about where we are with SurfProtect, what we have already done and are doing to make sure we have the stability back quickly, and what we are doing going forward.

 

The last quarter of 2018 has seen a massive increase in customers using SurfProtect Quantum. With more than 500 new schools moving on the new Quantum platform. 2019 will see our team contacting existing SurfProtect Cloud customer and migrating them and offering to do the same for our Fusion customers who are still using the Stormshield firewalls. It is therefore not surprising to hear that the load on our infrastructure has more than doubled in around 2 months, and with the migration of SurfProtect Cloud customers, plus the usual Jan-April school moves, the expectation is that we will see the load a least quadruple before the end of Q1 2019.

 

And we have to be ready for this, which is what has been causing some of the issues over the past couple of weeks.

 

About half of the SurfProtect development team is busy adding further resilience and new features to the product, the others have been hard at work implementing a solution which will allow us to take such an increase in traffic. SurfProtect Quantum is currently filtering nearly 5 Gbps of traffic every day at peak time, with this traffic expected to reach over 10 Gbps before the end of Q1 and over 20 Gbps by the end of Q2.

 

This substantial growth, while very great from a business perspective, is not without its challenges. Our engineers are currently working hard adding new servers our customers can use, to make sure we have enough capacity way ahead of time to make sure this growth can be painless. This work should complete for early January (prior to the schools' return from the Christmas break), with some servers already installed in Manchester and London.

 

Also before the end of the year, our team will be able to provide per-customer Surfprotect AD and Proxy servers, in order to better spread the load across the SurfProtect infrastructure

 

Without going into very technical details, our proxy customers are currently using `ad.quantum.exa-networks.co.uk` and `proxy.quantum.exa-networks.co.uk` to get their traffic filtered. The browsers are then connecting to these destinations to get the traffic filtered.

 

In order to apply a divide and conquer approach to the issue, our engineers have developed a solution which allows us to provide different customers personalised answers.

 

The way the internet works is by resolving these name to unique computer IP. This process uses a system called DNS (which many of you will be familiar with) and is similar to looking up a phone number in the yellow pages.

 

The solution is quite similar to a busy call centre in say London, which to better help its customers open some new office in Bradford, and get the local directory (yellow pages) updated for all northern customer with the Bradford office number. In this analogy, we have to make sure that the issue of out-of-date copies of the yellow pages ringing the wrong number does not apply. That’s why the answer we give has to have a very short lifetime, to make sure the answer is always the right one not out of date.

 

Since early this week, when your customers are asking for the location of the SurfProtect proxies, our backend systems have been able to direct them to the nearest and most capable machine to deal with the traffic.

 

We are looking forward to using this feature to redirect customers with problematic applications toward some dedicated servers configured to help us diagnose the issue(s) without requiring any intervention from the customer themselves. Hopefully allowing for faster issue resolution.

 

Our analytics database, held at its peak this month well over 100 billion records (we are inserting 10s of thousands of records per second during the day ), which all our customer can now search in real-time. Because of the huge growth in those logs one of our main projects for this last quarter and next is to work on changing some of our underlying technologies.

 

This can occasionally cause some interruptions as we are deploying the new software or hardware to cope with this, and again, my sincerest apologies for this. We do our utmost to keep disruption to a minimum.

 

HTTPS now count for over 2/3 of all connections.

It continues to cause challenges for all filtering providers, especially when Google and now others, including CloudFlare which host around 10 million domains, are taking technical measure to actively prevent any form of traffic inspection, including for filtering. The team is therefore busy to pro-actively work on solving the challenge that future mass ESNI deployment will undoubtedly cause. https://blog.cloudflare.com/esni/

 

Unlike some filtering vendors who may decide to simply open cloud service providers such as Amazon Web Service, Microsoft Azure, CloudFlare or Google Cloud, SurfProtect identifies the actual service to make sure any accessed URL can be filtered logged and is available in the analytics for review. This is in line with requirement laid out by the government on Counter-Terrorism and the DfE but can cause massive headaches.

 

Over the last year, as an example, these were some of the challenges we had to figure out a way around.

• Microsoft Outlook relying on HTTPS connection failing to fall back to HTTP as was previously the case.

• TLS key negotiation issue which was required to be able to change the key negotiation cypher for certain releases of Windows 10.

• For security, many phone/tablet applications do verify the certificate authority used for signing the certificate, preventing HTTPS filtering!

• Many applications not expecting HTTP filtering and using port 443 (SSL/HTTPS) for custom protocol, each of them requiring a unique and tailored solution, and even different version of the same product, such as Twitter works completely differently at a network level on an App to the web.

 

While most of this work has been performed in stealth mode with little or no impact, it also tends to go unnoticed and can sometimes look like there is no development on SurfProtect. A statement that could not be further away from the truth. The development team continues to expand and we recently brought in new operations and project manager to oversee the deployment of the many new features and updates.

 

The team is hard at work to make sure the new version of TLS (3.0) the protocol securing HTTPS connection can be analysed as some applications are now switching to only support TLS 3.0. As well working toward other new features which I hope to announce very soon.

 

To those on the thread who had put some heart warming comments, thank you, I think many people forget that the developers are real people, who have all now seen this thread, and critiscm is hard to take sometimes for all of us, but it is important for people to be aware of it, and for them to see the positive comments is most welcome. We will continue to do our best to make sure that those who have not had the best experience recently, get to see the improvement quickly.

Edited December 20, 2018 by EXA_Mark

[LukeRowberry]

We're not an exa customer but I have to say it is excellent to see such a detailed response from a company.

[round2it]

As mentioned earlier it is good that exa continue to work on the product.

 

I do rate exa very highly.

Big deal the YouTube clips of Christmas jingles stuttered today

it could be worse they could be run by mls/capita.

 

Keep up the good work.

[_techie_]

Agree with localZuk here... buy feed from Exa and then provision your own filtering solution from someone else. We use EXA and Smoothwall UTM. Can’t fault EXA tbh. Smoothwall do cloud filtering if that’s your option..

[mavhc]

If you're not using their filtering why not pick someone cheaper?

[mavhc]

Surfprotect has been pretty good recently, although did fall over on friday and monday, it's moved from failing more often for a short amount of time to not failing often, but when it does, more harshly. Guess it's due to updates being installed rather than limits of capacity now. Looks like it will be more stable once those are complete.

 

My time based filtering hack (wpad file that checks time and sends to a different proxy at break times so games work) has annoyed teachers who suddenly got youtube etc blocked at break times. So now GPO sets different PAC files for staff vs pupils, one with the time settings, one without.

[localzuk]

If you're not using their filtering why not pick someone cheaper?

Depends where you are as to whether someone is cheaper or not. EXA were the second cheapest for our needs when we did our comparisons. The cheapest was a company being run out of someone's house so we were a little dubious!

[mavhc]

Interesting, do they always use OpenReach, does OpenReach always charge the same?

[localzuk]

Interesting, do they always use OpenReach, does OpenReach always charge the same?

Here? Yes, its always OpenReach. Even when our tails were apparently not OpenReach, OpenReach actually did all the work... But no, OpenReach charge for their products by zone - different zones have different pricing structures. But I imagine they'd sell to resellers at the same price in that zone.

Edited January 15, 2019 by localzuk

[mavhc]

Sadly didn't get a discount even though a site already had openreach fiber, so they only changed a sticker at my end

[epoch_roots]

Surfprotect has been pretty good recently, although did fall over on friday and monday, it's moved from failing more often for a short amount of time to not failing often, but when it does, more harshly. Guess it's due to updates being installed rather than limits of capacity now. Looks like it will be more stable once those are complete.

 

 

Ours was down again this morning. Apparently due to another update deployed last night.

Edited January 17, 2019 by epoch_roots