NASSTAR Connection Issues

[ITGURU]

Doesn't look like anyone has started this yet, but just to get an idea, who has also had issues with NASSTAR connectivity since yesterday afternoon? Seems to be affecting multiple regions.

What do you all do for backup lines if you have one and do you have manual or automatic failover?

Anyone else want to go help them resolve the issue?!

[robk]

It's certainly affecting us in Derbyshire..

[NKPaul]

It's affecting us in Lincolnshire

[pete]

Lincolnshire - 4 schools down for 25 hrs now. Began 13:50 yesterday (20th Sep) - noticed high load on the filter (hung sessions waiting for a response) and then issues with external DNS resolution. Called the helldesk and the wait time (30 minutes) indicated it wasn't just us.

 

Email (365 & Google Workspace) is occasionally delivered if onsite and externally available services (hosted on site) aren't available. Some websites may work if the browser has cached majority of page assets, but very hit-and-miss.

 

Calling for an update this morning, I got the impression from the helldesk that CommsCare weren't onsite/engaged until this morning. Given emPSN Twitter and website didn't post a status update until this morning, that about tracks.

 

https://www.empsn.org.uk/support/blog/2022/09/21/empsn-major-incident-inc01207579-p1-loss-of-connectivity/

[AndyGibs]

A more accurate title would be emPSN issues. Yeah I’m another school in Lincolnshire affected by this. What made the situation more confusing was 20mins prior to this a nasstar engineer was on site to upgrade our internet to 1Gbps when they had to cancel the install because they couldn’t access the exchange. So when they left and the internet went I started worrying they had disconnected us or something! Took about 30mins of diagnosis before calling emPSN and finding out it’s on their side! What a relief! But also a pain! It’s a shame they left the issue sitting overnight, when I called they had an engineer booked for 11am but they was going to try and get there earlier. I see in the update they have wiped the Cisco OS and started over. I’ve been checking and the internet is still down.

[smarties11]

Lincolnshire School here down since 1.30pm yesterday. Connection is live but very slow and intermittent. Called immediately at 1.35pm, was on hold 20 mins and ticket logged at 1.55pm. NASSTAR treated as isolated incident (advised to reboot router etc) despite me informing them I had confirmation from several other local Schools that they too were affected, but because they had no central comms on that they could only treat as isolated. A parent ticket was raised by NASSTAR at 2.55pm, recognising a central issue, almost 90 minutes after the incident began. Contacted emPSN to express dissatisfaction and was advised that they became aware of the issue at 1.30pm, so a bit of a communications issue here resulting in NASSTAR wasting time logging individual tickets and advising customers of isolated incident.

 

Recent updates seem to suggest a sync issue between their pair of resilient (ha!) firewalls?

 

There was a period of time whereby comms between emPSN and NASSTAR was being appended to the parent ticket and then being distributed to Schools via e-mail. It seems there were issues around 9pm or so whereby there were no Schools available for testing, NASSTAR giving the impression that they were giving up until the morning, with emPSN expressing concern over this in terms of a long period of downtime not reflecting well on them when they are in the middle of recontracting their customers. Someone from F1 in Lincoln then offered their services for remote testing (they support many Schools in Lincs particularly primaries I understand)

 

The case then seemed to hinge around a commscare engineer who was due to arrive at 11am this morning (I believe he arrived slightly earlier than this). Most recent update suggesting firewall sync issues.

 

We had service restored at 18:18 this evening but only for a period of 10 minutes, it seems this coincided with them hard resetting the firewalls and re-installing the IOS looking at incident timings.

 

I'll be very interested to see the post incident report on this and have requested to be on the distribution. I'm sure the issue is a complex one however this level of downtime and impact on Teaching and Learning just isn't acceptable.

 

We already have plans in motion for a redundant line however this won't be until later this term / early next. We have a SmoothWall which can handle the failover and firewalling, we'll likely use Lightspeed or YouFibre who are in our area (and have already run fibres into our BT manhole on-site) and offer 1gbps speeds so much faster than FTTC.

Edited September 21, 2022 by smarties11

[Wubbalubbadub]

We still have sights down.. last update was they have more people investigating...

This is shockingly bad!

[Aprice]

One of ours has been down, thankfully we had a 1GB lease line put in over the summer and a Smoothwall delivered the other day. We just were not expecting to be commissioning it yesterday.

 

Now that's all our customers moved over from these sorts of networks

[MatthewL]

We had a blip on some of our offnet Kcom connections last night which are Nasstar but were all back up this morning, they are FTTC ones and not uncommon to blip over night, see if we have any more in the morning.

[nathan]

Two of our schools are down including their backup lines (bit pointless).

 

For such a huge outage I'm surprised there isn't more being made of it, i remember reading the pages upon pages about the Schools Broadband issues.

[pete]

Still down here. Inbound access is available for o2, Virgin and some (but not all) BT customers.

 

Currently sourcing 4G options with ethernet ports to configure as alternate gateways.

[ITGURU]

Two of our schools are down including their backup lines (bit pointless).

 

For such a huge outage I'm surprised there isn't more being made of it, i remember reading the pages upon pages about the Schools Broadband issues.

 

Backup is only any good if goes via different cable, route, and provider.

[bitznpcz]

Most of my schools have been OK internet wise, but I've had to remove some proxy exceptions to get some websites working. Had a problem with cloud backups and VoIP lines though.

 

I lost all connectivity around 3pm today but this was restored around 4pm but it's still intermittent.

 

Had problems with my RMM tools as well which has made support a nightmare. I've clocked up lots of extra miles trying to help schools as best I can, but it's been a very stressful week!!! Let's hope it gets resolved over the weekend, but updates are few and far between and TBH, I'm not sure they know what the cause or resolution is.

[Lorfarius]

It's been a nightmare to be honest, ours went down on Tuesday morning and barely had any connection all week. Luckily we have a single 3rd party backup broandband line in an office for SIMs so still been able to get some access but so glad we didn't get a second line with KCOM. Those seem to be gone as well.

 

There's at least 40 schools have problems with it and latest news is that BT are involved after multiple outages/firewall replacement. A select few PCs have been able to get online but not able to access anything with videos or pictures so very limited service if at all, yesterday the lot cut out at 2.30pm. Tried messing with proxy settings but hasn't made any difference.

 

Been with schools for almost 20 years and never seen anything like this before, expect a serious response from the company and not just some sorry its fixed so back to normal later. Our contract is up next year so think we will be looking to move.

Edited September 24, 2022 by Lorfarius

[smarties11]

We came back online around midnight and it's been stable since, I've been able to run the offsite backup copy jobs successfully this morning and all our overnight cloud sync processes ran correctly.

 

It looks like they know where the issue is now and have perhaps isolated it. Latest update is that BT are enroute to their node 4 datacentre. I hope this means things are looking more positive for Monday!

 

It's certainly made us reevaluate, we're pressing ahead immediately now with a backup line, which was in the roadmap for later this year.

[bitznpcz]

We came back online around midnight and it's been stable since, I've been able to run the offsite backup copy jobs successfully this morning and all our overnight cloud sync processes ran correctly.

 

It looks like they know where the issue is now and have perhaps isolated it. Latest update is that BT are enroute to their node 4 datacentre. I hope this means things are looking more positive for Monday!

 

It's certainly made us reevaluate, we're pressing ahead immediately now with a backup line, which was in the roadmap for later this year.

We'll be recommending backup lines as well, particularly those with VoIP.

 

It's been stable all day on our monitored servers so fingers crossed for Monday.

 

We've already sent renewal agreements in so I expect a full apology and report on what went wrong and what can be done better next time... There's been too much head scratching for my liking!!!

[Lorfarius]

Seems to be all up and running again with a message to say its sorted. Checked our network and seems to let me connect without issue and browse various websites. Still 5 days of downtime.

[bitznpcz]

Anyone else having issues again today? Really slow connections and intermittent VoIP.

 

Logged calls with Empsn (CASE-202220599) and Nasstar (INC01250922).

[ITGURU]

Anyone else having issues again today? Really slow connections and intermittent VoIP.

 

Logged calls with Empsn (CASE-202220599) and Nasstar (INC01250922).

 

no issues across our 4 sites this morning, running at normal speed and bandwidth usage.

[Lorfarius]

We aren't having any issues either. Has anyone heard from NASSTAR? Nothing since they fixed the problem.

[ITGURU]

We aren't having any issues either. Has anyone heard from NASSTAR? Nothing since they fixed the problem.

 

I have been advised a CRA report will be issued 10 days after resolved.

[Lorfarius]

Been told that the agreement with them is for 99.75% up time so suspect there will be some compensation at least.

[smarties11]

All good here this morning. We've had no issues since early hours of Sat morning.

[bitznpcz]

It's a TalkTalk Business issue:

 

There is currently an outage within the TTB (TalkTalk) Network, which is impacting a number of our sites on the emPSN Network, this isn't a repeat of last weeks issue.

 

TTB is currently running a P1 Incident and is liaising with our Network Provider (Nasstar Master ticket INC01250500) as this progresses.

[ITGURU]

It's a TalkTalk Business issue:

 

There is currently an outage within the TTB (TalkTalk) Network, which is impacting a number of our sites on the emPSN Network, this isn't a repeat of last weeks issue.

 

TTB is currently running a P1 Incident and is liaising with our Network Provider (Nasstar Master ticket INC01250500) as this progresses.

 

Must be only certain areas, as one of our schools go to TTB node then into the emPSN network, but the site is working fine both to internet and between other sites.