What on earth is going on with Northern Grid?

[adhutton]

What on earth is going on with Northern Grid?

 

We've had a few short broadband outages this week, including one this morning for about 15 minutes.

 

Has anyone else on the Northern Grid/BT broadband service been affected?

 

Andrew

[Guest Guest]

Yes. Yesterday lunch iirc (maybe the day before). Ran around like headless chickens thinking it was TMG only for everything to spring back to life after 15 mins.

[denon101]

We have had periods of slowness.... I thought it was us. As I have just changed the entire site over to windows 7. Sorry but I am pleased that others are having troubles on the NGFL connection. Puts my mind at rest!

[nev104]

We had two outages one on Thursday and one yesterday, we rang them up and apparently bt were doing some work.

[adhutton]

More Northern Grid/BT broadband outages again today!

 

About 10 minutes on each occasion, two already this morning!

 

Rang Northern Grid yesterday to learn that nearly all of their staff have gone, leaving only a couple of people in charge.

 

I don't know about any other schools, but we signed-up for a 5 year deal last year and if the service is now going to become unreliable then we are not happy.

 

I would be very interested to hear from any one connected with Northern Grid or any other schools who may have more information regarding this situation and the likely future of our schools broadband via NG.

 

Andrew

[MattDLEA]

From what i understand one of my schools had issues yesterday and then again today when i was in. Was doing a antivirus update so thought it was me, but glad to hear its a problem at there end.

I phoned my other school and they were experiencing issues to. 10 minutes on then 3 minutes off. Some others in our town are having similar issues.

We are tied in for 5 years too, however neither schools are happy with the service both speed and reliability wise.

 

Just what are the alternatives?

[adhutton]

Had this email from Northern Grid late yesterday:

 

"Dear Colleague,

 

A brief update on the issues today with the network.

 

There is an issue on the firewall which BT are working to resolve with their 3rd party suppliers. On their recommendation BT have re-booted the firewall but as we know that hasn't resolved the underlying problem.

 

BT have uploaded the logs from the firewall to the third party support and they are analysing that data now. They will be providing updates throughout the night and will update you accordingly as soon as we receive them.

 

Regards

 

Lianne"

 

 

I really hope they get this sorted out today!

 

Andrew

[MattDLEA]

Is there any software that we can use to identify how long the network is dropped for our schools now want to monitor this

[adhutton]

It's a DOS attack, apparently. Just received this from Northern Grid:

 

"Colleagues,

 

BT have acknowledged that there was a series of intermittent connectivity outages over the last week, apologised for the disruption to schools across the Northern Grid network and recognise that this was a major incident.

 

Northern Grid continue to work with BT to help them understand the extent of disruption to teaching and learning when internet connectivity is lost for even the briefest of periods of time. We know the challenges and pressures our teaching colleagues face when a crucial element of a learning opportunity is missed because ‘the internet went down’ and are working hard, on your behalf to ensure this is not repeated in future.

 

BT have identified the cause of the outages and believe the issue will be resolved imminently.

 

In summary

 

Effectively this was a significant continual denial of service attack to specific IP subnet addresses which was are not in use.

As the addresses were not in use, the firewall was unable to redirect the traffic and this caused the loss of service experienced across the network during each attack.

BT are currently implementing rules to block external traffic directed to IP subnet ranges which are not in use.

BT have recorded immediate improvements and the network is expected to be fully functional today.

Please be assured that Northern Grid will continue to monitor the performance of the network, and work on your behalf, to ensure you have the connectivity you need to deliver high quality teaching and learning every day.

 

Regards

 

Lianne"

 

Wow!

 

Andrew

[MattDLEA]

Even worse today no connections at all this morning

[adhutton]

It's been terrible all day here.

 

I just can't understand how a big organisation like BT are unable to fix this!

 

It's dragged on for well over a week now!

 

Andrew

[MattDLEA]

Can i ask a question and it may be a dumb question. At home I pay £15 a month for example for infinity broadband with speedtest I get 50mb download and 30mb upload. I know that is a private line.

But why are the schools paying lots more for a lot slower connection with a worse service?? The only benifit I see is that there is the smooth wall cloud filtering.

[adhutton]

Its very poor here again this morning!

 

What I can't understand is why its taking them soooooo long to sort this out???

 

They have said its been a massive DOS attack, but surely it doesnt take over a week to re-configure a firewall????

 

If this was affecting domestic BT broadband customers in this way it would be on the national news!!!

 

Not happy

 

Andrew

[cpjitservices]

I think they need better DOS mitigation, depending on what type of DOS attack it is usually you get back up and running in a day, if need be pull the plug / reboot everything and find the source.

[Roopert]

I wont moan too much, but just to add that we also share your woes!

[plexer]

Can i ask a question and it may be a dumb question. At home I pay £15 a month for example for infinity broadband with speedtest I get 50mb download and 30mb upload. I know that is a private line.

But why are the schools paying lots more for a lot slower connection with a worse service?? The only benifit I see is that there is the smooth wall cloud filtering.

 

Without knowing everything the school supplied services provide it's hard to say or compare the two, your home connection is contended i.e shared with other homes so speed can vary widely, schools connections include the firewalls, filtering, possibly email services, access to things like the regional broadband consortium content and other such services.

 

Ben

[adhutton]

Its just come back on for us, about 10 minutes ago.

 

Lets see how long it lasts!

 

Andrew

[hardtailstar]

Issue is down in Kent too, apparently Kent schools all had an issue this morning.

[SBennett]

Issue is down in Kent too, apparently Kent schools all had an issue this morning.

 

Indeed. Up and down all morning.

[Reverend_Dan]

Sorry to hear about this, We pulled out of northern grid mainly a cost issue, we went separately had cheaper quotes from a few suppliers (exa networks, RM, Virgin, BT) but in the end went with RM who were underselling a Virgin 100 meg fibre by 2k and got a 5 year contract with them for just shy of 10k per year. Got a new Tina box (cachepilot equivalent) off Equiinet with protex filtering VM cant remember exact costs but around 2k with hardware and support contract. This did take a bit of ironing out kinks but touch wood has been reliable ever since.

[Sharon_Wright]

Ours came back about the same time, we had intermittent service last week and no internet yesterday and this morning. NGFL do not seem to grasp that we rely on this for lessons - at this time of year a wasted lesson costs our students in exam prep time etc. Can we get a refund for lost days?

[adhutton]

Ours came back about the same time, we had intermittent service last week and no internet yesterday and this morning. NGFL do not seem to grasp that we rely on this for lessons - at this time of year a wasted lesson costs our students in exam prep time etc. Can we get a refund for lost days?

 

I completely agree. We signed up for 5 years with them last year and we've had loads of bother since they have gone over to BT.

 

When NGFL were supplied by Easynet it was rock solid for 99.9% of the time!

 

Part of the problem with NGFL, in my opinion, is that most of their staff have now gone and they appear to be running on a staff of about 2 or 3 people. I am not saying its the current staff's fault, but all of their experienced people have gone and that has got to be affecting their response to schools!

 

Lets hope it settles down now.

 

Andrew

[adhutton]

Just had this update from NGFL:

 

"Dear Colleagues,

 

We have received the following update from BT regarding the outages we have experienced: -

 

Network access is once again restored. CPU processors are working as they should.

 

This restoration was a result of blocking the 3 new IP addresses which were causing an issue.

 

The firewall team are now able to put the rules into the firewall which will block unallocated IP addresses to prevent external access from having this effect.

 

I will update you once this is done.

 

The Firewall team and checkpoint are still working on the firewall to establish a root cause and source ID if possible for the DOS attack and to ensure there are no underlying issues caused by recent events.

 

Thanks

 

Janet"

 

 

Andrew

Edited April 30, 2013 by adhutton

[Guest Guest]

Just had this update from NGFL:

 

"Dear Colleagues,

 

We have received the following update from BT regarding the outages we have experienced: -

 

Network access is once again restored. CPU processors are working as they should.

 

This restoration was a result of blocking the 3 new IP addresses which were causing an issue.

 

The firewall team are now able to put the rules into the firewall which will block unallocated IP addresses to prevent external access from having this effect.

 

I will update you once this is done.

 

The Firewall team and checkpoint are still working on the firewall to establish a root cause and source ID if possible for the DOS attack and to ensure there are no underlying issues caused by recent events.

 

Thanks

 

Janet"

 

 

Andrew

 

Happy to see they didn't have the cheek to call it a DDOS attack. Not being a wikid hack0r I may be naive but firstly, how can 3 or so IPs bring down a firewall? Secondly why isn't there measures in place on the firewall to block IPs which are DOSing it. And thirdly why did it take over a week for the engineers to find these 3 IPs?

[cpjitservices]

It'll take them so long to find the IP's because no doubt in the routing table there will be hundreds of thousands of IP's and thats not there own thats everyone else's too from BGP peers. It'll probs be a Juniper MX series router too so thats a lot of IP's it can hold in it's tables, so tracing an IP isnt as easy as you think it would be if your trailing hundreds of thousands of IP's. Once they find a 'suspect' IP they need to find the source, it may not be an IP directly connected to them it could be from one of the upstream providers/peers, this means they need to contact those peers to resolve it at there end and block it on there firewalls / routers locally.

 

It's a painful process.