Schools Broadband issues

[nathan]

Sorry was replying to Primus. I'm not defending them so vociferously. I think im being fair.

 

I understand the frustration caused by this, it's hit us too. But I also understand because I've been in the situation where something isn't working in a school due to a supplier and staff in our trust have always been understanding. I don't take that for granted and extend that understanding to our suppliers - with in reasom

 

Primus, I just don't understand the constant negative posts when you don't actually know what's going on - for instance the emails sent out explaining things.

 

Of course no offence meant in it all

Edited December 1, 2016 by nathan

[nathan]

I agree with @Primus

 

We have test kit and test updates etc before they are released into the wild. Im also very fussy on how things work. Some people say im OTT but Its all for the end user experience and at the end of the day that is our job.

 

How do you know they haven't got test hardware?

 

It's a really new patch isn't it?

[Primus]

Perhaps because it's clear they haven't got a clue what's going on. If you look back you were complaining in May that it had been great up until the last few months. So it's been bad since what - February/March?

 

In all that time they haven't managed to get it stable for a lot of people.

 

It's gross incompetence. Their posts on here make it quite clear what's going on.

 

You also have no idea whether I do or do not have colleagues in schools where SB provide connectivity who are telling or showing me what emails have been sent out.

 

In short you appear to now be close to defending the indefensible.

 

Schools pay an awful lot of money for their connectivity, it's vital and this protracted issue is appalling.

[FN-GM]

How do you know they haven't got test hardware?

 

It's a really new patch isn't it?

 

If it had been tested they would have found this issue before it went into production.

 

If they can't replicate this on the test system put the test kit into service until the issue is fixed, but I can't see this helping as they have changed the hardware so it must be a software issue, so shouldn't make a difference.

 

So either:

A: They don't test

B: Test network doesn't match the production network and thats why it didn't show up, if thats the case its pointless testing.

Edited December 1, 2016 by FN-GM

[Primus]

@nathan if they had test hardware they wouldn't be saying they can't roll out the firmware upgrade/patch for fear it will break things.

 

They would have, you know, tested it and found out?!

[Primus]

As for it being a really new patch, they've been talking about it on here for a couple of weeks at least.

[lou-ashtonhurst]

Hi folks

 

 

I thought I'd just drop you all a note with a bit more of an explanation of today’s incident, the timeline of events so far and why we are in the position we are now in. I'll leave the commercial discussions to Dave - this post is purely about the tech side of things for those that are interested.

 

The first incident occurred on 23/11 at 2pm, the second on the 24th at 1pm, the third at near enough midday on the 25th - a separation of approximately 23 hours each time. After the first incident, we reported it through to Fortinet and awaited their debug - but we weren't overly concerned. These things do happen and initially this looked like a random, one off occurrence - all our standard troubleshooting showed no problems and while we awaited Fortinet's diagnosis, we put this down to a random unit crash. As I mentioned in the earlier post, we were made aware of problems with the HA after the second occurrence which we resolved that night in an emergency maintenance slot. The next day we had the third occurrence - after which we replaced the hardware. Each of these steps was taken as soon as possible in an emergency maintenance window and on the basis of decisions of myself, our Technical Director, our senior engineers and Fortinet's senior engineers.

 

We had been aware of a potential issue with a process called miglogd, which since the update to firmware 5.2.8 had been regularly crashing (but not with significant impact to the box). Over the last few weeks we have implemented a workaround on Fortinet's recommendation, knowing that a new firmware release was required to resolve it and was in the works. As with any vendor, Fortinet quite rightly run significant testing before any firmware upgrade is rolled out.

 

The debug output from the first night led us to miglogd, the second night showed HA problems, the third we suspected a hardware fault on the ASIC as traffic seemingly wasn't being forwarded from the WAN port. These were all approx 23 hours apart.

 

As you know, today at 10.30 we experienced a fourth occurrence. This didn't match the previous pattern of 23 hours and given we had already replaced the hardware, this was looking more like a software problem. The decision was made to allow the downtime to continue longer than usual to allow Fortinet TAC to debug on the live unit, in the hope of nailing the cause once and for all. We had a cut off of 1.30pm, at which point we would reboot the box regardless of whether we had gotten to the bottom of the issue, so as not to disrupt afternoon lessons.

 

One of our senior engineers held a teleconference with a senior TAC engineer and conducted the live troubleshooting - while myself, Mat and our Product Manager Rob went through the debug files. This time, the problem was different again - we were seeing our edge router was receiving no traffic from the WAN port of the Fortigate, however traffic captures on the Fortigate unit were showing ARP-Reply packets being sent. These tcpdumps capture from the kernel - not the ASIC/NIC (based on NP6 architecture) and so this left us with the possibility of a software issue affecting the ASIC/NIC, a problem with the SFPs in the ports or potentially an issue with the input on the edge router. I'm not ashamed to say that this point (close to 5pm), we all took a break (service was restored, and we were awaiting handover from the European service centre to the American service centre as this was still a P1 ticket).

 

During the break, firmware version 5.2.10 was released, along with its release notes. This is the firmware version we were waiting for in relation to the miglogd issue, however, please find below some of other issues resolved in this version:

 

 

385115 Miglogd constantly crashing after upgrade to 5.2.8

386683 FG-1500D kernel panics after roughly 24 hours of uptime

387212 HA gets out of sync frequently and hasync becomes zombie

388032 Corrupted packets may cause malfunction of NP6, which causes NP ports to be unable to accept and forward traffic. Affected models: All NP6 platforms.

387675 ARP-Reply packets drops in NP6

 

For those who aren't aware of our setup - the device that has been having a problem is a 1500D. The 1500D platform is based on NP6 architecture.

 

We have this evening updated to 5.2.10 after discussion internally and with TAC. At the moment, all looks stable and we are hopeful we will not see further issues - however I'm sure you can appreciate I cannot guarantee this.

 

We will be monitoring closely tomorrow and if we do experience a further issue, the platform will be rebuilt on new routing hardware on Sunday evening.

 

I do apologise for the inconvenience these issues have caused, we are working as hard as we can to resolve this as soon as possible.

 

Kind Regards

Lou Ashtonhurst

NOC Manager

Talk Straight/Schools Broadband

[Edu-IT]

I agree with @Primus

 

We have test kit and test updates etc before they are released into the wild. Im also very fussy on how things work. Some people say im OTT but Its all for the end user experience and at the end of the day that is our job.

There's a difference between what we sould need in schools and what they would need to replicate their network to create a test one.

 

As this appears to be a software issue, no matter what hardware they use, this could happen on any Fortigate running the software if the circumstances are right and they therefore have no choice but to let Fortinet release a software fix which will be subject to Fortinet's own processes.

 

I think its only fair to separate their issues. There have been issues with Lightspeed, that is different to these issues of late. Together they all add up to downtime but each have a different cause. It's not as if SB are sitting on issues and ignoring them. They appear to be trying to be proactive but they are reliant on third parties doing the same.

 

If you aren't happy with the service then by all means engage with SB but to bash them entirely as though this is a simple problem they are neglecting to fix perhaps isn't fair and this problem presumably could affect anyone using Fortigate in these particular circumstances.

[Bob_the_Goon]

Thank you for the update Lou. Time to chill and rest. Good luck for tomorrow.

[mavhc]

Thanks for the detailed update.

 

Hope you have an SLA with Fortigate then to get a load of money back.

 

https://pulpphikshun.wordpress.com/2015/08/31/backdooring-a-fortios-vm/

 

I'm surprised you found it ok that your $30k box randomly crashed, if my network appliances don't run for years without problems I'd ditch them.

 

Wonder what other companies do, I'd be wary of mixing a router with a computer, can't replace the computer part because the software is bad without replacing the router. Wonder if a cluster of linux PCs would work better.

[Blue_Cookeh]

I don't think there would be nearly as much of a problem if the FortiGate boxes had died 3 times - what's taking the biscuit in this situation is the amount of issues there were initially with Lightspeed (we've had at least 4 periods of downtime, if not more due to those) ON TOP OF the FortiGate problems.

[FN-GM]

There's a difference between what we sould need in schools and what they would need to replicate their network to create a test one.

 

As this appears to be a software issue, no matter what hardware they use, this could happen on any Fortigate running the software if the circumstances are right and they therefore have no choice but to let Fortinet release a software fix which will be subject to Fortinet's own processes.

 

I think its only fair to separate their issues. There have been issues with Lightspeed, that is different to these issues of late. Together they all add up to downtime but each have a different cause. It's not as if SB are sitting on issues and ignoring them. They appear to be trying to be proactive but they are reliant on third parties doing the same.

 

If you aren't happy with the service then by all means engage with SB but to bash them entirely as though this is a simple problem they are neglecting to fix perhaps isn't fair and this problem presumably could affect anyone using Fortigate in these particular circumstances.

 

Exactly, its a software issue and they should be testing it properly. They admitted before there is an issue with firmware 5.2.8 , If they did test properly they would have found the issue before it went live.

 

The company has made no proper assurances they test stuff out. So in my eyes they are to blame.

Edited December 2, 2016 by FN-GM

[synaesthesia]

Perhaps because it's clear they haven't got a clue what's going on. If you look back you were complaining in May that it had been great up until the last few months. So it's been bad since what - February/March?

 

In all that time they haven't managed to get it stable for a lot of people.

 

It's gross incompetence. Their posts on here make it quite clear what's going on.

 

You also have no idea whether I do or do not have colleagues in schools where SB provide connectivity who are telling or showing me what emails have been sent out.

 

In short you appear to now be close to defending the indefensible.

 

Schools pay an awful lot of money for their connectivity, it's vital and this protracted issue is appalling.

 

That is a little harsh. I find myself in an unenviable position, where I know the guys at SB well, met with many of the upper brass, have a good working relationship with our account manager, director, technical managers and have always had an excellent level of customer service from their technical staff. I know full well they are doing everything possible, yet I still have to argue against them for my schools. I absolutely hate that. We had problems 1-2 years ago with service and after a couple of meetings they were extremely transparent with their failures and what they were doing to rectify them. They have done a fantastic job restructuring their staffing to cope with the enormous extra business they had and since then we can only be complimentary on that side of things. Secondarily regarding the testing of equipment, we've all been there; we've run tests for weeks, months on setting things up, seeing it all work perfectly and thinking "Yup, that's the setup." It goes live, and within a couple of weeks of heavy use by real users, things go pear shaped. I thought my twin server, twin site setup with DFS replication for users & shares was pretty much infallible and was awesome for redundancy. Testing was perfect. Real world scenario however proved me wrong.

Fortigate are not small players in their industry; they may not have the brand name power of HP or Cisco or financial prowess of Juniper but they're no D-Link either. They are aware there are problems and they are working around the clock to help SB resolve things.

Still, with all of this in mind I have to sit here and consider being the technical pawn in what may be a larger legal battle to go elsewhere. Regardless of that, I don't like seeing someone use terms like "they haven't got a clue what's going on" or "gross incompetence". These guys don't have the infrastructure that the likes of BT and Virgin have but how much do we all moan at home when Virgin's crapness strikes? Lack of reliability topped with the world's worst customer service doesn't stop them growing! But they are neither incompetent or clueless. They are still learning and growing. They will forever have my respect.

[nathan]

As for it being a really new patch, they've been talking about it on here for a couple of weeks at least.

 

And? That doesn't mean the patch has been released for a couple of weeks.

 

For the record, I'm not trying to defend them to the hills. I'm trying to be fair.

 

I wouldn't want an SLT coming up to me and saying we want to terminate your employment because this 3rd part supplier that you are tied into has let us down.

[FN-GM]

These guys don't have the infrastructure that the likes of BT and Virgin have but how much do we all moan at home when Virgin's crapness strikes? Lack of reliability topped with the world's worst customer service doesn't stop them growing!

 

You can't really compare a consumer home product to an business / enterprise level product.

[synaesthesia]

Of course, but the hardware used is typically identical. Biggest difference I'm aware of other than software would be a higher endpoint concentration on consumer setups.

[Edu-IT]

Exactly, its a software issue and they should be testing it properly. They admitted before there is an issue with firmware 5.2.8 , If they did test properly they would have found the issue before it went live.

 

The company has made no proper assurances they test stuff out. So in my eyes they are to blame.

Indeed and I'm confident they don't roll out software on a whim, based on what Dave has said in this post. At the same time if Fortigate issue software then they will have tested it extensively too. Things happen unexpectedly. It happens in all products. The difference is whether a company sits on its backside doing nothing about it or engages the right resources to do what it takes to resolve.

 

I say again, SB don't create the software for the hardware. They buy in the services of an established brand. Therefore, they have to work with them. It's not as if SB have created software and intentionally deployed it knowing it has issues. Do you think they would do that?

 

It's an inconvenience but is the thought that SB aren't doing enough to rectify it? It seems from this thread they are doing all they can regardless of what it takes and time of day.

Edited December 2, 2016 by Edu-IT

[nikaso]

And? That doesn't mean the patch has been released for a couple of weeks.

 

For the record, I'm not trying to defend them to the hills. I'm trying to be fair.

 

I wouldn't want an SLT coming up to me and saying we want to terminate your employment because this 3rd part supplier that you are tied into has let us down.

 

I appreciate what you are saying and agree none of us would want that however lets say that you owned a company that sold robotic teddy bear assistants....

 

You are renting them to customers happily but one day the customer teddy bears eyeballs start rotating through 360 degrees constantly.....this is extremely disconcerting and you deactivate the teddy bears while your eyeball supplier investigates.

 

Customers are unhappy that they don't have their furry robotic assistants and it takes a while to resolve the issue after several false starts.

 

Fast forward a few months and disaster strikes......the teddies heads starts spinning through 360 degrees, you keep remotely restarting the teddies but each time a different limb starts doing the exorcist tango....eventually you think you have got the problem sorted and the teddies are back to normal.......However for the customers they have a demonic teddy sitting in the corner of their room that they don't feel able to trust.

 

You can't then blame the customers for wanting rid of their frenzied furry companions earlier than their contract finish date.

 

 

(Sorry for the complete randomness of the robotic furry teddies, I think I have been watching too much anime.)

[Wubbalubbadub]

I appreciate what you are saying and agree none of us would want that however lets say that you owned a company that sold robotic teddy bear assistants....

 

You are renting them to customers happily but one day the customer teddy bears eyeballs start rotating through 360 degrees constantly.....this is extremely disconcerting and you deactivate the teddy bears while your eyeball supplier investigates.

 

Customers are unhappy that they don't have their furry robotic assistants and it takes a while to resolve the issue after several false starts.

 

Fast forward a few months and disaster strikes......the teddies heads starts spinning through 360 degrees, you keep remotely restarting the teddies but each time a different limb starts doing the exorcist tango....eventually you think you have got the problem sorted and the teddies are back to normal.......However for the customers they have a demonic teddy sitting in the corner of their room that they don't feel able to trust.

 

You can't then blame the customers for wanting rid of their frenzied furry companions earlier than their contract finish date.

 

 

(Sorry for the complete randomness of the robotic furry teddies, I think I have been watching too much anime.)

This wins!

 

I have tears running down my face!

 

But on a serious note.. very valid point!

[FN-GM]

I say again, SB don't create the software for the hardware. They buy in the services of an established brand. Therefore, they have to work with them. It's not as if SB have created software and intentionally deployed it knowing it has issues. Do you think they would do that?

 

That doesn't make a blind bit of difference. If they used a home made product that would be fine as long as it works.

 

 

Indeed and I'm confident they don't roll out software on a whim

 

Im not...

[nathan]

I appreciate what you are saying and agree none of us would want that however lets say that you owned a company that sold robotic teddy bear assistants....

 

You are renting them to customers happily but one day the customer teddy bears eyeballs start rotating through 360 degrees constantly.....this is extremely disconcerting and you deactivate the teddy bears while your eyeball supplier investigates.

 

Customers are unhappy that they don't have their furry robotic assistants and it takes a while to resolve the issue after several false starts.

 

Fast forward a few months and disaster strikes......the teddies heads starts spinning through 360 degrees, you keep remotely restarting the teddies but each time a different limb starts doing the exorcist tango....eventually you think you have got the problem sorted and the teddies are back to normal.......However for the customers they have a demonic teddy sitting in the corner of their room that they don't feel able to trust.

 

You can't then blame the customers for wanting rid of their frenzied furry companions earlier than their contract finish date.

 

 

(Sorry for the complete randomness of the robotic furry teddies, I think I have been watching too much anime.)

 

Although i was just giggling with that reply, i'm not for one moment suggesting that customers don't have a right to complain. They most certainly do. The point i was trying to make is that we have all been in the same situation (albeit more than likely on a smaller scale) and i think we need to keep that in mind before hurling abuse there way.

[jertsy]

@SchoolsBroadband my account manager has been in touch, so cheers for that.

[Edu-IT]

That doesn't make a blind bit of difference. If they used a home made product that would be fine as long as it works.

 

 

 

 

Im not...

Based on?

[Nixon77]

This is really nothing to do with me, as I don't and have never used the service, however I will throw my short opinion in.

 

You signed up for a product, it seems it has continually had issues, what the business has or has not done to resolve the issue is pretty irrelevant. You pay for the product, it should be delivered.

 

I would ditch it as soon as possible. Business is business as they say. Most businesses don't act until they start losing customers.

Edited December 3, 2016 by Nixon77

[Blue_Cookeh]

I think this is a primary example of why datacenters shouldn't stick to one vendor, really.