LGFL connection outage and bad technical support

[xicor]

Hi All

 

has anyone else noticed a shift in how LGFL support works ?

 

previously when an outage happens the LGFL team jump on it straight away, they do their checks ie

 

1. check if there is an outage in the area
   
2. check if they can connect to the router or the firewall or any of the equipment
   
3. update me with their findings and let me know if they need anything from me onsite, ie to check the lights or to restart the hardware, they usually message me and call my number
   

 

nowadays and specifically the last few months they don't bother and refuse to check anything on their end without me telling them first which lights are doing what on the equipment, I mean surely there is some checks they can do and they did in the past before even getting to the stage of which lights are which...

 

I've had better ISP support from talk talk....

 

anyone else or is it just my imagination :/

[PaddyNewman]

Hi,

Without getting myself in trouble, I'll try to explain what we see when you call in with an outage as that's your specific pointer here.

 

As a starter, we can see if the firewall is up, we have an up or down marker on the support site deployment page, also if you call in and we cannot see your firewall, we can see your schools ranges highlighted in orange.

 

We also have a tool that monitors every school firewall/core devices/services we host etc as any ISP/company tends to.

 

I'm more than happy to go in to things deeper but this is extremely public and I know full well my managers (and above) are watching posts here so I do want to keep any 'company info' to a minimum, but we definitely know of your status when you call in.

 

Typically we only ask for more when its 'orange' on our call log or 'red' on our actual real-time monitor. That is just to get an idea of what it could be, any red or orange lights are a worry, any blank lights with cables in are possibly a problem.

 

For example on a Cisco 5508, an orange flashing status light can indicate hardware/storage faults on the firewall. That could simply need a reboot (maybe a 1 in 10) or a replacement (most likely). A static orange light could mean it doesn't even load from flash and its instant game over. If by some miracle they are all green AND you are down, the firewall could have locked up, not the technical term but they do this occasionally and a reboot will either fix, or take you to the orange flashing abyss. If it happens often, we'd replace the firewall (we do note when firewalls have routinely failed and mention it in internal updates in cases so we aren't just saying that to get you off the phone/close the case etc)

 

I'm not senior, or indeed management in any way but I can point this to those with phenomenal cosmic powers to try and work out what happened with your call. Sorry you felt that way, I certainly would go through any standard checks one of which is getting a response from your firewalls management IP to make sure we haven't lost contact / check polling history for said device. That isn't standard as our helpdesk aren't typically able to SSH to firewalls, but they do have the monitoring tools in place. If it needs a bit more training then we shall do that! I like my job, so I don't want to start dropping anything here that could end up in my job being terminated, but I do also value the opinions of others who actually use the services, and I'd much rather you walked away happy, even if its not the answer you wanted.

 

Do fire any concerns towards me, I am happy to assist but anything technically 'support' should remain in the LGfL realms as A - Everyone can see it here B - Its easier for escalation purposes and C - I'm not being paid right now, this is my free time and support out of hours when I could be watching some TV or filling my face with pizza just fills me with sadness that I'd rather not have

 

 

Oh... from a non-business stand point, you must have had a really good talk talk employee, I left after a week:p

 

 

Always happy to help!

 

Paddy

[xicor]

Hi Paddy

 

for the record, you specifically have been very helpful in the past and have helped the school through few issues we had, if your management is watching this space they should make some notes on that!]

 

that aside, what you're saying makes sense and is pretty much what used to happen, the team would tell me what they have investigated and what they need from me, I had two outages in the last 2 months, in the first one i spent so much time just trying to convince the guy on the phone to just do some checks from their end, his words were we cant pass this issue to the firewall team without knowing the lights..... i had to keep insisting on him doing the checks before he said he will look into it.... in that particular issue i went onsite and restarted the kit and that brought everything back, i then asked the team to investigate what caused in the logs but no one ever got back to me on that.

 

this issue now see ticket log below

 

[TABLE=width: 1]

[TR=bgcolor: palegoldenrod]

[TD]Public post[/TD]

[TD][/TD]

[TD]15 Jul 21, 16:53[/TD]

[TD]n/a[/TD]

[TD=align: center]-[/TD]

[TD]any updates please ?[/TD]

[/TR]

[TR]

[TD]Public post[/TD]

[TD][/TD]

[TD]15 Jul 21, 16:17[/TD]

[TD]n/a[/TD]

[TD=align: center]-[/TD]

[TD]i dont have that at the moment, can you please run some checks from your end to establish where the connection issue is ? are you able to see the firewall is there any outages in the area ?[/TD]

[/TR]

[TR=bgcolor: palegoldenrod]

[TD]Public post[/TD]

[TD][/TD]

[TD]15 Jul 21, 14:37[/TD]

[TD]n/a[/TD]

[TD=align: center]-[/TD]

[TD]Hello ,

 

PLease provide us with the light information on the firewall, please.

 

Kind regards

 

[/TD]

[/TR]

[TR]

[TD]Public post[/TD]

[TD][/TD]

[TD]15 Jul 21, 14:35[/TD]

[TD]n/a[/TD]

[TD=align: center]-[/TD]

[TD]we just restarted all the equipment there but still no internet[/TD]

[/TR]

[TR=bgcolor: palegoldenrod]

[TD]Public post[/TD]

[TD][/TD]

[TD]15 Jul 21, 14:28[/TD]

[TD]n/a[/TD]

[TD=align: center]-[/TD]

[TD]Hi guys, please can you check form your end what the issue is and what devices you can connect to ? this same issue happened few weeks ago we had to unplug the power and plug it back for a restart and this fixed this issue, at the time i asked your team to look into the logs to see what caused this issue but i never heard back, please can you look into this asap as I have staff working onsite[/TD]

[/TR]

[/TABLE]

 

and no update after that until 12pm today! and still no explanation of what tests have been made and where the issue could be.... this level of support from LGFL was never like that, and indeed it shouldn't considering how much it costs.

 

ont he talk talk bit, at least they attempted to connect to the router and advised on where the issue is haha

 

Hi,

Without getting myself in trouble, I'll try to explain what we see when you call in with an outage as that's your specific pointer here.

 

As a starter, we can see if the firewall is up, we have an up or down marker on the support site deployment page, also if you call in and we cannot see your firewall, we can see your schools ranges highlighted in orange.

 

We also have a tool that monitors every school firewall/core devices/services we host etc as any ISP/company tends to.

 

I'm more than happy to go in to things deeper but this is extremely public and I know full well my managers (and above) are watching posts here so I do want to keep any 'company info' to a minimum, but we definitely know of your status when you call in.

 

Typically we only ask for more when its 'orange' on our call log or 'red' on our actual real-time monitor. That is just to get an idea of what it could be, any red or orange lights are a worry, any blank lights with cables in are possibly a problem.

 

For example on a Cisco 5508, an orange flashing status light can indicate hardware/storage faults on the firewall. That could simply need a reboot (maybe a 1 in 10) or a replacement (most likely). A static orange light could mean it doesn't even load from flash and its instant game over. If by some miracle they are all green AND you are down, the firewall could have locked up, not the technical term but they do this occasionally and a reboot will either fix, or take you to the orange flashing abyss. If it happens often, we'd replace the firewall (we do note when firewalls have routinely failed and mention it in internal updates in cases so we aren't just saying that to get you off the phone/close the case etc)

 

I'm not senior, or indeed management in any way but I can point this to those with phenomenal cosmic powers to try and work out what happened with your call. Sorry you felt that way, I certainly would go through any standard checks one of which is getting a response from your firewalls management IP to make sure we haven't lost contact / check polling history for said device. That isn't standard as our helpdesk aren't typically able to SSH to firewalls, but they do have the monitoring tools in place. If it needs a bit more training then we shall do that! I like my job, so I don't want to start dropping anything here that could end up in my job being terminated, but I do also value the opinions of others who actually use the services, and I'd much rather you walked away happy, even if its not the answer you wanted.

 

Do fire any concerns towards me, I am happy to assist but anything technically 'support' should remain in the LGfL realms as A - Everyone can see it here B - Its easier for escalation purposes and C - I'm not being paid right now, this is my free time and support out of hours when I could be watching some TV or filling my face with pizza just fills me with sadness that I'd rather not have

 

 

Oh... from a non-business stand point, you must have had a really good talk talk employee, I left after a week:p

 

 

Always happy to help!

 

Paddy

[PaddyNewman]

Hi,

 

Appreciate the kind words.

 

For perspective, our monitor was about as red as it has been in a while for reasons posted here - https://www.computerweekly.com/news/252504039/Virgin-Media-outage-Electical-explosion-in-South-East-London-takes-down-network

 

According to that case, an automatic alert from our monitor reported you down 15/7 @ 13:06. Was claimed by a colleague in 1.5 hours and our initial first ask for light information was given, purely as we couldn't see the kit as per my initial update here. Its the process we follow to rule out our firewall, we can build and ship one of those same day typically so if it was that, we would have built it then and likely booked it for a pre 10am delivery the following day. That is the reason for the light question, I can see from the case and the kit onsite, you would have had an alcatel or similar NTU which would show a lack of light on the fibre port which would mean we go straight to VMB. So thats just for clarity as to why we asked for the light, not berating you for not being able to see the light information, it happens, you could be at site A when site B goes down.

 

The 'can I get to it' checks on our end would have failed as the 'down' marker is active, we cannot ping the management of the firewall so it means its down.

 

I can see it was passed to our networking team, albeit late this morning which showed that the router had no connectivity from VMBs end, so a Virgin 'fault' and I use that word lightly as its sort of part of the wider outage yesterday not directly caused or avoidable by VMB.

 

Should we have done more sooner, yes, I think we should. I am always on hand if our support team need a look in from a network perspective and I'll re-advertise that fact to them and also give them some additional checks they can possibly do, its coming up to the summer holidays and we have had a nice influx of staff so we will be training and this is something I can add to the current plan, so apologies for the delay on this one.

 

I can't promise this for everyone as I'd need someone to do my job for me but if you experience an outage and you are truly stuck for an update, please call and ask for me. If I am available I will take the call, but hopefully you don't have any repeated outage like this again and if you do, its handled with a bit more of a hands on investigation at stage one.

 

The support manager at AdEPT is aware of this post and is likely checking for updates also, so this will be something that will get looked in to in our upcoming training over summer, once again sorry it wasn't handled to your expectations and we will fix that!

[JohnJackson]

Saw this note today. I think Paddy has responded at length. However, if colleagues aren't getting the qualirty of support you feel you're entitled to then please get in touch with me. [email protected] If we can fix it, we will. We're also launching a service called Concierge from the the autumn which wraps around the helpdesk and will give schools, MATs, Councils and support organisations a personalised service.

[minimoo]

@JohnJackson - Adept are lucky to have Paddy, and we are lucky that he takes the time to talk on here occasionally

@PaddyNewman - here's one for your dev team - any way that when a school logs in to the support home page, it could show what status your dashboard thinks the school is in? There might be something on one of the menu's, already, but i'm guessing you guys have a different internal feed

[PaddyNewman]

@minimoo Whilst its not directly on the homepage, Network Status > At a glance should give you the standard traffic light set up for your school. Green is obviously good, red is bad!

 

If there is something more specific you want though? Happy to ask!

[Edu-IT]

@minimoo Whilst its not directly on the homepage, Network Status > At a glance should give you the standard traffic light set up for your school. Green is obviously good, red is bad!

 

If there is something more specific you want though? Happy to ask!

Not directly related but the Adept portal allows me - and I assume others - to view name and date of birth of other people on the platform. Doesn’t feel right - I did raise a comment but not heard back. Is this an area you can help?

[PaddyNewman]

Help is I strong word, but I can certainly relay it.

 

I don't recall it ever being a problem, but I can check and ask. How did you comment to us before and I'll see where it got stuck.

 

We use date of birth for validation purposes occasionally and it also helps determine staff and students so it's a 'required' field but if it's something that you want us to hide or something, I can ask, for sure!

 

Can I be nosey and ask what the problem is, not in a rude way but it's handy to know why it's a problem and if we can make changes to ease the issues of others, it's something we will try and do

 

What we can't do, is not use it, it's a required field from the MIS for staff and students I believe.

 

Let me know, I've passed on to a colleague now but really helps having feedback and queries!

[PaddyNewman]

A quick update, apparently you should see a yes/no rather than the DOB? Can you confirm if you see different? Happy to have a private message with screenshot or email me, [email protected] if you see otherwise?