Port on core switch keeps dropping

[machin05]

Hi all,

 

Real head scratcher this morning that I'm hoping someone can help me with!

 

The port on our core switch (HP A5500) that our Smoothwall box is connected to keeps going up and down. It will drop for 3 seconds then come back up but this can have prolonged impact on users internet connectivity as it takes a minute for the IP of the Smoothwall to respond again resulting in proxy authentication errors in the browser etc.

 

Must have happened around 15 times so far today.

 

It is something we've seen very sporadically before (once a fortnight and never more than twice in a day) so I've never been able to get to the bottom of it. I believe it started after we enabled STP (RSTP) and loopback protection across all our switches. However, there's nothing in the logs to suggest it's the loopback protection that is disabling the port. The messages in Smoothwall look like this:

 

09:10:07 Kernel [6166017.202045] e1000e 0000:01:00.0 ethA: NIC Link is Down

09:10:10 Kernel [6166020.380084] e1000e 0000:01:00.0 ethA: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

The logs from the switch look like this:

 

04/01/2023 11:18:23 IP REDACTED %%10IFNET/3/LINK_UPDOWN(l): GigabitEthernet1/0/36 link status is UP.

04/01/2023 11:18:20 IP REDACTED %%10IFNET/3/LINK_UPDOWN(l): GigabitEthernet1/0/36 link status is DOWN.

 

 

I have disabled STP on the port that the Smoothwall is connected to (port 36) but that hasn't prevented it from randomly going down. It has, however, increased the speed at which the Smoothwall IP comes back online after the status is back up though (doesn't really solve the issue, but prevents users from experiencing as many connectivity issues).

 

There doesn't seem to be a pattern to when it drops out. Traffic on the Smoothwall isn't particularly high so I' pretty sure it's not a bandwidth issue.

 

We did have our lease line upgraded from 100mb to 1Gb over the Christmas holiday, so that might be a factor, however no configuration changes on the core switch were needed, and as I said, we did have this issue before, albeit a lot less frequently. Today is the first day back for all the students. We had a staff INSET yesterday and there were a couple of dropouts then too.

 

Currently I'm at a bit of a loss as to where to begin troubleshooting now! I am going to configure another port with the correct VLANs on the core switch and change the connection to that port to rule that out first of all. Aside from that, maybe changing the physical network cables?

 

Is it worth disabling STP on the core for troubleshooting purposes?

 

Any expertise would be greatly appreciated as I'm really flummoxed here - genuinely don't even know if it's Smoothwall, the switch itself, or physical connectivity causing the dropouts!

 

Cheers,

 

Will

[machin05]

Unsure if this is related either, but having looked through the syslogs on the core, I'm getting a recurring issue with port 3:

 

%%10LLDP/5/LLDP_DUPLEX_INCONSISTENT(l): The mode of duplex mismatch discovered on GigabitEthernet1/0/3 (Full-duplex), with D�[�� (Unknown-duplex).

 

It's another switch that's connected to port 3, however that switch isn't reporting any duplex errors in its syslog.

[steveg]

I'd definitely change the cable first, start with the most simple.

 

May also be worth changing the smoothwall nic from auto negotiation speed to setting it manually. Maybe even 100 full duplex first, just to see if the issue goes away. Obviously not a long term solution with a gig internet connection.

[machin05]

I'd definitely change the cable first, start with the most simple.

 

May also be worth changing the smoothwall nic from auto negotiation speed to setting it manually. Maybe even 100 full duplex first, just to see if the issue goes away. Obviously not a long term solution with a gig internet connection.

 

Cheers for the reply - just to avoid confusion, the DUPLEX issue isn't on the same port as the Smoothwall, I only mentioned it as it's the only other recurring message I'm getting from the core switch syslog.

[steveg]

Would still be my default go-to for troubleshooting initially, especially dropping the speed.

[machin05]

Would still be my default go-to for troubleshooting initially, especially dropping the speed.

 

Do you mean change the duplex mode on the Smoothwall box itself, or on the switch config?

[JRA]

Is the Smoothie box official hardware or is it something else, perhaps with RealTek cards in it?

[machin05]

Is the Smoothie box official hardware or is it something else, perhaps with RealTek cards in it?

 

Official hardware. It's an S8 appliance.

 

As an aside, changing the port on the core switch hasn't changed anything. Same issues, different port number.

 

Incidentally, there were no dropouts at all over lunch, but there's been about 3 or 4 already during the last lesson. Could it be traffic related?

[steveg]

Had a look and not sure you can change the speed on the Smoothwall nic (not from web interface anyway), so may be worth setting it on the switch to troubleshoot

 

Could possibly be something traffic related, as you mention the internet connection also got upgraded. Will be a strange error if it is, but I wouldn't rule it out.

[robk]

Which NIC on the S8 are you using?

 

I saw a similar issue when I used the management NIC for traffic.

 

Seems the nic interface was a different Chipset and didn't like the heavy load.

 

(This was after a migration to a new box and I had migrated the old config).

 

Would fit with increased Internet connection so more traffic.

 

Worth a look. I am sure smoothwall support would be able to confirm if this was possibly the problem.

[machin05]

Which NIC on the S8 are you using?

 

I saw a similar issue when I used the management NIC for traffic.

 

Seems the nic interface was a different Chipset and didn't like the heavy load.

 

(This was after a migration to a new box and I had migrated the old config).

 

Would fit with increased Internet connection so more traffic.

 

Worth a look. I am sure smoothwall support would be able to confirm if this was possibly the problem.

 

This is very interesting. Our current box is a config migration from an old box. We're using NIC1 for VLAN1 (the one that keeps dropping) and NIC3 for guest and BYOD VLANs (no dropouts on this NIC). All the chipsets look the same on the GUI though.

 

 

Which NIC would be the management NIC?

[machin05]

....and first drop out of the day 10 minutes into the first lesson

[Oaktech]

Has it got latest firmware? We've had a similar issue cured by a switch firmware update.

[machin05]

Has it got latest firmware? We've had a similar issue cured by a switch firmware update.

 

Having real difficulty in finding any firmware updates for our switch. It's fairly old - HP A5500-48G EI

[willtech]

if you go to: https://asp.arubanetworks.com/downloads

 

Just the J model number e.g. JG240A it should display them.

[robk]

This is very interesting. Our current box is a config migration from an old box. We're using NIC1 for VLAN1 (the one that keeps dropping) and NIC3 for guest and BYOD VLANs (no dropouts on this NIC). All the chipsets look the same on the GUI though.

 

[ATTACH=CONFIG]67631[/ATTACH]

 

Which NIC would be the management NIC?

For us it's NIC 1. Different chipset to us though.

 

Might be worth trying a different NIC though.

[NetworkServices]

Have you tried disabling BPDU guard on the switch port?

 

no spanning-tree bpdu-protection

[psydii]

Do the old ex-3com / H3C products have J numbers? Latest version I could see is A5500_EI_5.20.R2222P12 March 2020.

 

As a troubleshooting step, if a new cable makes no difference, try putting another switch between the smoothwall and the 5500. That should confirm/eliminate issues of compatibility between the nics.

[machin05]

Have you tried disabling BPDU guard on the switch port?

 

no spanning-tree bpdu-protection

 

Yes, I did that fairly early on in my troubleshooting. It helped in as much as it brought the port back online a lot quicker, but didn't prevent it dropping in the first place.

[machin05]

Do the old ex-3com / H3C products have J numbers? Latest version I could see is A5500_EI_5.20.R2222P12 March 2020.

 

As a troubleshooting step, if a new cable makes no difference, try putting another switch between the smoothwall and the 5500. That should confirm/eliminate issues of compatibility between the nics.

 

Yep, it's got a J number. It's HP branded but old 3COM tech, I believe.

[machin05]

Do the old ex-3com / H3C products have J numbers? Latest version I could see is A5500_EI_5.20.R2222P12 March 2020.

 

As a troubleshooting step, if a new cable makes no difference, try putting another switch between the smoothwall and the 5500. That should confirm/eliminate issues of compatibility between the nics.

 

Yeah, not a bad shout.

 

Not to tempt fate, but Smoothwall have been on our box remotely - I'm not sure what they've actually done, but I suspect they've restarted the NIC as everyone lost connection briefly and the proxy authentication popped up on everyone's screen. Looking at the logs there is a NIC Link Up message but no NIC Link Down one prior to it! Anyway, since then I've not had a dropout, although Smoothwall themselves have escalated it to 2nd line.

 

Only other significant change I've made is to put a bandwidth limit on everyone's connection so no one can connect at more then 15mbps.

[machin05]

Problem seems to be resolved now - Smoothwall support connected remotely and, in their words, they "...updated the NIC from the backend"

 

Not entirely sure what that means, but we did lose connection for a minute or 2 with lots of proxy authentication errors. However, since then it's been rock solid so, touch wood, the problem has been solved.

 

Thanks again to everyone who took the time to respond and offer their advice - it's much appreciated!

[robk]

Problem seems to be resolved now - Smoothwall support connected remotely and, in their words, they "...updated the NIC from the backend"

 

Not entirely sure what that means, but we did lose connection for a minute or 2 with lots of proxy authentication errors. However, since then it's been rock solid so, touch wood, the problem has been solved.

 

Thanks again to everyone who took the time to respond and offer their advice - it's much appreciated!

 

That sounds like new NIC drivers, (similar fix tried on ours) good to hear its all up and working.