chrisjako Posted November 19, 2013 Posted November 19, 2013 I'm trying to troubleshoot a major issue we are having, we've just built some new PC's (100 in total) - I opted to go for an intel board DH87RL and the onboard NIC is an INTEL I217-V at random times of the day our whole network which is consists of HP procurve's will just crash out, the switches look like they are being looped, spanning tree protocol is enabled so this should not happen and in fact doesn't if we try to loop it manually so we can rule that out, to resolve the problem at first we simply rebooted switches and everything came back online and was fine again. During the next crash occurrences we then started to manually unplug fibres so get to the root of the issue and then once we found the offending switches we narrowed it down to the ports, in the many times we have managed to find an offending end port its always been once of the new Intel PC's. Bios has been updated, NIC drivers have been updated and the problem is still happening, when we reach an offending PC its normally just sat there waiting for someone to login and simply unplugging and re-plugging the cat5 cable fixes the issue (hence the reason in the beginning rebooting the switches fixed the issue) Has anyone ever had any similar issues with NIC's crashing the network? It's new to me in my 15 year career have never seen it happen before and none of my team have ever heard of it happening before and had to see it for themselves to believe it! I have a case open with intel support who seem sceptical about it, send them some logs of my switches yesterday but it doesn't help me in the meantime We built these PC's ourselves over the summer holidays to save on cost, the thought of having to rebuild another 100 to swap out the mainboards just doesn't seem like a viable option at the moment, also forking out another £1k for PCI-E nics also doesn't seem fair when we selected to purchase the Intel DH87RL board specifically to begin with as its supposed to be a flagship workstation motherboard which wouldn't have such issues!
stevewill6 Posted November 19, 2013 Posted November 19, 2013 Could it be a faulty Network Card? I had a PC with a faulty NIC in my last school and it would crash the switch. We replaced the NIC and the issue went away.
mrbios Posted November 19, 2013 Posted November 19, 2013 Have you tried updating switch firmware? (it's the only thing you haven't listed) Anything occuring on the logs of a PC when the event occurs? and what are your switch logs saying when the issue arises?
3s-gtech Posted November 19, 2013 Posted November 19, 2013 We had it on some old P4s with Intel NICs, they'd create a packet storm which would eventually take the whole network out (HP Procurve too). When we traced it back, we'd find it was one or two PCs. We tried fixes, in the end I just chucked them - the motherboards were faulty in those instances. Not an option with 100 new boards though! Are you perhaps getting some odd behaviour when the system is trying to shut them off to save power (Device Manager/Network adapters/Properties/Power Management) ?
EvLPhenom Posted November 19, 2013 Posted November 19, 2013 We have had issues with some old P4s. As @3s-gtech has said they would create a packet storm that would eventually grind the network to a halt. This usually happened when the machines would go into standby and wouldn't "wake" back up again. In the end we stopped the machines going into standby during the school day and was shut down at night. They were replaced shortly after. Sorry its not much help but at least you know your not the only one. May be worth checking stand by and see if you can replicate?
chrisjako Posted November 19, 2013 Author Posted November 19, 2013 Yes all switch firmware is up to date, however we had to downgrade our 5400ZL due to the fact in PCM it showed as 100% RX errors in the top traffic overview the only log which is related to the intel card shows when it was disconnected and when it was reconnected, no other errors to do with networking or realted
chrisjako Posted November 19, 2013 Author Posted November 19, 2013 We had it on some old P4s with Intel NICs, they'd create a packet storm which would eventually take the whole network out (HP Procurve too). When we traced it back, we'd find it was one or two PCs. We tried fixes, in the end I just chucked them - the motherboards were faulty in those instances. Not an option with 100 new boards though! Are you perhaps getting some odd behaviour when the system is trying to shut them off to save power (Device Manager/Network adapters/Properties/Power Management) ? No power management in place apart from impero that opens them at 8:30am and shuts them down on lunchtimes and end of school, the crashes occur nowhere near these times
chrisjako Posted November 19, 2013 Author Posted November 19, 2013 Good to hear other people have had the same issue, in fact researching it I found this article which has opened my eyes to how important it is to use non-Microsoft drivers and keep your NICs up to date anyway: HP ProCurve Networking Information Library - HP ProCurve Networking I think we will have to bite the bullet and fork out £1k in PCI-E cards as the network has been unstable for nearly a month now whilst we have been trying to get this resolved
chrisjako Posted November 20, 2013 Author Posted November 20, 2013 when the crash happened today wireshark captured this
edutech4schools Posted November 20, 2013 Posted November 20, 2013 Had a load of network crashes and a lot of head scratching until I found the cause was a HP black and white laser printer. When removed from network all worked fine.
chrisjako Posted November 20, 2013 Author Posted November 20, 2013 Had a load of network crashes and a lot of head scratching until I found the cause was a HP black and white laser printer. When removed from network all worked fine. May have been using IPv6 and screwing up like these PC's
wperson Posted February 17, 2014 Posted February 17, 2014 did you ever find a solution for this problem.. we too are having the exact same issues you have stated.. we also have many pc's with the intel dh87rl boards where 3 of them this problem keeps re-occurring. tried reloading software, switching network ports and cables.. nothing seems to resolve this issue except a disconnect
DMcCoy Posted February 17, 2014 Posted February 17, 2014 Do people with the issues have multiple vlans, or a single one, plus is IGMP on or off on all the procurves/vlans?
seawolf Posted February 18, 2014 Posted February 18, 2014 I'm trying to troubleshoot a major issue we are having, we've just built some new PC's (100 in total) - I opted to go for an intel board DH87RL and the onboard NIC is an INTEL I217-V at random times of the day our whole network which is consists of HP procurve's will just crash out, the switches look like they are being looped, spanning tree protocol is enabled so this should not happen and in fact doesn't if we try to loop it manually so we can rule that out, to resolve the problem at first we simply rebooted switches and everything came back online and was fine again. During the next crash occurrences we then started to manually unplug fibres so get to the root of the issue and then once we found the offending switches we narrowed it down to the ports, in the many times we have managed to find an offending end port its always been once of the new Intel PC's. Bios has been updated, NIC drivers have been updated and the problem is still happening, when we reach an offending PC its normally just sat there waiting for someone to login and simply unplugging and re-plugging the cat5 cable fixes the issue (hence the reason in the beginning rebooting the switches fixed the issue) Has anyone ever had any similar issues with NIC's crashing the network? It's new to me in my 15 year career have never seen it happen before and none of my team have ever heard of it happening before and had to see it for themselves to believe it! I have a case open with intel support who seem sceptical about it, send them some logs of my switches yesterday but it doesn't help me in the meantime We built these PC's ourselves over the summer holidays to save on cost, the thought of having to rebuild another 100 to swap out the mainboards just doesn't seem like a viable option at the moment, also forking out another £1k for PCI-E nics also doesn't seem fair when we selected to purchase the Intel DH87RL board specifically to begin with as its supposed to be a flagship workstation motherboard which wouldn't have such issues! We had a single faulty NIC on a desktop that began flooding the network with broadcast traffic. It would cause the switch it was connected with to become completely non-responsive and then the traffic would slowly begin cascading across the network. We identified it with PRTG and the top 10 bandwidth report. Fixed the faulty NIC and solved the problem. Maybe you have a bad batch of NICs (low probability) or mismatched drivers and OS, or an incompatibility with the motherboard, NIC, OS combination.
seawolf Posted February 18, 2014 Posted February 18, 2014 when the crash happened today wireshark captured Unless you are actually using IPv6, I would recommend disabling it in the clients. We do.
seawolf Posted February 18, 2014 Posted February 18, 2014 Do people with the issues have multiple vlans, or a single one, plus is IGMP on or off on all the procurves/vlans? Multiple VLANs, IGMP turned on for the VLANs where we do multicasting.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now