hello everyone,
i have recently run into a very, very strange problem and i'm hoping at least one of you have run into the same or similar... it might just save my sanity!
let me explain:
we've been having very brief periods of complete packet loss across our main backbone switch (all our switches are HP Procurves of various vintages). during this phenomenon the cpu on the switch spikes and the activity LEDs illuminate in a more solid fashion. This loss of traffic rarely lasts longer than 30 seconds, usually only 5-10. This happens very sporadically, sometimes 2-3 times a day, sometimes not at all.
now, i appreciate this *looks* like it might be a loop in the network, but we have STP enabled on all our switches and artificially inducing loops the lan, across switches causes no such loss. the switch log accessible from telnet reveals that the ports are instantly blocked by STP.
Anyway, further investigation seems to have identified the root cause of this issue -
It's *something* to do with the Marvell Yukon onboard LAN cards in some of our machines.
I had a machine in the office which had been reported as prodically displaying the no domain error and dropping the network and when i tried to reimage it, it would, with a 100% reproducibility cause a complete loss of traffic on the backbone switch - every single time. replace the lan card. problem goes away completely for a few weeks.
now, we have over 60 of these machines, but fortunately, these are all (mostly) contained on a couple of switches, so for a test, i've disconnected the switch where all these computers are plugged in and the problem (so far) has gone away.
so speculating this is likely caused by the lan card spewing either loads of broadcasts or some other malformed transmissions, maybe during bootup causing the switch to be temporarily overloaded dealing with those broadcasts and causing it to drop a few packets - the times of the network drops seems to mostly coincide with lesson start too. it doesn't seem to be related to a specific driver, i've tried several for the card, including the DOS one and they all cause the effect on the defective machine. Note: i'm not using multicast or anything else like that on the machine, it was a simple image download.
but, when this happens, it seems that the switch gets so bogged down with it all that it's very difficult to see that's going on the switch console as the connection is usually dropped.
is there going to be an easy way for me to work out where the rouge device is? other than by turning off each port in turn on the switch and seeing if the problem goes away?
any info/feedback greatly appreciated.
Mark