Networking Issues

[HodgeHi]

Hello all,

 

My day has started off pretty badly. I have been having networking issues on and off now for a while. It seems as though there is a broadcast storm happening which cripples the network. I have enabled the STP protocol but this doesn't seem to do anything. All the switches seem to be affected but restarting the switch with the servers connected generally resolves the issue.

 

I'm not entirely sure what I need to look for and where to start troubleshooting an issue like this.

 

My switches are SRW2048 switches.

 

I have a number of things I could enable but not sure what they are and how they would affect the network. For example Flow control is currently off for all ports. Should i enable this. The STP setting has a few options as well. Which one is the right option?

 

Hope someone can help me with this. It's driving me nuts!

 

Thanks.

[sted]

unfortunately best think i can think to do is unplug network leads one by one (after trying to isolate which switch it is by unlinking them and watching the lights to cut down the number of cables you have to check) some kind teacher/cleaner/kid has probably looped the network

[HodgeHi]

I forgot to mention that this problem occurred over the weekend. The Teachers came in this morning around 7-ish and could not log on. But it is very intermittent. It hasn't happened for a while now. But it's re-occurring frequently again at the moment. Sometimes 2-3 time a day.

[AngryITGuy]

Agree with sted, you need to disconnect your switches from each other to try and isolate the problem on your network. Assuming its a cabling problem (ie a loopback) you will need to unplug the cables from the switch and watch for when the light show calms down!

 

It could also be caused by a virus infected machine on the network, so running something like Wireshark will help to isolate any machines on your network that are broadcasting huge amounts of traffic and data across the network.

 

Does the broadcast storm happen at any particular time of the day or is it just random?

 

How many servers do you have at the school?

[mstarkie]

I dont know the layout of your school but first thing worth checking is any work areas where staff/children can plug their own laptop in. Most common fault in my experience is people see a network cable dangling and plug it into the point next to the one its already plugged into (without realising).

 

If nothing obvious you could set up a laptop on one of main switches on a repeating ping (ping -t) and start fault finding.

 

However your switches are managed and there should be away to connect in to these directly and it should tell you what is at fault. But im not completely sure on this.

 

Dont worry we have all been there.

 

MS

[HodgeHi]

Thanks guys for the help. Really appreciated.

 

I don't think that it is a cable looping issue though. I experienced this at my last place. The reason i don't think it is this is because when i reset the switch the problem goes away. This is without messing with any cables. If it was a looped cable the issue would surely come back as soon as the switch brings the looped data points back online?

 

The virus thing may have something to it though. I will take a look at the Endpoint console and see what comes up.

 

"Does the broadcast storm happen at any particular time of the day or is it just random?"

 

It is very random, which is why I am beginning to suspect a possible virus. Maybe brought in on a memory stick which only starts being activated once plugged in?

 

Is this plausible?

[danrhodes]

Try running Agilent Network Analyzer, it will tell you where the broadcast traffic is coming from.

 

Agilent | Free Download - Network Analyzer Software Standard Edition

 

I leave it running, its very good to spot these kind of problems early as it has a good GUI.

 

D

[danrhodes]

Thanks guys for the help. Really appreciated.

 

I don't think that it is a cable looping issue though. I experienced this at my last place. The reason i don't think it is this is because when i reset the switch the problem goes away. This is without messing with any cables. If it was a looped cable the issue would surely come back as soon as the switch brings the looped data points back online?

 

The virus thing may have something to it though. I will take a look at the Endpoint console and see what comes up.

 

"Does the broadcast storm happen at any particular time of the day or is it just random?"

 

It is very random, which is why I am beginning to suspect a possible virus. Maybe brought in on a memory stick which only starts being activated once plugged in?

 

Is this plausible?

 

 

It seems more plausable that it may be a faulty NIC on a computer or printer that is getting turned on at a certain time and flooding the network with broadcast traffic, use the tool above and you will be able to track where all the traffic is origionating.

[AngryITGuy]

 

It is very random, which is why I am beginning to suspect a possible virus. Maybe brought in on a memory stick which only starts being activated once plugged in?

 

Is this plausible?

 

 

Very much so if the virus has created a 'autorun.inf' file on the memory stick the virus will run eachtime it is plugged in!

 

We had a similar problem here a few years back during an OFSTED Inspection think the suspect file being ran was called 'win.exe' which came from an infected memory stick and ended up infecting networked machines!

 

Does sound like your problem is likely to be a virus of some sorts, hope you find the source of the problem soon!

Edited May 4, 2010 by AngryITGuy

[danrhodes]

Yeah, there has been an increase of these annoying things running autorun.inf from usb sitcks lately! Just about had my fill of them now, its time to think of something new!

 

D

[HodgeHi]

Thanks guys.

 

@ DanRhodes,

 

Thanks for the heads up with this app. Will give it a shot as soon as i get some time.

 

@ AngryITGuy,

 

This is interesting. You mention the 'autorun.inf'. I have recently been dealing with this all over Staff memory Sticks. I have Endpoint configured to scan on access but it fails to pick them up or delete them. I am leaning more and more to the virus thing as we chat. But I won't rule out other things. IE our main server (DC) is running really slow which also has the endpoint admin console installed. I know, it's not probably recommended but we didn't have another server at the time

 

In the 6 weeks I will be moving this across to a different server (along with the print queues )

[AngryITGuy]

Scanning the servers would also be a good bet but you might want to leave that till after school as it's likely to slow down your netowork, but then again if it's already slow will anyone notice?

 

You might want to setup a machine to ping one of the servers overnight to see if the problem still occurs with no users on the network this should indicate if its a problem with your switches/servers rather than client machines.

 

Something like Colasoft Ping Tool would do nicely its free and has a graphical interface to show you the response times.

[cookie_monster]

Try running Agilent Network Analyzer, it will tell you where the broadcast traffic is coming from.

 

Agilent | Free Download - Network Analyzer Software Standard Edition

 

I leave it running, its very good to spot these kind of problems early as it has a good GUI.

 

D

 

 

Agilent Network Analyzer looks good, does it need to be on a box that's plugged into a switch port that's set to promiscuous mode?

[danrhodes]

No just plug it into any port it does not need it to be on promiscuous mode that I am aware, I have always just plugged it in and off you go.

 

D

[kennysarmy]

I am trying to get hold of the Network Analyzer Software Standard Edition software - but can't seem to see a download link on their site

 

Can anyone email me it or help please?

[Obzen]

I am also interested in Network Analyzer Software Standard Edition but can't seem to find it, does anyone link me to it or send?