If your network suddenly goes slow.....

[kennysarmy]

Where do you look?

 

Has happened twice today - enough that the phone has gone hot....

 

I've checked event logs for every server and nothing to suggest a problem.

I've checked the VMWare stuff and performance graphs suggest nothing either.

 

I've set a pc in my office to ping and output to a file for the 3 core switches and one of the servers and will analyse the results in the morning.

 

What other tools could I use to monitor what happens?

 

Hate these random, can't replicate issues....

[tmcd35]

Sudden complete slowdowns of the entire network is usually a sign of a network storm caused by loop - lan cable between two sockets. When it happens it's usually a case of wandering around the school until you find the culprit. Harder to find are faulty nics that can cause the same problem. Setting up spanning tree on your switches can help alleviate the problem.

 

Of course it's not the only cause, just the most likely (from experience). This term we've had major network slow down issues here that where caused by bad profiles.

 

We retired a couple of old file servers and of course every ones profiles contained 'recent document' links to files on the old servers. Now we did anticipate the problem and deleted all the profiles stored on the server in order to get new ones created. We didn't take in to account that instead of creating fresh profiles Windows would just upload the old copies of the profiles from staff laptops back on to the server. So everyone has been complaining their computers are running very very slow because they are searching for files that don't exist.

[strawberry]

I'd set machines pinging each other to see if its a data transfer problem, but it also might be a multitude of single instance problems, for instance if your new anti virus is doing a scan at the same time.

[kennysarmy]

Pings to my core switch look pretty solid...

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=8ms TTL=255

Reply from 10.7.43.171: bytes=32 time=20ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=8ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=29ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=14ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=6ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=30ms TTL=255

Reply from 10.7.43.171: bytes=32 time=5ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=4ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=3ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=32ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

Reply from 10.7.43.171: bytes=32 time=2ms TTL=255

Reply from 10.7.43.171: bytes=32 time=1ms TTL=255

 

Have checked the SOPHOS console and nothing there looks out of the ordinary...

[pete]

I don't need to look - Nagios is texting me about what's wrong and Cacti is creating pretty graphs to cross-reference that.

 

Network monitoring is your friend - it greatly decreases the amount of problems that can't be fixed before end-users experience the effects, it increases your confidence in the kit (or points out poor performing kit) and it means you have solid baseline figures of how it should be to compare against the current status,

 

To accomplish this there are loads of tools, some trickier than others, but all free (or they have a free, limited version):

 

Nagios: Shows a snapshot of what's happening and available Right Now, with some historical logging.

Cacti: Historical trends to identify bottlenecks/wierdness at certain times of day.

Zenoss or Zabbix: Which combine the functions of Cacti and Nagios.

OSSIM: provides Nagios, Ntop and a bunch of security tools (Snort, Ossec, Nessus etc).

There's also The Dude or Spiceworks if you must run monitoring software on Windows.

 

This takes time to setup, but it repays the effort quickly. I'd rather do without vim than lose monitoring,

Edited September 10, 2010 by pete