VERY SLOW NETWORK?

[GRitchie]

Morning,

Started yesterday morning, but for some reason our entire network is on a go slow.

 

Logins sticking on the 'welcome' screen forever.

 

Only recent things that have changed are:

• Updated Impero to beta 5.1.12
  
• Implemented new core switch (HP 2530-48G)
  

 

I can't determine what it is, and I don't know where to look tbh!

 

Help! I'm a network down!

[win]

My guess would be a bottleneck at the switch. At the next opportunity, you can narrow it down by swapping it with the old one (if you still have it)

[Steve21]

Only a personal opinion, but would take a slower approach to changing stuff over so fast Seems like a lots being done fast from a few threads recently.

 

But in regards to the "core" switch, don't you have any VLANs? As a 2530 isn't a layer 3 switch so couldn't do routing etc, What did you swap out as the old core?

 

Steve

[NokiX]

I believe this may be related:

 

http://www.edugeek.net/forums/network-classroom-management/158774-impero-5105-slow-network.html#post1407158

 

Try a roll back of your impero server as it's likely this will resolve the issue since it appears it's not just yourself with these issues.

[win]

on second thoughts, there seems to be a lot of threads now about the impero update so that looks like the number 1 suspect

[GRitchie]

Long story @Steve21

Old switch was a netgear switch

New switch is the HP2530-48G. I'd love to have a 'real' core switch, but for money reasons we can't at the minute. So to try and slowly get the network to gigabit speeds, I'm putting these HP's in place ASAP.

 

Eventually, I'll swap the 2530-48 'acting as' core switch, to a real HP core switch with the fibre modules etc. (I hope)

 

 

No VLANs.

 

 

 

Surprisingly, I'm not actually doing a lot of what I'm posting just yet. Just building up a plan/ideas of how long things will take and cost. The help from these threads means I can put things together to present a bit easier :-)

[Steve21]

Ah that's alright as long as it's not VLANd Just thought could be lack of routing causing problems if you didn't have a L3

 

The other things I'd check in terms of switch side, is if you had multiple connections from the server to switch did you migrate things like trunks across? Else could have caused loops

 

Steve

[markwilfan]

Could be a packet storm if it is not Impero. Look for any looped cables. Your HP switch will be able to help with that if it is.

 

Worth a read into spanning tree. But loop protect will help if it is this

 

loop-protect all

loop-protect trap loop-detected

loop-protect disable-timer 60

 

what are the LEDs on the port doing? Are they blinking or solid

[GRitchie]

No, fortunately no VLANs at the minute @Steve21. Also, there were no trunks either :-)

 

Thanks @markwilfan - I'm in talks with Impero atm. If rolling back doesn't fix it, I'll look into this too :-)

[markwilfan]

No probs @GRitchie

 

I feel your pain. We have quite a big site and sprawling network. Spanning tree hadn't been configured here as it is quite scary if you get it wrong. Definitely worth a look. 1 looped cable floored our network without loop protect and spanning tree so it is defo a possibility.

 

Needless to say STP and loop protect is sitewide now

[rad]

WSUS updates? Had a few new computer builds do updates this morning.

[twin--turbo]

I would suspect a loop / spanning tree issue too.

 

TT

[mikeyd101]

i'd be looking at traffic on the switch with something like wireshark, this should give some indicators where the traffic is coming from rather than making blind changes. Also the management on the switches looking at individual port traffic.

 

I'd also double check for network loops and you recent Impero and possible reasons.

[twin--turbo]

WSUS updates? Had a few new computer builds do updates this morning.

 

LOL.. Yeah watch out for PC's/Servers that are not on WSUS and doing direct internet updates. Windows Update (and WSUS ) swallows as much bandwidth as it can. One of my colleagues proved this today trying to fetch updates for 2012 during the day.

 

 

If it looks like network, then disconnect all the Distro/Edge switches from the new core and see if things improve. Then add them back in one at a time and see if a particular one causes increased traffic. If it does unplug every cable from the edge switch and add each back in.

 

The common way for students to DOS a network is to loop an outlet into the outlet next to it.. Without spanning tree and BPDU Guarding the network will fall to it's knees.

 

TT

[psydii]

I'd reach straight for Windows Performance Analyzer to a boot/logon analysis, and then dive off into wireshark if WPA suggested it was the network.

 

Given how deeply it hooks into Windows it does sound likely to be a problem with the beta-release software you've just installed though.

[twin--turbo]

Only a personal opinion, but would take a slower approach to changing stuff over so fast Seems like a lots being done fast from a few threads recently.

 

Indeed. I did love the days when I was care free and just got on and did stuff.

 

Now constrained by "Change Control" which can slow things down ( it's going to take 2 weeks to get to test a printer driver) .

 

But it gives time for consideration.

 

And as all changes are tracked we can review every aspect of what we have done...

 

 

 

 

Bloomin pain in the backside when the "Change Police" pull you up for trying to slip something in under the radar

 

TT

[GRitchie]

Okay, Impero seemed to fix the problem, and now it hasn't.

Still slow :-(

 

I'm concerned that either

A) Impero is still the issue, we upgraded rather than downgraded

B) There's a loop. But finding one could prove to be rather challenging...

[Steve21]

Okay, Impero seemed to fix the problem, and now it hasn't.

Still slow :-(

 

I'm concerned that either

A) Impero is still the issue, we upgraded rather than downgraded

B) There's a loop. But finding one could prove to be rather challenging...

 

Turn off Impero temporarily and see if it resolves it.

 

Personally I'd go with the network side of things if that has been changed recently.

 

Easiest way would be first look at the switches (depending how many there are) and see if any are all full lights (full lights on, compared to flashing = lots of traffic)

 

Secondly, try disconnecting all other switches but core and see what response you get logging in directly to the core

 

Steve

[FragglePete]

Okay, Impero seemed to fix the problem, and now it hasn't.

Still slow :-(

 

I'm concerned that either

A) Impero is still the issue, we upgraded rather than downgraded

B) There's a loop. But finding one could prove to be rather challenging...

 

I would still be very suspect of Impero - their track record of late is worrying.

 

Years ago, when we were testing Windows 7 with Impero 3 we had major speed issues with it and could prove it with side by side comparisons. To their credit back then, they responded quickly and dealt with the issues as we found them.

 

As for a network storm/loop, my experience of these is that it brings down the network completely rather than making it slow. But as people say, look for the switch that is all lit up with solid lights when it would normally be flashing with activity.

 

Pete

[GRitchie]

Right. I've had a wander around, can't see any obvious loops (couldn't check everywhere) and all the switches seem to be fine, given a few a restart.

 

It's intermittent with staff, and ping responses from the core switch are sky high on those machines with problems.

 

I'll be honest, I have no idea how to use wireshark, so I don't know how to use that to determine anything.

 

 

I go directly into the core switch (via a PP) and my ping responses are fairly quick. The odd flicker up to 30ms but nothing more.

[Latham]

*Sorry - posted in the wrong thread! Mods can delete this. Apologies. Edited January 13, 2016 by Latham

[mikeyd101]

You don't need to know that much to use wireshark, it just gives you a view on the traffic (source / destination and description). There is plenty of information like this video below I quickly found to get started with wireshark. I had a problem recently with some new hp elitedesk 800 and they were outputting tens of thousands of ipv6 packets per second when sleeping, turned out to be the network driver. but no way to begin to troubleshoot it without visibility to what and where traffic is generated by.

 

I'm not great with wireshark as I don't use it day to day so I just find something that i think is generating lots of traffic and google the destination port, protocol or message to find out more about what type of message it is.

 

 

with problem like slow network your just really looking at bulk volume of traffic, with what ips are generating all that traffic, or broadcast storms/loops where all the data will look the same. Also if the problem is Intermittent it might be when something specific happens, like my problem above where machines were sleeping which was also intermittent.

 

if you have ips assigned to your switches you can use some other tools like ping plotter or the dude to monitor longer term ping responses e.g over a couple of hours.

[pantscat]

What switches have you got throughout your site? are they managed?

If so - look at the port stats/counters for all the switches and see if there are any that are showing any particularly unusual spikes of activity - especially "errors rx".

 

Get loop-protect up and running on the core switch, also - what are you pinging from the core switch that means you get a response time of 30ms? You should be getting 1ms or less if it's directly connected.

 

More than likely a network loop or a duff device spamming out crap.

[GRitchie]

Still no idea what's going on here. Feels like a loop, as I've had a similar experience in my previous workplace with the same symptoms.

I've tried analysing with WireShark, but everything looks normal? Nothing sending huge bytes etc. (not that I'm overly familiar with WireShark).

 

Switches all have loop protection (those that are manageable) turned on, so it shouldn't be happening...

 

It's across the entire site, but not everyone is affected.

[pantscat]

Is it happening to PCs across the site? Or just one particular area?

I'd unplug all the network segments from your core and then plug them in one by one to see if/where the problem occurs.