Network Infrastructure

[rusty155]

I have posted a problem in another area with regards to this but would I would appreciate some more general advice on upgrading our network backbone. I am getting all sorts of problems here and I am sure that a big part of it is due to the infrastructure here.

 

I'll include as much information as I can of the current setup, and would be grateful if others could perhaps post their setups, and their opinions of the best way to upgrade here.

 

We have 3 comms cabinets on site, I have included a very rough diagram of the setup in our main cabinet as this is the worst, although the other two are similar!

 

http://i124.photobucket.com/albums/p36/marmot155/files/Network.jpg

 

All of the switches are pretty much at full capacity. The diagram is slightly inaccurate as there are actually two pairs of fibre going from the core to each of the other to cabinets. Each of these pairs are linked to another switch which in turn have at least 2/3 switches which are connected by 100Mbit copper.

 

We have approx 320 client machines for 1100 students and 80 teachers.

 

The 3 servers are all DCs and file servers, with one of them also acting as our mail server. They are all P3 1000mhz.

 

How bad is this? and what should I ideally have for everything to run smoothly. I have my own ideas but would really appreciate some input.

 

Thanks.

[mrforgetful]

Off the top of my head (and I'm no network designer) it looks like you've got way too many uplinked 24 port switches.

 

I'd say you need to be using 48 ports connected via Fibre to a main core switch near your servers.

 

But like I said, I'm no genius.

[ChrisC]

To to totally honest depending on the physical distance between the core switch ( the maxswitch), you really want to have a 1gb copper or fibre link out to each, rather than daisy-chaining them, it would help reduce bottlenecks no end!

 

see file attached for the optimum infrastructure

 

Chris

[Lee_K_81]

YOu've got way too many daisy-chained switches. I would recommend that your 3Com 24 Ports all plug directly into youe NetworX MaxSwitch. That should help speed thigs up a bit. If you buy any more patch them directly into your MaxSwitch too. What do you use for your mail? It is usually recommended that your mail server is not a DC as this drastically increases the load.

[rusty155]

Yes! Thank you! I have been trying to get the school to fork out on a new core switch for over a year now!!

 

 

The maxswitch is at full capacity so i cannot link any of the 24 port switches straight into it. It only has 4 gbit fibre ports which link to the other 2 cabinets and 4 gbit copper - 3 of which are taken up by the servers and the other 1 by the first 3com switch!

 

I would like to replace the core with a Procurve chassis ideally, with modules for at least 12 gbit copper links (for the servers, and all switches) with room for future expansion. It must have at least 4 fibre ports also for linking to the other two cabinets. But then there are also similar issues in the remaining 2 cabinets, although the daisy chaining is not quite so insane!!

 

But it seems like an impossible task to get the smt to part with over 5grand. I would really like to meet the guy who kept daisy chaining all of these switches because it is giving me a constant headache!!

 

Any suggestions on convincing them to spend the money before the whole lot falls down, and I get the blame for shoddy extensions in the past??!!!

[mrforgetful]

Start turning off the one with SMT computers on it and tell them it's due to the infrastructure.

[ChrisC]

kill the server for a bit, explain that the switches are havingh trouble communicating?

 

How evil, but proving a point that would be!

 

Do you get many staff complaining of network speed? tell smt it would fix most if not all of those grumbles!

 

Chris

[Guest PatBoland]

Same story here!! Tried all the above tricks but SMT still do not get it. We just soldier on...

[ChrisC]

that's terrible!

 

Did you even mention how it impacts on teaching and learning?

That's always a good point to press on them!

 

Any chance you can get teachers on side to back you up and make the right noise @ smt?

 

Chris

[ranj]

Upgrading/replacing the your main file servers would give you advantages as well.

[budgester]

First thing I would do is actually get a handle of what connections you actually have.

 

I.e.

 

Core Cab

80 Ports

 

Cab1

80 Ports

2xMultimode fibre to core

 

Cab 2

2xMultimode fibre to core

 

etc.

 

What you are aiming for is a star with as few hops between switches as possible.

 

If you have any hubs whatsoever pull em out and replace them with a switch, doesn't have to be expensive.

 

If you can get decent switches everywhere, then you have the option of aggregating your fibre pairs, to give you 2gb instead of 2 * 1gb, this will allow better burst capacity.

 

There are lots of other considerations, i.e what is generating the traffic, is the specific places where you have bottlenecks.

 

Regards

 

Budgester

[rusty155]

Thanks for all of the replies.

 

Are there any good (free??!!) network monitoring tools to keep an eye on network traffic and expose bottlenecks?

 

This would certainy be a good start i think!

[marco84]

try NAGIOS

[PiqueABoo]

Hmmm.. haven't got one so they might be rubbish, but you can get a NetGear 12x1Gb L2 switch that claims you can add 12 fibre ports (mini-GBICs) for less than £400.

[eduabncs]

Nagios is good. But for monitoring bandwith. cpu & memorey use etc MRTG is better IMHO.

[localzuk]

Nagios is good. But for monitoring bandwith. cpu & memorey use etc MRTG is better IMHO.

 

I believe Cacti has replaced MRTG hasn't it?

 

Cacti is a very nice tool for spotting odd network behaviour - such as 100Mb spikes on individual workstations.

[Geoff]

No, rrdtool has replaced MRTG. Cacti simply provides a nice GUI interface to configuring and organising your graphs.

 

Cacti and Nagios really aim to do different things. Where as Cacti can tell you what was happening on your network last week Nagios can tell you what is happening now. So you really need to use both tools together to get a complete overview.

[sahmeepee]

Thanks for all of the replies.

 

Are there any good (free??!!) network monitoring tools to keep an eye on network traffic and expose bottlenecks?

 

I like PRTG on Windows. The free version only gives you 3 monitors at once, but I don't think the paid-for version is expensive. Very easy to set up.

[m25man]

If your diagram is correct it would appear to be fairly optimal for the equipment you have.

If the 3coms are correctly stacked they should be counted as a single switch and not 3 hops. These can be trunked to improve bandwidth from core to the more populated stacks.

 

I would be on the look out for broadcast traffic as you do not illustrate any routers or L3 routing switches configured you seem to be operating a single broadcast domain where one bad nic will kill the lot!

 

Unless you are load balancing these servers then use perfmon to watch the disk queue lengths as it is more likely with so many end nodes and so few resources the bottlenecks could be at the disks and not at the lan at all.

[rusty155]

Ok big update i'm afraid!

 

Ok!! I've been able to spend some time on this problem and have discovered the following:

 

First of all, the problem that I am having is not related to MS Access at all! I has just been highlighted by it due to the apparent sensitivity of Access to network issues.

 

The problem actually lies with the 3COM switches. They are all Superstack 3 4200 series switches (4226T more specifically).

 

They appear, for whatever reason, to be resetting themselves approx every 10-20 minutes. This is causing the workstations attached to them to drop off the network for about 10-15 secs before they pick up an address again.

 

If I try to manage the switches through a web browser, I receive the error:

 

"Error in SNMP read. Failed to get system information."

 

I am then prevented from viewing / changing any settings for the switch.

 

If I Telnet in, I am able to view most settings, and can reset to factory defaults. I have done this, which does not reslove the problem. I am not able however to view or change any settings with regard to the IP address. If I attempt to, the following error occurs:

 

"Invalid option, or combination of options selected"

 

Out of 12 3COM switches, across 3 separate cabinets, this occurs on 5 switches. The strange thing is, that the switches that I can manage are uplinked to at least one other 3COM switch. That is, the problem 3COM switches are either directly linked to the core switch, or pass through a non-3COM switch before the core.

 

Another strange occurance with regard to the first stack of 4 switches in the diagram is that I can get in to the management of the 2nd, 3rd and 4th switches in the stack, and thus view the stack as a whole. I am however unable to view or change the device settings of the stack - just see an overview. Within this overview, I can see the IP Address of the switch that I have pulled up through the browser, but the other switches in the stack show an IP of 0.0.0.0. This is the case for the last 3 Switches in the stack - I recieve the snmp error if I try to open up the management page of the first switch in the stack.

 

I am sorry for the bombardment!! - I have been looking into this for the past couple of days with an engineer sent from RM without much success. (He has never seen or heard of anything like this before!)

 

I am severely stuck at this point, and sinking increasingly out of my depth!!

 

As a further note, I have contacted 3COM who only offer support for 90 days after purchase, and are only able to assist via a paid support option which I really do not want to pursue at this time!! I cannot find anything of use on their knowledgebase either!

 

HELP!!

[Geoff]

Obvious things to check:

 

1) Are you suffering from a network loop and thus an ARP storm? Try enabling STP. Check with a packet sniffer for excess ARP traffic.

 

2) Make sure your switches are running the latest firmware. You may be the victim of a bug.

[rusty155]

Ok thanks Geoff - firmware update on all of the switches did the trick! :oops:

 

I thought about updating the firmware early on but thought i'd try a few other things first.

 

Why would these problems suddenly happen? one day fine and the next chaos!!?? Is buggy firmware common in switches and how does it come about?

 

Thanks again for all the help!

[western]

This problem seems to occur on a lot of the 3com switches, you may find that it will start happening again even if you have firmware upgraded them. I hav a customer with around 30 3com switches and they have all reset at some point radom like. We think we have sorted the problem out now by putting in a aggrigation switch (4200g) using the spf slots for the fibre in and the gig ports to manage the stack rather than cascading. To put it simply the 3com switches do not work well with the cascading as they are designed to be solo / edge switches (they in theory should cascade but dont always work!). If it starts happening again i would suggest buying some of these!!!