How do you wire your cabinets?

[maniac]

Just after some opinions here on the way people do things.

 

Out wiring topology is generally quite good. We have a central wiring cabinet, and a dedicated fibre link to each cabinet at gigabit. Some of our cabinets are quite large, one serves 3 ICT suites, loads of offices and classrooms and a large part of the wireless system. This used to be wired so the switches were chained off one-another, fibre went into first one, then copper gigabit into the next, into the next etc. This linked about 8 swithes or so in one line.

 

I wasn't very happy with this, seeing that if a switch failed then all the rest underneath it would also be affected, and also thought it created a bit of a bottle neck on the network, so I changed it this summer using a HP 2824 switch, 2x gigabit fibre coming into that from our main cabinet, then a gigabit line to each switch in said cabinet.

 

I've never really been taught best practice when dealing with wiring, I've always used my common sense, and I've now got the possibility of doing this in the second of our big cabinets. I was after peoples opinions as to whether it's actually worth it or not, or if there's any real benefit to changing it from the switchs chaining off one another, to a locallised gigabit switch to act as distrabution.

 

Any opinions gratefully received.

 

Cheers,

 

Mike.

[MkII]

Well the reduced bandwidth of the last switch down the line doesn't bear thinking about for a start.

[Teth]

Is there only a single pair of Fibres terminated in the large cabinet?

 

I would have thought that very unusual as a fibre "cable" tends to have a minumum of 3 pairs and if your pulling the cable you might as well terminate them all. I would use more of these fibre pairs to get more bandwidth and redundancy from your Core switch out to that cabinet. Fibre GBICs are cheap these days.

[enjay]

The set-up that you changed your first cabinet to sounds ideal - repeat that in all the others. Daisy-chained switches are best avoided wherever possible, especially (as Mark points out) from the point-of-view of the poor souls on the bottom switch. Your PCs should be as few steps from the server as possible, on as fast a line as possible, so get yourself a nice big fully-gig switch in each cabinet and then feed the individual switches off that.

[localzuk]

The concept I go by is 'least hops'. There should be as few hops between the server and the end client as possible. Here we have a fibre going from the core to every switch. So, if one switch dies, only that switch is affected. It also means that bandwidth doesn't decrease depending on where you are in a chain.

 

So using an aggregator like you are doing reduces the number of hops considerably, and should increase network latency and bandwith to a good number of devices.

 

Also, when you say you have a fibre to each cab, is it only a 2 core fibre (ie, can it only support a single link)? If it has more cores, why not trunk them together to make a 2 or 4GB link back to the core? That way, when you have your switches connected to the 2824, they have 2 or 4 times the bandwidth available.

[bossman]

Full star topology:

Gbic core switch central comms cab with all other Gig switches linked by fibre (OM3) directly back to core switch. Server switch links via Gbic to core switch voila full gig capacity. When we move on to 10gig switches all we have to do is swap out the switches as the infrastructure is already in place.

[Oops_my_bad]

Full star topology:

Gbic core switch central comms cab with all other Gig switches linked by fibre (OM3) directly back to core switch. Server switch links via Gbic to core switch voila full gig capacity. When we move on to 10gig switches all we have to do is swap out the switches as the infrastructure is already in place.

That's exactly the way Becta recommends (dictates? ) you do it, and also the way corporates have their infrastructure

 

Only potential pitfall is you have a single point of failure - ie your core switch. But just keep one spare and configured ready just in case

[enjay]

Only potential pitfall is you have a single point of failure - ie your core switch. But just keep one spare and configured ready just in case

You can never totally eradicate the single-point of failure, even if you had some fail-over on the switches, the server still only has one Ethernet cable in it after all!

 

We do exactly the same as you - i.e. have a cheapy spare switch sitting on the shelf which we can drop in should one go wrong somewhere. It's not a very impressive switch, so I would always swap it back out when the replacement arrived, but it does at least keep the network running during that time.

[localzuk]

Only potential pitfall is you have a single point of failure - ie your core switch. But just keep one spare and configured ready just in case

You can never totally eradicate the single-point of failure, even if you had some fail-over on the switches, the server still only has one Ethernet cable in it after all!

 

Our servers have 2 cables...

 

We do exactly the same as you - i.e. have a cheapy spare switch sitting on the shelf which we can drop in should one go wrong somewhere. It's not a very impressive switch, so I would always swap it back out when the replacement arrived, but it does at least keep the network running during that time.

 

I still have all our old Dlink stuff, and cabling with which to hook it up should something fail. The warranty on our switches is 24 hour replacement though, so it should only be using the old tat for a short while...

[Joedetic]

"You can never totally eradicate the single-point of failure, even if you had some fail-over on the switches, the server still only has one Ethernet cable in it after all!"

 

*cough* Ours had dual gigabit with failover *cough*

 

Anyway, yes, we had a big procurve chassis switch in the server room which handled the core switching with fibre links to all the other cabinates in the school which were then in turn 100mbit to the desktop.

 

In the server room there was gigabit copper between all the switches for the ICT department (there were 6 or 7 of those i seem to remember) and the chassis switch (in the same cab). All the fibre terminated in the server room also.

 

We didn't have a fail over core switch in place in case of failures. There just wouldn't have been the room for it for a start! But there were lots of replacements for other switches hanging around and GBic modules aswell i think.

[enjay]

Only potential pitfall is you have a single point of failure - ie your core switch. But just keep one spare and configured ready just in case

You can never totally eradicate the single-point of failure, even if you had some fail-over on the switches, the server still only has one Ethernet cable in it after all!

 

Our servers have 2 cables...

Pfft :-) You take my point, though. You cannot prevent every single possible point of failure (short of having a spare server mirrored and ready to go!) but you can aim to reduce the number of such points as far as is possible.

[SYNACK]

We have also have a star topology with all switches linked directly to the core. The main cabinet which houses the core also has several access layer switches in it. These are linked to the core by 4GB trunks over 4x1gb Ethernet ports. Each server has dual(edit: two individual) 1GB network cards which are also trunked into the core switch.

 

We are just in the process of adding additional fiber links to the other cabinets which will take them up to two 1gb fiber links each (from the one they have now).

 

The only true way to get rid of the single point of failure is to use a full mesh topology which is extremely expensive. In this each distribution cabinet connects to all of the other cabinets this means that any cabinet can fail and be routed around by the network. You of course need your servers connected with fail over links to more than one switch to accomplish this.

[Teth]

Out of intrest and hopefully without derailing the thread too much.

 

For adapter teaming on your servers what do people use.

 

Load balancing by Windows TCP/IP

 

802.3 LAG Protocol (which obviosuly requires support on switches)

 

Or HP / CISCO Proprietry solutions?

 

I've had some problems in the past getting our Broadcom cards to negotiate LAG correctly. One of them abjectly refuses until checksum offload is disabled. I've always meant to do some throughput testing on the different solutions but never found the time.

[DMcCoy]

With the dual port NICs its not that redundant anyway. Same card

[localzuk]

With the dual port NICs its not that redundant anyway. Same card

 

Indeed! Thats why you should have 2 separate cards in your server. And 2 servers, for redundancy, and 2 switches, and 2 ups's connected to 2 separate power rings, and 2 aircon systems

 

By the time we're done we have enough equipment to send a shuttle into space... And no money.

[Joedetic]

We had two separate Intel ProSet NICS in our DCs that were teamed using their software in failover mode.

[CyberNerd]

With the dual port NICs its not that redundant anyway. Same card

 

Indeed! Thats why you should have 2 separate cards in your server. And 2 servers, for redundancy, and 2 switches, and 2 ups's connected to 2 separate power rings, and 2 aircon systems

 

By the time we're done we have enough equipment to send a shuttle into space... And no money.

 

vmware for consolidation

[localzuk]

vmware doesn't really provide redundancy though does it? In fact, it adds more single points of failure than before...

[limbo]

We have the problem of three separate buildings so the best we can achieve is a core stack in each building joined by fibre (4Gb) (all Cisco Kit)

 

There are a few places where the sockets are not all running back to the central location so here a switch is placed in the room with a 1GB link back to the stack.

 

Every stack has a PoE switch to power the wireless network.

 

The servers are all connected to the main stack in the central building with two NIC's teamed using the HP software supplied. Apparently this means they can only recieve on the one NIC but can send data out through either.

[enjay]

Thats why you should have 2 separate cards in your server. And 2 servers, for redundancy, and 2 switches, and 2 ups's connected to 2 separate power rings, and 2 aircon systems

You're forgetting the spare IT Support Dept, just in case you're caught up in whatever happened to the first network :-)

[CyberNerd]

if a physical server dies esx will automatically migrate the server to another box - providing 100% uptime. Even with the cheapskate version I can run 3 virtual servers on one box - then have a spare to move them to if the server dies. so rather than have 3 physical servers + 3 physical spares I can run it all on 1 physical server with 1 spare,

[dave.81]

I run our system similar to the first post, well in the main cab anyway the others are a work in progress. Our main point that will cause mayhem when it fails is out core fibre switch in the main cab. 7 fibre links to 95% of the college if it fails so does the college. Although the admin/office department has a backup Cat5e cable just incase.

[DMcCoy]

if a physical server dies esx will automatically migrate the server to another box - providing 100% uptime. Even with the cheapskate version I can run 3 virtual servers on one box - then have a spare to move them to if the server dies. so rather than have 3 physical servers + 3 physical spares I can run it all on 1 physical server with 1 spare,

 

This is only possible with the vastly expensive high availability addon, even I don't have that sort of money to spend!

 

I can do it manually though.

[maniac]

Blimy, 6 hours away from the computer, and this is what happens to my thread! lol

 

In answer to some questions, the cabinet in question has 2 pairs of fibres, and they are running as a trunk giving a 2gb uplink directly to our core switch, so no problems there. The fibre in this place is a bit all over the place. Each cabinet has at least one pair going to it, some have 2, and some have more but we're not sure where they all go! One has 4 pairs back to our server room, but 2 of these come in, then go out again on to other cabinets.

 

I'm desparently trying to optimise the cabling in this place, I know ideally every switch would link directly back to the core switch, but this just can't happen as there's no money for new fibre, so what I'm doing is the next best thing and looks like it's the right thing to be doing, so carry on as I am I think!

 

cheers,

 

Mike.

[enjay]

Ah, now you didn't mention that you're unable to increase the number of fibre links...

 

Thinking of the cabs which have the two pairs trunked into a single 2gig uplink for a big stack of switches - is there any mileage in re-arranging these so that two smaller stacks each have 1gig uplinks? That seems to me (in my slightly tired end-of-the-day state) to offer a slightly better connection.

 

Also, consider re-ordering your stacks so that the switches feeding the lowest-usage computers are at the bottom. That way the ones which get hurt most for bandwidth are the ones which need it the least.