How fast are your links to feeder cabinets? What are you using Fibre/Ethernet, both?

[ronnoco]

Hi all,

 

I'd be really interested to know what links schools have from their main cabinet to feeder cabinets. Have you got 1 fibre cable to each cabinet? are you using Ethernet or both, is the fibre OM1 (1gb) or OM3 (10Gb) etc.?

 

We have one main cabinet with 2 X HP 4108GL's connected together via GB stack. Our 7 feeder cabinets all have an OM1 1Gb fibre cable going to one switch in the cabinet. Some cabinets have 1 switch, some 2 and some even 3 or 4. To increase speed quickly and cheaply we have run some CAT5E to a couple of the cabinets close by which have multiple switches. The top switch is already connected via fibre then for the other 2 or 3 switches we have added a CAT5E cable which runs back to a 1GB port on the core switch. It seems to work well and you can pick up 6 or 24 port modules for the 4018GL pretty cheap.

 

All our HP Procurve switches are running at 100/100 but ideally we would like to replace them for GB and look at replacing the core switch. Problem is, if the pipe back is generally just a 1Gb fibre, I really can't see there will be a gain and it seems too messy to be running the amount of Ethernet cable you would need to provide a suitable pipe -I would say you need 4GB per 48 10/100/1000 switch.

 

Are schools using OM3 10Gb fibre or have you done the same as us or have people got GB switches with just one 1Gb fibre cable back to the main cabinet?

 

10Gb fibre just looks too expensive to me!

 

Thanks very much in advance.

[RabbieBurns]

90% of our core to edge switches is 10gb fibre. Where we havent been able to stack switches, we LAG 4x1gb links between switches to increase bandwidth.

 

The remaining 10% are 1gb fibre runs, but this is down to the modules at either end, so these have been budgeted for and are being replaced soon.

[mavhc]

The real question is: how much of the capacity you have installed is being used, and what are you using to monitor it?

[pete]

^ Basically that, it's surprising how much a bog-standard 1GB link (or 2GB LAG) can handle for everyday usage.

 

Point Cacti at your switches, monitor for a typical month and see what your network is actually doing (and what the cpu load on your switches are).

 

Without a "we need to upgrade because of $foo" and here's the logs to prove it, I'd deny your request unless I personally knew your request had merit.

 

----

 

We have 2GB links between cabinets currently and we're not seeing bottlenecks there.

Edited June 23, 2011 by pete

[steveg]

We have 3 cabinets, 1 (server room) ,2 and 3. 1 and 2 are linked via fiber running at 10gig, 1 is connected to 3 via 4 cat6a cables trunked to give a 4 gig link. There is then another 4 gig trunk for redundancy running between cabs 2 and 3. All client connections are gig, and sever connections range from 1 gig to 4 gig.

 

Steve

[IanT]

Each switch on the edge has 2Gig per switch

[Marci]

2Gb LAG Fibre to each cabinet from the core, except for our MFL wing which has a 10Gb link.

[localzuk]

Every switch in our school connects back to the core via a 1Gb fibre. So if a cabinet has 3 switches in, then that cabinet has 3Gb connectivity back to the core.

[ChrisH]

Every switch in our school connects back to the core via a 1Gb fibre. So if a cabinet has 3 switches in, then that cabinet has 3Gb connectivity back to the core.

 

I'm the same pretty much.

[gshaw]

1Gb link back per switch here as well on OM2 fiber, 100Mb to the desktop.

 

Most switches are 24-port, got the odd 48-port here and there but tried to avoid them if I could due to the fan noise

[Michael]

It's pretty difficult to fill a 1Gb fibre link and replacing the switches themselves can make all the difference. The switching capacity and onboard memory is very influential to the speeds you can expect.

 

For example, if you compare the switching capacity of a HP 4108GL - 36.6 Gbps and compare this with a HP 2800 - 96 Gbps, even at the same 'connected' speeds, you're going to get much better transfer rates.

 

In cabinets where you have 3 or 4 switches, I'd recommend replacing them with a single 48 port switch if you haven't already. I also find sticking with the same brand - HP or Cisco can also make all the difference and keeps managing the network straight forward.

 

Eventually OM3 will be the standard, but as you say it's still very expensive. You can actually run 10Gb on very short OM2 links if you want to save some cash. Alternatively CAT6 upto 55m will support 10Gb too. Any new fibre links should be OM3, but OM1 still has a lot of life left in it

[CyberNerd]

We have 2x 10GB/s links from out core to our English/IT and Science near edge switches (where most the traffic goes). this has a 1GB/s failover on OM3.

Everything else has 2x1GB/s link on OM2 or copper.

[ronnoco]

Thanks everyone for all of the posts, much appreciated. It's really good to see what people with set-ups similar to us have done. Our monitoring hasn't been to good, we've recently installed Cacti and Nagios and are trying to get it working.

 

We run a creative and media diploma so there is lots of streaming video, large downloads going across the network. Our media suite was very slow because we had a single 1gb OM1 fibre serving 4 HP Procurve switches, mostly 48 port models. Running 3 CAT5E cables to the other 3 switches in the cabinet has helped significally.

 

Something else we're thinking about is getting a whole school fast wireless solution. I have thought about having all AP's wired straight back to the core switch to provide maximum speed....where within the 100M radius. This does seem a 'messy option' though but would surely provide the fastest speed and no work other than purchasing for example 2 X 24 port GB modules for our HP 4108GL and the actual physical cabling would be required. Dong it the traditional way of just having the AP points wired back the nearest cabs wouldn't be quick enough with our current set up.

 

I need to do some thinking, lots of options, upgrade the core switch, get better links to the feeder cabs, AP's back to the core switch, etc.

 

Thanks again...keep the responses coming!

Edited June 23, 2011 by ronnoco

[Michael]

We run a creative and media diploma so there is lots of streaming video, large downloads going across the network. Our media suite was very slow because we had a single 1gb OM1 fibre serving 4 HP Procurve switches, mostly 48 port models. Running 3 CAT5E cables to the other 3 switches in the cabinet has helped significally.

 

Other options to consider are strategically placed NAS boxes, either with SAS or SSD drives (if it exists). I don't think I've seen a NAS with SSD drives, or you could create one?

 

If you're already running 48 port switches, then you could look at trunking or bite the bullet and just get OM3 installed. There's certainly justification if you have plenty of video files moving across the network.

[IanT]

Every HP switch in my place has 2x 1Gb Link\trunk to a switch, everything is running on CAT6 with OM3 Fiber.

[CyberNerd]

Something else we're thinking about is getting a whole school fast wireless solution. I have thought about having all AP's wired straight back to the core switch to provide maximum speed....where within the 100M radius. This does seem a 'messy option' though but would surely provide the fastest speed and no work other than purchasing for example 2 X 24 port GB modules for our HP 4108GL and the actual physical cabling would be required. Dong it the traditional way of just having the AP points wired back the nearest cabs wouldn't be quick enough with our current set up.

 

Think of a wireless AP as a 30 port switch. To function correctly it will need a 1GB back to the (actual) switch in order to serve 30 connections. It's fine running them from near edge switches if you have 10GB/s backbone - otherwise run them all back to the core or you will be asking for trouble if the AP's all get loaded. If you have a large site you may have no option other than 10GB core if you want to avoid problems.

[wesleyw]

Our core is serviced by an Extreme 8810 running to 8 Extreme x450e switches at 10GB then to lower use cabs at 1GB to 9 DLink 1224TPs. I'd agree with CyberNerd the need for a good link back is needed our Aruba AP65s only connect at 100mb back to the closest switch which does have a knock-on effect with performance. We have now got a new sixth form centre build going up over the course of 9 months and this will let us experiment with a new N standard wireless solution however I need OM3+ from the existing site to the new build and for this project I think it needs to run directly from the core instead of the closest edge switch (should prove expensive) looks like 40GB connections are easily available for core switches now 10GB connections everywhere may become a reality in the next 5 years with 100GB becoming the infrastructure standard soon.

 

 

Wes

[snoerre]

We are using a ring of 4 fiber (1Gbit/s) lines connected with LACP. Switch is HP E4800. 10Gbit/s isn´t needed and won´t be within the next 5 years. LACP with 1Gbit/s modules is cheaper...

[CyberNerd]

We are using a ring of 4 fiber (1Gbit/s) lines connected with LACP. Switch is HP E4800. 10Gbit/s isn´t needed and won´t be within the next 5 years. LACP with 1Gbit/s modules is cheaper...

 

Rather depends on how many connections you need. Good luck if your expecting 15 AP's with 30 connections each to run off a gigabit near edge switch with only 1 or 2 GB uplinks

[ITGURU]

10GB from core to edge switches in Star topology - No switches are uplinked or daisy chained. even if a cab has 2 switches they are separately fed.

 

Outside mobiles are still only on 1 gig though as only a few pcs.

 

Best advantage of 10gb is when imaging pcs

Edited February 20, 2017 by ITGURU

[Katy]

Wow that's digging up an old thread, but interesting to see how it's changed in 6 years.

 

We have a single 1Gb link from each edge switch housed in the main server room (as they are in the next rack along from the core switch), 10Gb to each edge cab housed elsewhere, then on the 10Gb enabled cabinets we have 4 x 1Gbps LACP to each edge switch. There wasn't sufficient fibre installed to allow us to have a dedicated feed per edge switch.

[thebatman]

We have a core switch in each main building (3 in total) theys are linked by 10Gb fibre. Servers, storage are at 10Gb to core siwtches. All other switches are linked back to the core switchs via 1Gb fibre or copper.

[mavhc]

10GB from core to edge switches in Star topology - No switches are uplinked or daisy chained. even if a cab has 2 switches they are separately fed.

 

Outside mobiles are still only on 1 gig though as only a few pcs.

 

Best advantage of 10gb is when imaging pcs

 

You don't multicast?

[ITGURU]

You don't multicast?

 

If i'm imaging several computers in the same room it means all of them can image at 1 gb and then down a 10 gb pipe which gives a greater bandwidth than 1gb to each pc and 1gb fibre.

[AlanD]

Not much point in focussing on remote cabling if the server structure itself is not going to deliver anything. A RAID with 10 drives is going to struggle to fill a gigabit link and indeed might only have gigabit links to your main switch...hopefully some link agrregagated ones....

 

And massive disk RAIDs, even those with SSDs are not always much better as they uses SAS expanders which limit overall data rates. Only if you have NVMe drives in the server and the server has 10G link to main switch would I start worrying about what was further away. Yes, aggregating some 1Gb links to main switch would be a useful addition.

 

Running video editing to lots of stations via a network is of course questionable as a concept...and I doubt that is going to work very well without serious investment throughout. Better to have a load of local storage to work with as temporary space.

 

We have old pre om1 fibre which runs just fine at 10gb...albeit over a distance of 100m or less. And we use aggregated links throughout between core switches and perimeter, for redundancy if not bandwidth. Yes we have some mono mode high grade fibre for long runs. We think 1gb to desktop makes a differnce for user experience even though our servers would struggle collectively to deliver this to multiple stations sat a time.

 

...and of course wireless and ever increasing external bandwidth, schools I know of gave 1gb leased lines, need bandwdth in the infrastructure to deliver it to ac access points.