10GB networks backbones needed?

[IanT]

Dont quite understand why your surprised!

[east]

Things have changed a lot in a few years though Ian, a few years ago I would have suggested dual 1GbE to be fine myself. However looking forward, with the popularity of mobile devices and increased bandwidth requirements, I don't think that multiple gigabit links will cut it. If the budget is available to upgrade to 10GbE then I would say do it while you can as it's not something that you would regret.

[CyberNerd]

Dont quite understand why your surprised!

 

because that amount of devices on one file server with 2 NIC's equates to about 1.5Kb/s.

The only way I could imagine that you could go it is with multiple clusters and FC SAN.

[IanT]

Things have changed a lot in a few years though Ian, a few years ago I would have suggested dual 1GbE to be fine myself. However looking forward, with the popularity of mobile devices and increased bandwidth requirements, I don't think that multiple gigabit links will cut it. If the budget is available to upgrade to 10GbE then I would say do it while you can as it's not something that you would regret.

 

Yes, I'm aware of that "east".

[psydii]

Some scenarios where 10GB/s will help:

 

1000 users all logging on for the first time to that computer with roaming profiles from 6 load-balanced RP servers with the RPs host as VMs with the VHDs on a 10Gb iscsi SAN SSDs.. then 10GB from your VM hosts to your core, and from your core to the next tier of switches will give you better through put than if you only had 1Gb links.

 

Software installation and large image deployments/updates. If you are transferring a large number of large files between clients and servers then 10GB/s links will make a difference. However you need to look at your server links to your SAN and you need to carefully load balance the work across your highly tuned file servers and physical disks to make any use of it.

 

10GB/s will also help with traditional backup and restore speeds. If you are still running traditional backups.

 

If you are primarily going BYOD then your load will be distributed across all your ports into your core and from there the bottleneck will be to the internet, unless you are lucky enough to have a multi-Gb/s pipe, in whihch case the chances are your bottleneck will be your proxy and firewall throughput.

 

When I had silly money available and worked somewhere that regularly had 30/60 students shifting multi-GB files about the LAN on an hourly basis, 10GB from core to the top of rack switches made sense.

 

In a school where we don't do that then the only 10GB link is between the two core switches. The servers have 4x 1Gb to the SAN and Core. This is an ORDER of MAGNITUDE cheaper than 10GB to all cabinets, and if anything performs better.

 

HTH

[PotNoodleTech]

Can't say I agree. Infact I totally disagree!

 

I can easily saturate a 1GB link just using my (SSD equipped) workstation let alone the other 400 workstations that are plugged in. My server infrastructure is capable of giving so much more than 1GB can manage. Currently running on 4 x 1G trunks but that is not enough. Will be moving to 10GB backbones ASAP (probably 2013 looking at budget).

 

10GBE is getting cheaper all the time. If your looking at revamping your infrastructure now I think it would be misguided to make do with 1GB even when trunked. Don't forget trunking is not the be all and end all. Trunking does not scale linearly. A 4x1GB trunk will not automatically give you a "4GB" link. In some situations trunking is no faster than a single link.

 

10GBE is definatly the way to go especially if you can do it over copper to keep costs down.

 

My 2p.

[Mehmet]

Can't say I agree. Infact I totally disagree!

 

I can easily saturate a 1GB link just using my (SSD equipped) workstation let alone the other 400 workstations that are plugged in. My server infrastructure is capable of giving so much more than 1GB can manage. Currently running on 4 x 1G trunks but that is not enough. Will be moving to 10GB backbones ASAP (probably 2013 looking at budget).

 

10GBE is getting cheaper all the time. If your looking at revamping your infrastructure now I think it would be misguided to make do with 1GB even when trunked. Don't forget trunking is not the be all and end all. Trunking does not scale linearly. A 4x1GB trunk will not automatically give you a "4GB" link. In some situations trunking is no faster than a single link.

 

10GBE is definatly the way to go especially if you can do it over copper to keep costs down.

 

My 2p.

 

If it was the norm for a single workstation to utilise a gigabit connection then it would make little difference whether you have a 10 gigabit connection or a 100 gigabit connection as it simply would not be enough... so I don't believe that what you have said is entirely accurate. Also, are workstations connected using gigabit links? No, more like 100mbps connections.

[Mehmet]

10GBE is getting cheaper all the time. If your looking at revamping your infrastructure now I think it would be misguided to make do with 1GB even when trunked. Don't forget trunking is not the be all and end all. Trunking does not scale linearly. A 4x1GB trunk will not automatically give you a "4GB" link. In some situations trunking is no faster than a single link.

 

In what case will an aggregation of 4 links be no faster than a single link?

[PotNoodleTech]

In what case will an aggregation of 4 links be no faster than a single link?

 

An example would be when using a Layer 3 switch to connect two servers together with each server having a 4 x 1GB link. If you start a data transfer benchmark between the two servers, only one of the 1GB links on each server will be used for the transfer as the first port to send or recieve data is the one selected for use from then on.

 

Between two devices LACP only selects one port to use so that frames are not transmitted totally out of order (i.e if it used all 4 ports at the same time some frames on one link, some frames on another link, all would end up geting confused by the switch as packet loss). So effectively both your servers only end up with 1 x 1GB link being used in this situation. Have you ever tried this yourself? I have and it is correct. Some advanced layer 4 switches avoid this but most switches most schools use don't (my HP Procurces suffer this).

 

Trunking only really works when many devices are communicating with many other devices through the trunk. Each device selects a port to send frames through. The more devices there are the more evenly balanced the trunk will be.

 

So - trunks to connect two servers together? (e.g a domain controller and a terminal server?) No advantage to port trunking. Trunks to connect two switches together each with 100 devices? Definate advantage.

 

Edit - also on all my trunks I have never seen a totally even split on the trunk. One or two ports always have more than their fair share of data. Again this can be corrected by using full on enterprise switches - but then using 5 £figure switches to get high end trunking performance is not in the spirit of this thread as if you can afford switrches like that you can easily afford 10GBe!!!

 

Fact - 1 x 10GBE is always better than 10 x 1GBE trunk

 

That is all I was realy getting at in the frist place - if your buying new infrastructure throught why compromise? You'll only end up paying for it in 3 years time.

Edited April 4, 2012 by PotNoodleTech

[glennda]

In what case will an aggregation of 4 links be no faster than a single link?

 

if you have 4 1gb links bonded you will get a max of 1gb per connection but can have 4 different 1gb connections

 

each cards max is still 1gb

[Rozzer]

Well to make my situation clear.

 

I have a 7 year old core switch which you can no longer buy modules, unless of ebay. It makes sense to buy a core switch which is capable of 10GB. I have set a three year plan to do all buildings but the first year will be focused on our new building. This has the highest number of mobile/static workstations. Each year will be a lot cheaper as it will be down to access switch upgrades and transceiver. Obviously the first year is going to be a lot of money but after the 3rd year the infrastructure should be secure for another 7 years. I do worry about our internet connection if we decided to make more use of cloud based technology.

[Mehmet]

An example would be when using a Layer 3 switch to connect two servers together with each server having a 4 x 1GB link. If you start a data transfer benchmark between the two servers, only one of the 1GB links on each server will be used for the transfer as the first port to send or recieve data is the one selected for use from then on.

 

Between two devices LACP only selects one port to use so that frames are not transmitted totally out of order (i.e if it used all 4 ports at the same time some frames on one link, some frames on another link, all would end up geting confused by the switch as packet loss). So effectively both your servers only end up with 1 x 1GB link being used in this situation. Have you ever tried this yourself? I have and it is correct. Some advanced layer 4 switches avoid this but most switches most schools use don't (my HP Procurces suffer this).

 

Trunking only really works when many devices are communicating with many other devices through the trunk. Each device selects a port to send frames through. The more devices there are the more evenly balanced the trunk will be.

 

So - trunks to connect two servers together? (e.g a domain controller and a terminal server?) No advantage to port trunking. Trunks to connect two switches together each with 100 devices? Definate advantage.

 

Edit - also on all my trunks I have never seen a totally even split on the trunk. One or two ports always have more than their fair share of data. Again this can be corrected by using full on enterprise switches - but then using 5 £figure switches to get high end trunking performance is not in the spirit of this thread as if you can afford switrches like that you can easily afford 10GBe!!!

 

Fact - 1 x 10GBE is always better than 10 x 1GBE trunk

 

That is all I was realy getting at in the frist place - if your buying new infrastructure throught why compromise? You'll only end up paying for it in 3 years time.

 

That wouldn't be the case if you had multiple data transfers from different clients though, which is is more realistic... I can't see why you would need a 10 gigabit connection between two servers, but I could be wrong. I can see why you'd go for 10 gigabit with the future in mind, but in all honesty I don't think many networks need that sort of bandwidth, and 10 gigabit switches are just so much more expensive I'd want to be damn sure it's going to make a difference to network performance.

[CyberNerd]

I can't see why you would need a 10 gigabit connection between two servers, but I could be wrong.

 

It's actually really useful if you were using a non cluster capable file system, it means you can do things like mount your VLE into user shared areas/homedrives etc. This would mean you would (for example) be able to stream video over the VLE from files located in user homedrives (or shared areas). Also pretty handy for backups.

Also consider if your running multiple servers on one physical host, although most setups have 10GbE virtual switches you still suffer from multiple clients accessing one physical server.

[Mehmet]

It's actually really useful if you were using a non cluster capable file system, it means you can do things like mount your VLE into user shared areas/homedrives etc. This would mean you would (for example) be able to stream video over the VLE from files located in user homedrives (or shared areas). Also pretty handy for backups.

Also consider if your running multiple servers on one physical host, although most setups have 10GbE virtual switches you still suffer from multiple clients accessing one physical server.

 

But wouldn't link aggregation do the job in that case?

[CyberNerd]

But wouldn't link aggregation do the job in that case?

 

more the merrier as far as I'm concerned.

[Mehmet]

more the merrier as far as I'm concerned.

 

:D

[CyberNerd]

Well to make my situation clear.

 

I have a 7 year old core switch which you can no longer buy modules, unless of ebay. It makes sense to buy a core switch which is capable of 10GB. I have set a three year plan to do all buildings but the first year will be focused on our new building. This has the highest number of mobile/static workstations. Each year will be a lot cheaper as it will be down to access switch upgrades and transceiver. Obviously the first year is going to be a lot of money but after the 3rd year the infrastructure should be secure for another 7 years. I do worry about our internet connection if we decided to make more use of cloud based technology.

 

I am assessing my current network and trying to future proof the network by looking at 10GB backbone. Only reason I am looking into it because we are looking at one to one devices to every child so potentially we would have over 1500 devices on the network. Majority of it will be on the wireless.

 

I'm in a near identical situation. Went with 4x A5800's as core switches, and some switches for our bladecentre. The A5800's have 4x 10GbE SFP+ ports each as standard and you can add another 4 per switch when you need. You can then move your old 1Gb's switches out to the near edge, or ideally replace them with PoE for the wireless AP's (4800 series would be good for this, as they also take 10Gb/s modules for uplinks) We also upgraded our internet connection and added an ADSL failover.

[glennda]

Personally I went for the Chassis Option when upgrading the core in the same position as yourself - we have an HP 5406Zl which currently only has GB fiber modules. But the idea is eventually add 10GB modules into the chassis and have 10GB links to a second server room (at present our backup room) but with a new block possibly happening at some point (in the future) it will either be in there or the backup room.

 

We have purchased the second chassis switch and will be setting that up in the second location partnered with the other so they appear as a single switch. Each room has virtual hosts/san and each edge switch has a link to either room so that one room can go down and it fails over to the second etc

 

the HA/failover part is scheduled for next financial year or when a new building is constructed - which ever makes sense

 

just a shame i will miss it all!

[psydii]

IMO POE E4800's for WAPs is overkill. V1910's are plenty good enough. Use LACP to give 2GB/s+ thoughput back to your core, but since wireless performance drops off dramaticzlly as you increase the number of clients per AP (specifically, per shared collision domain) you will very rarely see an AP utilising it more than 30% of its WIFI bandwidth. Scaling out wifi perfomance is your biggest problem - getting a wifi infrastructure to satuarate 1GB/s is quite a challange in itself, and you have to consider to what will these devices be talking to the most?

 

It is your wired desktops doing video or photo editing work that could benefit from benefit from 10Gb'/s interswitch/server links. But you need the server infrastructre to deliver that sort of performance.

 

Don't forget that you can always install a 10GB/s capable switch at the top of your distribution racks as and when you need them.

 

In school use, generally, you've got hundred of clients reading/writing non sequential data to your SAN. Unless you've got SSD's for your 'hotest' data, the performance issue is not likely to be your LAN but the spinning platters and how fast the read/write heads can get to the requested data. If you have got SSD's in a san then you've likely got enough cash to put in 10GB links without needing to fret whether you **need** them.

 

Windows Fileserver performance paper is worth a read: Performance Tuning Guidelines for Windows Server 2008 R2

 

What you really need to ask yourself is how is your lan used and how do you see it evolving over the next 1, 3, 5 years? How much would it cost to put in 10GB/s now, when would it start to make a difference to the end users? Is the cost of the kit decreasing faster than the value of money?

 

As for the question of 10GB between servers... well if your dataset is large, having a pipe between your backup system and your production services can be essential. When I last ran a traditional server/backup system we couldn't take a full backup during term time because the 1GB links (even when aggregated) were too slow to backup the entire system in less than three days.

 

If the driver is 1:1 wireless devices I would, in order of priority: look to invest in a wireless network that can support it (aruba has/had a good reputation in this regard), ensuring there was at least a 1gb/s link from the WAP connected switches back to the core. Then look to Fibre Channel/10GBE for the SAN, and then moving to 10GB/s from the VM hosts to the core, next beefing up the WAN link, and finally the core to distribution links.

Edited April 4, 2012 by psydii

[CyberNerd]

IMO POE E4800's for WAPs is overkill. V1910's are plenty good enough. Use LACP to give 2GB/s+ thoughput back to your core, but since wireless performance drops off dramaticzlly as you increase the number of clients per AP (specifically, per shared collision domain) you will very rarely see an AP utilising it more than 30% of its WIFI bandwidth. Scaling out wifi perfomance is your biggest problem - getting a wifi infrastructure to satuarate 1GB/s is quite a challange in itself, and you have to consider to what will these devices be talking to the most?

 

It really depends on the size of the network - I've got 10 or 12 AP's running off one 4800, with one AP per classroom that's 300 machines. 300 machines at 30% bandwidth on 802.11n saturate a 2Gb/s core. Scaling the wifi isn't my issue - I can easily add more AP's to a classroom.

[Rozzer]

Well our fibre is all OM3 so the cost is all in the hardware. I am also looking at a modular core swell for growth.

[SYNACK]

IMO POE E4800's for WAPs is overkill. V1910's are plenty good enough.

 

V series is fine apart from the warrenty and upgrade path, you kind of have nowhere to go other than a whole new switch. Saying that we ended up the the e2510 series ones in one of my schools without facility for 10GB just because of the cost. In another they are a lot more longsighted so we used the more expencive e2910 switches that could take 10GBe modules in the future.

 

Its not so much putting 10GBe in but preping for it if you end up needing it is still a good plan and for the time being the 2910s are way better switches than the 2510s so the benifit is immediate anyway. We still need to pull the old fibre and put in OM3 before we get to upgrading the core and going 10GBe so it is still a way down the road.

[psydii]

Any V1910's with an HP part number sticker on the back have lifetime NBD replacement warranty (like procurve) and are so cheap that having a couple of spares on the shelf is quite viable even in cash constrained environments. (can you tell I'm a fan?)

 

As time goes on the kit gets cheaper, but the labour gets more expensive. So, yes definately make sure any links that go in now are 10GBe rated, but only pay for 10GBe switches when you are pushing the limits of what can be done with LACP.

 

Unless you have the cash now, and know you wont in a few years. Then it is a tougher descision.

[camel]

Last summer I started a 3 year plan to upgrade our network. Starting at the core, we needed layer 3 switching at core and while at it to increase the number of fibre links between our buildings from a single 1Gb to 2 or 4 1Gb bonds.

 

I looked at several options inc HP, Cisco, Netgear. Decided on the a few Netgear GSM7352S's and a GSM7228PS for POE/POE+. They have 2 10Gb SFP+ ports per unit and we have them stacked on 48Gb redundant ring. Cost for features was deciding factor, plus Netgear lifetime warranty.

 

What intrigued me was the SFP+ LRM modules for the netgear switches (all suppliers have these). These modules allowed me to reuse the existing OM1 fibres at 10Gb speeds. Distances upto 220m though but with the size of our site that was more than possible. This was a logical step forward and allowed us to incorporate 10Gb backbone for future proofing, and though it is overkill in normal day to day use (atm), the replication speeds and reduced backup times moving data between backup and live make logical sense.

 

linked up to SAN with dual 10Gb SFP+ links to core.

 

paid about £1600 for the 48 port + 2 sfp+ GSM7352S units £280 for each of the LRM modules.

[snoerre]

(4800 series would be good for this, as they also take 10Gb/s modules for uplinks) .

Running also a bunch of 4800. LACP with 2x 1Gbe is running fine, 4x or 8x would be simply too much for us. 10Gbe slot modules are a little expensive. Next switch generation for new building will be 5500 HI series, though.

 

(can you tell I'm a fan?)

If you are a true fan, you know the truth about the V1910, like me ^^

 

300 machines at 30% bandwidth on 802.11n saturate a 2Gb/s core.

 

Well, but in reality, there´s no continuous download at that speed. Surfing youtube, Skype, chatting, ... Nothing really exhausting.

Edited April 5, 2012 by snoerre