Network Speed Between Switches

[enjay]

We're in the process of speccing out the addition of a wireless network, which obviously involves a review/overhaul of our existing wired network.

 

Once the wireless is fully operational, we will have some 150 wired devices and 500 wireless devices (assuming we don't permit pupils to connect their phones as well as their laptop/tablet); the busiest edge switch could plausibly have moments where it is supporting 200 devices.

 

Since we're buying new switches anyway to provide POE to the APs, what do people think about taking the opportunity to upgrade the links between core and edge switches to 10Gb? Advice welcome...

[Domino]

Yes. if you can afford it, do it. Having a solid backbone just makes everything easier.

[CyberNerd]

I would (and did) put in a 10GBE backbone.

If you have 500 wireless devices then you need to think of each switch as a 30port access point so 10GBE to the near edge is worth doing.

[Mehmet]

 

Since we're buying new switches anyway to provide POE to the APs, what do people think about taking the opportunity to upgrade the links between core and edge switches to 10Gb? Advice welcome...

 

It's difficult to say without seeing your current network, and usage. Does your current core switches have 10 gigabit ports?

 

If you need more bandwidth, you also have the option of aggregating links.

[enjay]

Does your current core switches have 10 gigabit ports?

No, but as I said, I'm buying new switches so that doesn't matter.

 

If you need more bandwidth, you also have the option of aggregating links.

That is true, although that would only give me max 2Gb inter-switch, and we could max that on some of our busier switches. That said, whatever happens the server will only be 10Gb, so I'm not sure how much benefit we'd get from pulling all our edge switches up to 10Gb as well.

[localzuk]

No, but as I said, I'm buying new switches so that doesn't matter.

 

 

That is true, although that would only give me max 2Gb inter-switch, and we could max that on some of our busier switches. That said, whatever happens the server will only be 10Gb, so I'm not sure how much benefit we'd get from pulling all our edge switches up to 10Gb as well.

 

You can aggregate 10Gb links too, including on the server. We've got the capacity here for our servers to be bumped up to 20Gb if needs be, by adding an extra cable to each and aggregating them.

 

10GbE for a switch would could end up handling 200 users? I'd be pretty keen on that.

 

The question is, what are the wireless devices going to be doing? Are they iPads? So mostly internet? Or are they going to be normal 'Windows' laptops connecting to central infrastructure? If the latter, and you'll basically be having 650 devices connecting to servers at the same time, I'd be looking to make sure the servers could easily end up at 20GbE on short notice - if it all goes belly up because of load. However, I feel the issue won't be network speed at the server side - it'll be IOPs on your storage.

[Michael]

I'd certainly recommend installing switches with the option to install 10Gb fibre modules. It doesn't necessarily mean you need to use it now. 10Gb fibre modules with a direct cable (within a few metres) is reasonably priced, but connecting switches at 10Gb at a distance is incredibly pricey - both the modules and cabling.

[enjay]

You can aggregate 10Gb links too, including on the server.

Good point...

 

The question is, what are the wireless devices going to be doing?

Still to be decided, although terminal services / VDI is a strong contender so traffic will be more than just Internet.

 

If the latter, and you'll basically be having 650 devices connecting to servers at the same time

Not quite that many, as my numbers above include some multi-device users, but we could get 400+ users on the server simultaneously with one particular edge switch potentially carrying nearly 200 of them (depending on which classrooms are in use at the time).

[localzuk]

Still to be decided, although terminal services / VDI is a strong contender so traffic will be more than just Internet.

 

RDS/VDI are relatively low bandwidth really. Between 50kbps and 2Mbps depending on what is happening (the latter being the user playing a HD video).

 

Not quite that many, as my numbers above include some multi-device users, but we could get 400+ users on the server simultaneously with one particular edge switch potentially carrying nearly 200 of them (depending on which classrooms are in use at the time).

 

It'd be storage I'd be most worried about still in that circumstance. Its nothing *huge* in reality but still nothing to sniff at when ensuring your storage is powerful enough to respond in an efficient way. I'd be thinking of putting profiles on SSD personally as that'll be the most noticeable 'slow' bit.

[Mehmet]

That is true, although that would only give me max 2Gb inter-switch, and we could max that on some of our busier switches. That said, whatever happens the server will only be 10Gb, so I'm not sure how much benefit we'd get from pulling all our edge switches up to 10Gb as well.

 

Why would that give you max 2 gigabits? How many gigabit ports do you have on the switch?

[enjay]

Why would that give you max 2 gigabits? How many gigabit ports do you have on the switch?

 

The switches have a varying number of ports, however in each instance there are only 2 links between cabinets, either 2 Ethernet cables or a 4-core fibre depending on switch location.

 

Is it possible to aggregate more than two ports together? If so, it might be cheaper to run more 1Gb fibre and aggregate more ports than to move to 10Gb fibre...

[ict_support]

Netgear XSM and GS ranges will support 10Gb over om1 fibre up to about 270m and i have seen them running above 300m and the price point is not to bad. The down side is they are not HP / Cisco so they can be "well a bit netgear" however they do provide good value for money. Depends on budget, if you have the money go HP if not and you still want 10Gb links the netgears will do it.

[Mehmet]

The switches have a varying number of ports, however in each instance there are only 2 links between cabinets, either 2 Ethernet cables or a 4-core fibre depending on switch location.

 

Is it possible to aggregate more than two ports together? If so, it might be cheaper to run more 1Gb fibre and aggregate more ports than to move to 10Gb fibre...

 

As far as I know you can aggregate up to 8 ports... but please do check.

 

I'm interested in seeing your topology though, without seeing it it's hard to say whether you would benefit from moving to 10 gigabit ethernet.

 

Also you mention one of your edge devices may need to support 200 devices at one time; can I ask how? Is this an edge switch with a bunch of AP's connected to it?

[enjay]

I'm interested in seeing your topology though, without seeing it it's hard to say whether you would benefit from moving to 10 gigabit ethernet.

Topology goes something like this - main cabinet in server room, containing core switch and switches feeding two IT Suites; 3 edge cabinets on 2x Ethernet, 3 edge cabinets on 4-core fibre; each edge cabinet contains either 24- or 48-port fully Gigabit switch, and (when we buy them) either PoE switch or PoE injectors depending on quantity; cabinets typically support 20-30 wired devices, all Gigabit, and APs to support 100-200 wireless devices.

 

I intend to take the core switch to 10GbE to support next server which will be 10GbE, existing servers will be aggregated 2x 1Gb; what I can't decide is what speed I should use to link to the edges.

 

QUOTE=Mehmet;910788]Also you mention one of your edge devices may need to support 200 devices at one time; can I ask how? Is this an edge switch with a bunch of AP's connected to it?

Exactly that.

[Mehmet]

A diagram would be really helpful. Sounds like you have a core, with 8 switches attached to it (a physical star topology).

 

If you have 200 wireless devices traffic going over a single edge switch, then I guess it wouldn't be a bad idea to go with the 10 gigabit ethernet. Any idea how much it's going to cost, though?

 

As for the other edge switches (the ones connected to wired devices) from what you have said it doesn't sound necessary. I'm guessing your wired devices are 100BaseT, so a 2 gigabit aggregate from the edge to the core would be fine.

 

Where are the servers; directly connected to the core?

[enjay]

I don't have a diagram, but yes, it is a basic star: PC -> edge switch -> core, there's only one location which goes PC -> edge switch 1 -> edge switch 2 -> core, but that isn't a busy area, only around 50 devices at full load across both switches. The servers are connected directly to the core switch.

 

All desktops are 1Gb not 100Mb - do you still reckon that 2Gb aggregate would be enough? I know it would be sufficient now, but I need to plan ahead as well, as I don't want to have to buy more kit in a few years.

 

Not sure on cost, but that isn't my main concern - getting the right infrastructure is more important. You can do it cheap or you can do it right, and all that.

[Mehmet]

I don't have a diagram, but yes, it is a basic star: PC -> edge switch -> core, there's only one location which goes PC -> edge switch 1 -> edge switch 2 -> core, but that isn't a busy area, only around 50 devices at full load across both switches. The servers are connected directly to the core switch.

 

All desktops are 1Gb not 100Mb - do you still reckon that 2Gb aggregate would be enough? I know it would be sufficient now, but I need to plan ahead as well, as I don't want to have to buy more kit in a few years.

 

Not sure on cost, but that isn't my main concern - getting the right infrastructure is more important. You can do it cheap or you can do it right, and all that.

 

I wouldn't run gigabit to the desktops; unless you have a specific reason for doing so? If your network has the kind of traffic to justify gigabit to the desktop, then not even a 10 gigabit link is going to cut it. But you know your network better than anyone else.

[DMcCoy]

I wouldn't *not* run Gb to the desktops, there really is no point not doing so with the cost of it being so low. While 1Gb or 2Gb uplinks *can* be a bottleneck, it's still usually going to be faster for all clients, multicast imaging at 1Gb is always useful too.

[Mehmet]

I wouldn't *not* run Gb to the desktops, there really is no point not doing so with the cost of it being so low. While 1Gb or 2Gb uplinks *can* be a bottleneck, it's still usually going to be faster for all clients, multicast imaging at 1Gb is always useful too.

 

But that's going to mean it's only going to take a couple of desktops to start a file transfer with a server and the 2 gig links to be 100% utilised -- is that a good idea? Also, will the servers be able to cope with this kind of bandwidth?

 

I'm just a student, I don't have any real-world experience; I could be wrong.

[Chris_Cook]

But that's going to mean it's only going to take a couple of desktops to start a file transfer with a server and the 2 gig links to be 100% utilised -- is that a good idea? Also, will the servers be able to cope with this kind of bandwidth?

 

I'm just a student, I don't have any real-world experience; I could be wrong.

 

The limit on the servers will probably be the disks. We don't saturate our gigabit uplinks even where we have gigabit to the desktop. Having said that, the reduced latency is helpful.

 

It sounds like your assuming two users (or likely more) are accessing large files at the same time. Often this is not the case. When there is only one user, they get the whole bandwidth. The maths is more complicated than simple addition.

 

The important thing with optimising the overall network performance is measurement. Find out whats causing the slowdown, or whats unreliable and fix that. Its too easy to guess and get it wrong.

 

If I were putting in new cabling, i would try and make sure it is either gigabit or 10 gigabit capable in the case of uplinks. Wether you run it at full speed is a different question. 10 Gig interfaces are still very expensive, and we certainly would't see a justifiable benefit at the moment. Cabling can be in place for a long time, and can be a bigger job to replace. Some of our fibre is 10 years old and working fine still. Replacing a switch or network interface is much easier than cabling.

[Mehmet]

If I were putting in new cabling, i would try and make sure it is either gigabit or 10 gigabit capable in the case of uplinks. Wether you run it at full speed is a different question. 10 Gig interfaces are still very expensive, and we certainly would't see a justifiable benefit at the moment. Cabling can be in place for a long time, and can be a bigger job to replace. Some of our fibre is 10 years old and working fine still. Replacing a switch or network interface is much easier than cabling.

 

Of course. I wasn't concerned about running gigabit cables to the desktop, but if the switch ports and NIC are enabled for gigabit then that is potentially going to be a problem (I think). But obviously it all depends on typical network usage, etc... but as you said in most environments users aren't usually going to start a massive file transfer at the same time. But I still wouldn't give desktops gigabit; especially if it isn't necessary.

[glennda]

If you are planning for the Future I would go 10Gb - or at least have switches that can easily be upgraded to 10gb. If running fiber - make sure it can run at 10gb etc.

 

the physical cables will probably last longer than the switches so its often key to get those right first time round.

[m25man]

I've spent the last few days upgrading a load of switchgear, from 10/100 to Gigabit.

Only after switching over we then found several systems demonstrating problems, all were traced to defective Cat5e cabling installed well over 5 Years previously.

We found crossed cables & incomplete punch downs on all of the defective outlets.

 

By all means put 10Gbe in as we did, but don't over look the rats nest you already have who knows what lies beneath ready to upset your newly deployed 10Gbe backbone.

Take the opportunity to test/certify the existing stuff too.

[CyberNerd]

The limit on the servers will probably be the disks.

 

Disk I/O should be significantly faster than your network @1gb. A lot of it will be cached anyway so I doubt this will be too much of a problem.

[snoerre]

multicast imaging at 1Gb is always useful too.

 

Multicast imaging @ 1Gb/s? Great, but your software cannot be Ghostcast?