Network Speed Between Switches

[DMcCoy]

Multicast imaging @ 1Gb/s? Great, but your software cannot be Ghostcast?

 

32 bit ghost on Windows PE is certainly much faster than 100Mb, same with just WDS on R2. You *do* need reasonable switches, ideally all managed and supporting igmp filtering. In the end I was mostly limited by the 40-50MB/s write speed of the older machines I was imaging.

[DMcCoy]

But that's going to mean it's only going to take a couple of desktops to start a file transfer with a server and the 2 gig links to be 100% utilised -- is that a good idea? Also, will the servers be able to cope with this kind of bandwidth?

 

I'm just a student, I don't have any real-world experience; I could be wrong.

 

There is contention *everywhere* most clients always end up with a proportion of the available bandwidth dependant on the current contention ratio of the link. Two clients on two switches could use all the bandwidth from an iscsi SAN, but we don't restrict uplinks to 100Mb. If you really need some traffic to be prioritised then there is always QoS.

 

In practice 1Gb to desktops is nearly always faster, even on a contended link. 100Mb is pretty slow for a lot of applications now, even 1Gb is starting to limit some clients/applications.

[snoerre]

You *do* need reasonable switches, ideally all managed and supporting igmp filtering. In the end I was mostly limited by the 40-50MB/s write speed of the older machines I was imaging.

 

I actually have some nice switches (H3C S5500/S5800) and capable server hardware, but only have that low throughput. Asking myself whether it has something to do with the NIC driver. That 800 MB/minute is close to 100Mbit/s, although it says it is connected with 1Gbit.

[DMcCoy]

I actually have some nice switches (H3C S5500/S5800) and capable server hardware, but only have that low throughput. Asking myself whether it has something to do with the NIC driver. That 800 MB/minute is close to 100Mbit/s, although it says it is connected with 1Gbit.

 

If you don't have IGMP filtering on the edge switches too, you will often get caught out by printers etc slowing down the multicast. The multicast is only going to go as fast as the slowest device that responds (until it's dropped), often a good way to spot failing drives if a machine drops out of a ghost session repeatedly!

[maestromasada]

Upgrading all cabling to the standard 1Gb is a must, and adding the 10Gb to the server backbone as you are planning is an extra benefit, a big hit on the budget though, but if you can afford go for it.

 

Remember however that at the end it is all down to the disk, and sometimes it may be worth upgrading the server I/O throughput rather than the switches where it connects. There is no point of having a 10GB if the servers cannot deliver from the disk

 

And well, I take for granted that you are using VLANs? I assume you network is segmented, separating desktop traffic from wireless; if you are going to have initially 500 wireless devices you may as well split them in two vlans if possible to optimise traffic. On one of my research I found out that many networks are congested with broadcasts and unfiltered traffic, and network managers tend to buy more powerful switches with bigger bandwidth and faster processors just to cope with the congestion instead of tacking the source of the problem that initially could be a lack of vlans to filter traffic or an inadequate disk base I/O from the servers

 

Just a bit of advise, that’s all

[Mehmet]

There is contention *everywhere* most clients always end up with a proportion of the available bandwidth dependant on the current contention ratio of the link. Two clients on two switches could use all the bandwidth from an iscsi SAN, but we don't restrict uplinks to 100Mb. If you really need some traffic to be prioritised then there is always QoS.

 

But it's much more difficult to 100% utilise a 1 or 2 gigabit connection with 48 hosts @ 100mbps.

 

100Mb is pretty slow for a lot of applications now, even 1Gb is starting to limit some clients/applications.

 

If that is the case then I would have thought that even a 10 gigabit uplink isn't going to cut it.

 

I'm not trying to challenge anyone, I'm just here to learn.

[DMcCoy]

But it's much more difficult to 100% utilise a 1 or 2 gigabit connection with 48 hosts @ 100mbps.

 

 

 

If that is the case then I would have thought that even a 10 gigabit uplink isn't going to cut it.

 

I'm not trying to challenge anyone, I'm just here to learn.

 

Using the full 1Gb for an extended time could be an issue, but there is nothing wrong with high utilisation. It just means you are providing data to the clients as fast as possible, with a reasonably priced uplink. 10Gb if you can afford it sure, but it's not going to be a problem without.

[Roberto]

But that's going to mean it's only going to take a couple of desktops to start a file transfer with a server and the 2 gig links to be 100% utilised -- is that a good idea? Also, will the servers be able to cope with this kind of bandwidth?

 

I'm just a student, I don't have any real-world experience; I could be wrong.

...

But it's much more difficult to 100% utilise a 1 or 2 gigabit connection with 48 hosts @ 100mbps.

 

If that is the case then I would have thought that even a 10 gigabit uplink isn't going to cut it.

 

I'm not trying to challenge anyone, I'm just here to learn.

 

Ah fair enough. Lets back up a bit and discuss something fundamental about capacity on modern network devices.

 

Think of bandwidth/throughput on switches (and capacity on servers for that matter) as a cake you're trying to cut into slices and share between your clients, so the more clients you have the more there are sharing the cake. Your posts suggest your current thinking sees it as more like filling a jug - e.g. if you have a 2gb uplink and 2 1Gb clients the link is full, which is not the case.

 

To stretch these analogies further, buying a 10Gb capable switch as an upgrade to a 1Gb one, or combining several 1Gb conections, means you a larger 'cake' to share, which can work both in the sense of allowing more clients to have a share, and/or allowing those clients to have a bigger share.

 

More importantly, the clients sharing the switch are not all broadcasting at full pelt all the time. The nature of both ethernet connections and TCP/IP tends to lead to 'bursts' of data transmission, which lends itself quite well to sharing.

 

We've got edge switches that comfortably serve 300 devices, for example (HP 5412ZLs) with dual 10Gb uplinks and bandwidth has not been an issue on any of these.

Edited December 21, 2012 by Roberto

[Mehmet]

...

 

 

Ah fair enough. Lets back up a bit and discuss something fundamental about capacity on modern network devices.

 

Think of bandwidth/throughput on switches (and capacity on servers for that matter) as a cake you're trying to cut into slices and share between your clients, so the more clients you have the more there are sharing the cake. Your posts suggest your current thinking sees it as more like filling a jug - e.g. if you have a 2gb uplink and 2 1Gb clients the link is full, which is not the case.

 

To stretch these analogies further, buying a 10Gb capable switch as an upgrade to a 1Gb one, or combining several 1Gb conections, means you a larger 'cake' to share, which can work both in the sense of allowing more clients to have a share, and/or allowing those clients to have a bigger share.

 

More importantly, the clients sharing the switch are not all broadcasting at full pelt all the time. The nature of both ethernet connections and TCP/IP tends to lead to 'bursts' of data transmission, which lends itself quite well to sharing.

 

We've got edge switches that comfortably serve 300 devices, for example (HP 5412ZLs) with dual 10Gb uplinks and bandwidth has not been an issue on any of these.

 

I think perhaps I am not explaining my concern well enough.

 

8 access layer switches connected to a core switch. 1 of these switches has 48 AP's connected to it which services 500 wireless devices and therefore has a 10 gigabit uplink to the core. The remaining 7 have dual 1 gigabit ethernet links uplinks (so that's a 2 gigabit link). The server is connected directly to the core switch via 10 gigabit ethernet.

 

If you have such a set up, it is very easy to 100% utilise the 2 gigabit uplinks if the desktops are also running at gig. I imagine a couple of large file transfers would do it. Now how much utilisation will there be on the 10 gigabit server link? Will the server be able to cope?

 

I understand that this is only a valid concern if users are transferring large files; and if they are not, then no problem! But if they are not, then why spend on switches with 10 gigabit ports? What's the difference in price between a gig switch and a 10 gig switch? I understand that it's good to plan for the future, but if your network traffic increases so much, then I would have though even a 10 gig uplink isn't going to cut it. I don't know, you guys know more than I do.

[Tallwood_6]

You need to look at what the the users on these wireless devices will be doing on the network and i bet 80% of the time they will be using it for internet access. If that is indeed the case 1gb uplinks arnt going to be your bottle neck its going to be your broadband connection/ web filtering etc.

[snoerre]

If you don't have IGMP filtering on the edge switches too, you will often get caught out by printers etc slowing down the multicast. The multicast is only going to go as fast as the slowest device that responds (until it's dropped), often a good way to spot failing drives if a machine drops out of a ghost session repeatedly!

 

I tested in different (computer class) rooms with 16 to 32 computers, no printer, nothing. computers are decent. copying GB files from one to another I get ~115 MB/s. That´s real Gbit/s. Copying that one from one class room to another I get the same performance.

 

What throughput do you get? Which NIC driver to you use? Does anyone use some other multicast imaging tool?

Edited December 22, 2012 by snoerre

[Roberto]

I think perhaps I am not explaining my concern well enough.

 

I understand perfectly well what you're trying to say. Please read my reply again with the assumption that I understand what you're trying to say and that I'm attempting to explain to you why your concerns aren't quite right, rather than with the assumption that I've misunderstood you...

[Mehmet]

I understand perfectly well what you're trying to say. Please read my reply again with the assumption that I understand what you're trying to say and that I'm attempting to explain to you why your concerns aren't quite right, rather than with the assumption that I've misunderstood you...

 

I understand what you are saying. Thank you for sharing your knowledge

[enjay]

Thanks for the replies, everyone.

 

So, generally the consensus seems to be to provision 10Gb if I can afford it, especially when laying new cables, but not to worry too much about it just now if that isn't possible. Yes?

[Ric_]

Thanks for the replies, everyone.

 

So, generally the consensus seems to be to provision 10Gb if I can afford it, especially when laying new cables, but not to worry too much about it just now if that isn't possible. Yes?

 

Yes.

 

Also consider what would happen if that 'core' switch fails. Is it worth buying two switches and running a slightly more redundant setup? Of course this gets more difficult if you have physical servers as you won't be able to span your aggregated links across switches.

[localzuk]

Yes.

 

Also consider what would happen if that 'core' switch fails. Is it worth buying two switches and running a slightly more redundant setup? Of course this gets more difficult if you have physical servers as you won't be able to span your aggregated links across switches.

 

Redundant setups are great, if you can afford it, but to do it properly, you don't just need a second switch, you need double the GBICs too, double the fibre cores etc...

 

To do that here, we would have to find a further £7500 for GBICs and patch leads alone...

[enjay]

I have a spare switch, a spare pair of GBICs and a spare fibre patch lead (plus box-loads of Cat5e of course). This provides redundancy for most eventualities, and next-day delivery provides for the rest.

 

The only eventuality to catch me out so far is when a power cut killed two switches simultaneously, however using an old print server which had a 4-port switch built in, I was able to provide service to critical users whilst getting the replacement switches ordered.

[Ric_]

Redundant setups are great, if you can afford it, but to do it properly, you don't just need a second switch, you need double the GBICs too, double the fibre cores etc...

 

To do that here, we would have to find a further £7500 for GBICs and patch leads alone...

 

Sounds cheap to me

 

I was only throwing it in there as a consideration that often gets overlooked. Many people cannot afford it and 24-48 hours downtime is acceptable.

[CyberNerd]

Redundant setups are great, if you can afford it, but to do it properly, you don't just need a second switch, you need double the GBICs too, double the fibre cores etc...

 

To do that here, we would have to find a further £7500 for GBICs and patch leads alone...

 

You probably spend more than that on SIMS alone!

Definitely worth doing.

[localzuk]

You probably spend more than that on SIMS alone!

Definitely worth doing.

 

Oh, the extra switch would work out as another £12.5k. So, £20k - my GBICs bit was indicative of the extra cost here.

 

As I said, its a great idea, but expensive to do.

[enjay]

Oh, the extra switch would work out as another £12.5k. So, £20k - my GBICs bit was indicative of the extra cost here.

 

As I said, its a great idea, but expensive to do.

 

Your backbone switch was £12.5k?!

 

Remember that kit doesn't need to be even close to full spec - it is just a stop-gap while you wait for the next-day delivery of the replacement. The redundancy kit I have here is stuff which I've upgraded out, but hung on to - the switch and GBICs are from when I upgraded a building from 100Mb to 1Gb, the spare PCs are from the last batch which we were binning, same goes for the projector. They're not great, but they do the job.

 

I turn them on once a term to check they still work and to allow software updates to go on.

[localzuk]

Your backbone switch was £12.5k?!

 

Remember that kit doesn't need to be even close to full spec - it is just a stop-gap while you wait for the next-day delivery of the replacement. The redundancy kit I have here is stuff which I've upgraded out, but hung on to - the switch and GBICs are from when I upgraded a building from 100Mb to 1Gb, the spare PCs are from the last batch which we were binning, same goes for the projector. They're not great, but they do the job.

 

I turn them on once a term to check they still work and to allow software updates to go on.

 

For the sheer volume of things we have plugged into it, yes. A core switch to cover all our edge switches would cost us that. We have 15 cabinets that would need connecting up around the site. 15 x 10GbE ports isn't cheap.