PEO Posted June 23, 2009 Posted June 23, 2009 Hi gang Ok so we recently updated our servers, all is good .... now that I've ironed all the daft buggs out out of them and all is settled. Our biggest bottleneck is the switches around the school. most of the switches around the school are 3com managed switches 100/1000 and 4 are dell managed 100/1000. all the cabinets apart from the maths, server room, and admin office which is connected to out sims server are on the 1gb fibre backbone. some of the cabbinets are crowded with cables due to expansion and and worse the server room is connected by 1gb copper link to a switch before it even touches the main back bone switch. Obvioulsly we need to upgrade some links especially the the server room connection. My line manager has asked me to do a report on each cabinet and create a shopping list of what is needed to improve the network performance. I am happy with 3com switches they have server us really well the past 4 year but its time to defo upgrade. how would be the best way to produce an accurate report and what should I take in to consideration? apprecate any feedback or questions.
SYNACK Posted June 23, 2009 Posted June 23, 2009 The way I did this was with end to end fractions. Pretty much I looked at the existing structure and represented it on a map of cabinets, servers and representitive workstations. I made the cable width between them on the map (more like an organisational diagram) represent the bandwidth to 1gb was 10 times the width of 100mbit. This provided a graphical way for them to understand it as it looked kind of like road infrastructure. The next thing that I did was describe the journy of data from one place to another and show the fraction of bandwidth avalible. In your case I assume that the new servers have dual GB links so start at 2GB if you have two servers then these have to share the 1GB link back to the core they are instantly running at 500mbits (1/4 max), this gets onto the core and is then shared between all of the clients on a switch so if you have 24 clients on a switch then you get about 21mbits per host if only that lot of hosts is accessing the network. That is theoretical max so practically you get about 60%, so 12.6mbits. Divide this by 8 to get megabytes per second equals 1.6 megabytes a second so a 16mb profile will take a minumum of ten seconds as it only has access to 1/95th of the avalible bandwidth or 1/20th bursting. This assumes full usage all at one time which may never occour but depending on the amount of hosts and cabinets this could be a real scenario for the school if 1/10th of your stations are hitting the servers at any one time. You would then present the proposed case to them with a new diagram and new calculations showing the improvement in speed that could be attained. I have used this method on several occasions and it has always got the message accross quite well. Usually when presenting I give the handout to them first (a day or so before) and then have a scheduled Q&A session about it later incase they want any clarification. Hope it helps. 1
thedarxide Posted June 24, 2009 Posted June 24, 2009 Consider switches that will let you trunk ports. You may also see this listed as aggregation or bonding. That way, you can dedicate bandwidth to your uplinks. 1
PEO Posted June 24, 2009 Author Posted June 24, 2009 Did a survey today and I've just finished drawing the diagram of the current network. Some cabinets need there link back to the cabinet upgrading. I also noticed we have a spare fibre cable going to every cabinet that already has a fibre connection. Does this mean I could look in to link aggregation to each cabinet. 2gb backbone link to each cabinet and each cabinet has stacked switches? about 90 Desktops have 1 gb links. Im going to draw up another diagram of the improvments that need to be made but first I would like some feed back and advice on what should be done to improve.
thedarxide Posted June 24, 2009 Posted June 24, 2009 Yes, that's right. However, I'd look at getting a bigger core switch and ensure that all connections come back to this point, so that you have a very large star. At the moment, there is a lot of switch traversal going on. For instance, there is a lot of hopping going on to get from the admin office to the server room. 1
PEO Posted June 24, 2009 Author Posted June 24, 2009 Yes, that's right. However, I'd look at getting a bigger core switch and ensure that all connections come back to this point, so that you have a very large star. At the moment, there is a lot of switch traversal going on. For instance, there is a lot of hopping going on to get from the admin office to the server room. Yeh I noticed this straight away as I took notes... the data load on the science and humanities switch must slow as hell. Is the a way to calculate the speed of data flow between a maths machine and server room?
SYNACK Posted June 25, 2009 Posted June 25, 2009 Yikes, first up yes you can use the extra fibre with an extra fibre converter on each end as a secondary link. 90 stations on a single 1GB link eeek. If I was you I would be looking at 10GB fibre core links or massve trunking, we use a 2GB trunked fibre core at my primary schools for around 200 stations. The big question at this point is how is it routed, given the ip addresses for each cabinet there must be a router kicking around somewhere (or layer 3 switch) and this will be the heart of the network and the bit under the most load. Personally I would be looking for a device with layer 3 switching which just means that it can handle routing at about 50x the speed of a plain router. The next thing that I would look at is path minimization, try to get the path to any end point to involve the least number of switch jumps that it can, prefferably just the one from the core to the edge or two if you have distribution layer switches. It would be hard to get figures on how fast stuff can go without the number of hosts in each area and their effective link speed (100 or 1000 mbits). It is worth checking for writing on the fibres to see what they are, on the side of the cable there should be some writing indication the type, OM3, OM1 etc. If it is OM3 then the existing fibre will handle 10GB links if you can get the gear. By the looks of it there is another switch inbetween the core and the servers and so if practical the servers should be connected individually directly to the core or be attached to a switch that has a link speed as fast as all of the server NICs combined back to the core. If the switches have a managment page you should eb able to go into each one and grab a list of the active ports and what speed they are at so that you can add a host count to each switch at each speed for more calculation. When I had to do this at one network it had a 10mbit fibre backbone (a long time ago) and so the maths turned out quite nicely to prove my point. One of the areas only got about 3x the speed of a 56k modem at the time (worst case) which scared a few people. 1
PEO Posted June 25, 2009 Author Posted June 25, 2009 The Fibre cables appear to be OM1... this is the following writing I found on the cable OM1LTO4UBK 4x62.5u looks like we cant do 10gb backbone
Butuz Posted June 25, 2009 Posted June 25, 2009 Do you have 4 core or 8 core fibre runs? IF you buy new switches, you can use trunking/port aggregation to enable 2GB links over 4 core or 4GB links over 8 core. Its not 10 gig but it's considerably better than 1 gig and it's faily cheap to implement unlike 10 gig. Butuz 1
SYNACK Posted June 25, 2009 Posted June 25, 2009 No not with the existing fibre, you would need to add new runs to each cabinet. If you did end up looking at thae I would suggest pushing as many cores as are practical into the new fibre, this gives you redundancy and expantion ability for comparitivly little extra cost, an 8 core is not all that much more expencive than a four core (two pair) if you don't terminate all of the ends to start with. Still teaming over the existing fibres along with upping the bandwidth to the core from the servers will deffinatly help. How many servers do you have currently, or a better question what is the added up network speed of all of the server NICs? Edit: Looks like your current cable is two pair meaning that all of the runs have been terminated so you are limited to two links for the moment on each run. 1
localzuk Posted June 25, 2009 Posted June 25, 2009 You have, in places, 4 hops between servers and workstations. Personally, I'd be looking to turn it into a true star-topology. Replace the multi-hops with a single run for each switch back to the core switch. Ensure a 10GB link between the core and the server switch also. Then, each switch could happily operate on a 1GB link, and move to a 2GB link if needed. 1
PEO Posted June 25, 2009 Author Posted June 25, 2009 No not with the existing fibre, you would need to add new runs to each cabinet. If you did end up looking at thae I would suggest pushing as many cores as are practical into the new fibre, this gives you redundancy and expantion ability for comparitivly little extra cost, an 8 core is not all that much more expencive than a four core (two pair) if you don't terminate all of the ends to start with. Still teaming over the existing fibres along with upping the bandwidth to the core from the servers will deffinatly help. How many servers do you have currently, or a better question what is the added up network speed of all of the server NICs? Edit: Looks like your current cable is two pair meaning that all of the runs have been terminated so you are limited to two links for the moment on each run. 5 windows servers eache with the following 2 x iscsi 2 x ethernet teamed 1 x for powering on machines and adjusting bios 1 mac xserver 1 iscsi 1 ethernet cachepilot Hitachi SAN 100 2 x iscsi 2 x ethernet
PEO Posted June 25, 2009 Author Posted June 25, 2009 This is something close to what I think should look like on paper
PEO Posted June 25, 2009 Author Posted June 25, 2009 hmm would this be ok. eache server ethernet connection to be directly linked to the core switch. each pair teamed for a 2gb link. And leave the switch in the server rack for the iscsi
localzuk Posted June 25, 2009 Posted June 25, 2009 This is something close to what I think should look like on paper I'd go one stage further and remove those last chained switches too - have all of them link directly in. 1
SYNACK Posted June 25, 2009 Posted June 25, 2009 hmm would this be ok. eache server ethernet connection to be directly linked to the core switch. each pair teamed for a 2gb link. And leave the switch in the server rack for the iscsi Sounds good, you want your iSCSI traffic isolated and as close as it can be to the servers so having it on its own dedicated switch in the same cab would be perfect. providing direct teamed links for each of the servers directly to the core will also hugely optimize the traffic flow and performance. Also the closest that you can get to a star the better for pathing as this will reduce the hop count and the amount of through traffic making the actual usage and distribution much more efficient within your existing infrastructure. 1
PEO Posted June 25, 2009 Author Posted June 25, 2009 does anyone have a rough idea how much it would cost for OM3 cable per meter?
SYNACK Posted June 25, 2009 Posted June 25, 2009 (edited) does anyone have a rough idea how much it would cost for OM3 cable per meter? Pre Terminated Fibre Optic Cable, Multi & Single Mode Cables Multi Mode 50/125 4 Core Tight Buffered, Internal/External, LSZH, per metre £0.73 base price & £0.73 for each subsequent metre. Multi Mode 50/125 8 Core Tight Buffered, Internal/External, LSZH, per metre £1.24 base price & £1.24 for each subsequent metre. in Pre terminated stuff that you can pull through yourself looks like about £150 to terminate 8 core. Edited June 25, 2009 by SYNACK
teejay Posted June 25, 2009 Posted June 25, 2009 On 10Gb it's not the fibre cable that costs much but the network switches and fibre modules. To be honest with the size of network indicated by the number of switches on the diagram you probably won't need 10Gb. Link agregate your servers directly into the core switch and replace your copper links with fibre and use link aggregation to either 2Gb or 4Gb on the fibres with more than one switch attached. If you are pulling any new fibre, 10Gb fibre is hardly any dearer so pull that through for future upgrades and get at least 8 core so you can do as much link agg as possible.
PEO Posted June 25, 2009 Author Posted June 25, 2009 Pre Terminated Fibre Optic Cable, Multi & Single Mode Cables Multi Mode 50/125 4 Core Tight Buffered, Internal/External, LSZH, per metre £0.73 base price & £0.73 for each subsequent metre. Multi Mode 50/125 8 Core Tight Buffered, Internal/External, LSZH, per metre £1.24 base price & £1.24 for each subsequent metre. in Pre terminated stuff that you can pull through yourself looks like about £150 to terminate 8 core. Ok I've adjusted the plan, and proposal... what do you think. diagram attached
SYNACK Posted June 25, 2009 Posted June 25, 2009 Looking good, much less traversal of switching and much better core access for the servers. This setup would be a definite improvement over the existing one.Depending on how smart your switches are I would look at implementing STP on them and adding additional copper links between the swiches in the same cabinet making sure that the core is the master switch. This way it will give you redundancy by chopping over to the other switch in the cabinet if a single fiber goes out. This link would not be active at any other time other than if the main links failed. This gives you continuity in the network with only a small window of downtime.Ideally you want to be planning for additional links/faster links in future but the system above would be a vast improvement. One thing that I would say is to never give the impression that this is the last penultimate upgrade. There will always be more stuff that will come along and be required so be sure to convey to the powers that control the money flow that this is just one very positive step along the journy of good IT. 1
localzuk Posted June 25, 2009 Posted June 25, 2009 ooops did again forgot to add diagram Looks better. But i'd make one last change - the 3 switches daisy chained in the computer room, they are physically close to the core switch? If so, plug them directly in using copper. 1
SYNACK Posted June 25, 2009 Posted June 25, 2009 Looks better. But i'd make one last change - the 3 switches daisy chained in the computer room, they are physically close to the core switch? If so, plug them directly in using copper. I'd dissagree with that change, I thought the same at first but if you have a look at the key the links are stack links. These run at the very least at 10GB and so are much better suited to the duty than a trunked solution in this situation. 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now