Fibre Problems - Advice needed

[max_carpenter]

Evening All,

 

Pulling my hair out over this one! We have an ISCSI/Storage area network completely separate from our LAN, This consists of 2 physical servers and 2 SANS which are divided amongst two server rooms and the two rooms are connected via a 10Gbps Fibre Link. The switches are HP 2920. This setup has been working fine for 1 - 2 years

 

Ok so there is the background now for what h as happened, As part of a Server room refit to tidy it up I needed to replace the the patch fibre cable from one of the HP switches to the Fibre patch panel with a longer cable to assist a cab being shifted over. Should be a fairly simple job replaced it with an identical cable except it was longer, Well it didn't like that at all and couldn't get the link between the two rooms to come back up at all so in the end had to switch back to the original cable to get all back up and running.

 

Initially based on that you would say faulty cable apart from the cable was already in use for a different link that didn't need such a long cable so I simply swapped them over and putting it back on the old link worked fine.

 

Ok so scrap that cable I decided to buy a brand new fibre patch cable which I bought this one (https://www.ebuyer.com/537288-cables-direct-om3-fibre-optic-cable-lc-lc-aqua-fb3m-lclc-050) and then proceeded to replace the cable again another early morning, This time everything came backup and I was happy, Until over the weekend a handful of servers failed with Hyper-V saying storage unavailable, Managed to bring them back up but since the weekend this has continued to happen sporadically. All the time everything is fine a ping between the rooms works perfectly <1ms and all looks good, however when people start to report things are going slow there are intermittent packet drops but the packets that do succeed are always <1ms and then finally we get 3 to 8+ minutes at a time that all packets drop which is when the servers fail because of no storage.

 

I am preparing to put the old cable back in yet again but I cannot understand how two separate cables one being brand new are faulty yet the latest one is only faulty when it wants to be. Looking at the traffic on that port currently its using <1% utilisation and one switch has thousands of FCS receive errors but all ok on the other switch. Does anyone have any ideas what could be happening here?

 

Any help much appreciated.

 

Thanks,

Max

[FN-GM]

With fibre you at some point there needs to be a cross between RX and TX so the receive port on one side connects to the and send port on the other. Usually this is done within one of the patch cables. It might be worth un-clipping the connectors at one side of the cable and switching them over and see if it helps. Usually you won't get any activity at all without doing this but depending on the switch config you might get something.

 

Also how long is the fibre run? have you checked this is within limits of your SFP? That extra patch cable length might just push it over the limit. I think OM3 is 300m with 10 gigabit.

[LytchettNM]

You should be able to see statistics of the GBIC ports from each switches.

And yes if your running at 10Gb with OM3 it is 300 meters.

[max_carpenter]

Thanks for the replies so far,

 

The crossover is in place, The run paced out is 130 Meters so lets say half again to be WAY into the safe side of going around corners/obstacles that puts it to 195 Meters. Pretty sure we are safe under the 300 meter limit. The new patch I replaced in was an extra 2 meters from the one before.

[Ste_Harve]

Is the new patch the same diamater? Ie 62.5 with 62.5 or is it 62.5 with 50 or vice versa?

[AlanD]

I'd expect it to work with any combination of 62.5 or 50u patch or modules... But not with 9u single mode fibre... And equally... I wouldn't expect LR modules to work well with Mm fibre... But it might work over short distances... Edited June 12, 2018 by AlanD

[pete]

What does it say on the side of the known-good cable?

[mjk]

What does it say on the side of the known-good cable?

 

Importantly, what does it say on the side of the fibre connection between the two rooms?

My suspicion is that it's an ancient 62.5/125 fibre between the two rooms that has been repurposed to use 10gb/s and it's only working by the skin of it's teeth. Mixing core sizes will result in attenuation loss.

It could also be that the main cable is damaged (or is OM2) and adding just a few feet more pushes it over the edge. To know for sure you need to measure the decibel loss between both ends and then measure the loss with the patch cables. There is a table you can use here:

Specifications For Fiber Optic Networks

Edited June 13, 2018 by mjk

[AlanD]

and it's only working by the skin of it's teeth. Mixing core sizes will result in attenuation loss.

It could also be that the main cable is damaged (or is OM2) and adding just a few feet more pushes it over the edge. [/url]

 

There is a surprisingly large "safety" margin when it comes to fibre optics, lengths, losses, etc. And not particularly sensitive to using OM1 cable instead of OM2 either. Although I'm sure a contractor would use it an excuse to rip out everything you have. We use pre OM1 graded cable for 10Gb/s at 300m ...well actually a bit more.. and it doesn't give any errors in the switch logs. The only thing we did do was a replace mechanical fibre optic plus with "welded" pigtails....because terminations are nearly always the worst part of a fibre installation.