Odd internet issues

[synaesthesia]

Hi all,

 

Hope @Wave9 don't mind me posting this, but thought I'd reach out and see if anyone else has come across this issue.

 

We have a decent leased line by Wave9, carried by TalkTalk Business. We have fairly major connectivity issues to various sites, and Edugeek is particularly badly hit so we're using it as a testing example

 

The fault is difficult to describe but in a nutshell, sites often either take ages to load or don't load after a while giving us a 504 error. A refresh/reload however instantly brings it up. This is across the board.

 

Setup is simple enough; firewall/filtering via SophosXG, connected via fiber to the NTE.

We have ruled the Sophos appliance out as an issue; we've bypassed it entirely connecting directly to the NTE and the problem still exists.

We've also connected a 4G phone to the Sophos appliance, failed over to it (which by the way is seamless and awesome) and the problem goes away.

 

We can run a ping all day to an affected host such as Edugeek and it'll rarely drop a beat; no more than you'd expect from daily internet usage, with spikes during mass log off/log on times. Pings don't appear to be affected. It only appears to be HTTP traffic, or maybe even TCP.

 

Wave9 have been scratching heads over this, they've changed a few things trying to get things to behave like TCP header sizes, MTU etc I believe with no luck. There was a small possibility of a hardware/connection issue, so all fiber cables, SFPs and pigtails have been replaced with no difference. A late suggestion yesterday however was to try a VPN; this evening sure enough I installed PrivateInternetAccess on my workstation and lo and behold, the problem disappeared. Pings of 8ms solid, download speeds in excess of 190Mbit/s, and absolutely no pausing/hanging/failing to load on any sites.

 

Slightly long winded but this is something that's been causing quite a few problems for us. We absolutely cannot fault Wave9 who have gone over and above so far in trying to resolve this with us; we'll solidly recommend them to ANYONE needing connectivity. It's a matter now of trying to get TalkTalk to listen and act, if they need to!

[SchoolsBroadband]

Hi @synaesthesia,

 

sometimes DNS can act a bit weirdly, particularly if you've got cached sites so you might want to flush your dns or try using another one such as googles 8.8.8.8. Bar that MTU is a good one to try but that would be a more constant issue.

 

Talk Talk when you take a layer 3 connection from them (not layer 2 like we took from them when we serviced you) does use all sorts of caching wizardry going on in their network.

 

if you VPN your connection and thus DNS requests and HTTP(s) traffic so it doesn't touch Talk Talks caches then this could take it out of the equation hence why you start to see improved speeds and no web issues.

 

Wave 9 will need to speak to the right people at Talk Talk to get this fixed as from what you describe i'd lean towards that as the issue, particularly if the same thing happens if you just plug a laptop into the leased line with no Sophos box in the middle.

 

I'm not meaning to sound bad on Wave 9 (as this is the same for all companies with their model) but that's a big difference between layer 3 (re-sellers) and layer 2 (wholesale providers with their own ISP network) as the later has full control of the network and can do what ever they want with the traffic, where as a layer 3 provider has to simply pass on issues to their supplier as they can't fix any network / ISP related issues themselves.

 

Still here to help with any other suggestions if I can. Good luck!

 

Dave

[SchoolsBroadband]

One other thought,

 

have you used wireshark and deep inspected whats happening with packet captures?

 

Dave

[synaesthesia]

Yup, DNS has been ruled out entirely, and wireshark captures show no pattern in the delays other than the content delivery systems for edugeek showing up more often than others but that would appear to be coincidental (but caching could indeed have a lot to do with that). Certainly worth a look cheers. Fairly sure Virgin were our carrier with you guys though?

[SLS_]

I've seen similar behaviour caused by SSL decryption settings on webfilters (went away with settings turned fully off), but as you've physically bypassed your Sophos box, I guess this won't be the case. :-s

[mikes]

yeah we had the same with our FORTINET, it was caused by either SSL or some other scanning "feature", yahoo and google mail were affected.

happened during our inspection; the inspectors couldn't access their https webmail and the LEA wouldn't disable the "feature".

[synaesthesia]

Yup but as said, this is line based, all filtering etc entirely ruled out.

[SchoolsBroadband]

Is it a talk talk managed router or a wave 9 managed router that terminates the leased line as talk talk can offer managed or wires only?

[synaesthesia]

Wires only.

 

Problem solved! TalkTalk spent the majority of today on site and upon hearing my explanation the engineer already had an idea what it might be, albeit unlikely.

MTU size. Whilst set to 1500 by default, dropping this to 1458 instantly resolved the issue. Whether this is a proper fix or a bodge remains to be seen as it would be odd for a leased line to require a lower MTU. Regardless, a fix is a fix and if the bodge comes back to bite anyone, we, Wave9 and TalkTalk have a huge wodge of data pointing the blame squarely in TT's direction.

 

So thought it'd be worth an update in case anyone else has similar issues. And of course a huge hats off to Wave9 who have been exemplary in all manners, service, monitoring & communication; we can't recommend them enough.