Strange issue we can't get to the bottom of

[mrbios]

I'm after some ideas on this one, ISP can't find a solution, we can't find a solution. A horrible work around is in place and I'm potentially about to spend some money to "resolve" the issue.

 

Basically the setup is this:

Internal network > Smoothwall > Router > Internet

PBX > Router > Internet

100mb leased line

 

External calls don't go out via the smoothwall, the PBX has a direct connection to the router, its own external IP and it's own firewall (FreePBX one) basically to avoid passing SIP traffic through NAT.

ISP has enabled QoS on the line.

 

When bandwidth usage is high on the internal network calls degrade to the point of being completely useless.

I've enabled bandwidth management on the smoothwall to limit the amount of bandwidth internal traffic can use, i started this at 90, then 80 and now 70. Even at 70 some audio gaps are noticed. If i disconnect the smoothwall from the router, calls are crystal clear. After school or during lunchtime when usage is low, calls are crystal clear.

We've also checked MTU settings incase of any mismatch there.

 

Essentially we're losing out on 30mb of our leased line thanks to the issue that appears to be between the smoothwall and the router. My gut instinct is that it's the routers fault, but the ISP disagree.

 

My last straw solution is to get a GEA line installed, run the phones over that primarily and only have them go over the leased line as a failover. Which is a nice solution in itself as it's good to have a backup to something as essential as the phones, however in an ideal world the high SLA leased line would do both and a nice cheap adsl line would be the failover. However if my primary line for SIP is going to be on what should be the backup line, i'd rather a decent SLA and no contention ratios...hence a GEA line.

 

Thoughts? Any suggestions on where the issue could be? I'm all out of ideas.

[Steve21]

When you're talking router is this core switch e.g. VLAN routing, or internet router side? (Or both?) Or when you say high internal bandwidth is that internet traffic just not phone?

 

If you're passing all internal traffic via the router are you hitting it's routing/traffic cap? Even if it's not going to the internet

 

Steve

[mrbios]

When you're talking router is this core switch e.g. VLAN routing, or internet router side? (Or both?) Or when you say high internal bandwidth is that internet traffic just not phone?

 

If you're passing all internal traffic via the router are you hitting it's routing/traffic cap? Even if it's not going to the internet

 

Steve

 

Sorry when i say router I'm talking internet router, Mikrotik Routerboard to be exact. The core switch in this scenario starts at internal network, and when i say high internal bandwidth i mean large amounts of internet traffic....i didn't clarify that very well.

[Steve21]

Thought it would be but just wanted to check Then surely an easy test as to the issue would be when you disconnect the smoothwall, plug a laptop in and run a huge download/upload etc and see if it still affects it.

 

If "lets say" 100mb of smoothwall traffic breaks it, but 100mb of laptop traffic doesn't that should answer what's at fault. Personally I'd expect both to break it still.

 

But end of day "what" is plugged in shouldn't affect anything if the QoS is working correctly. It just sounds as if even if QoS is enabled it's not understanding the traffic coming into it etc

 

Steve

[mavhc]

"ISP has enabled QoS on the line." -- what does this actually mean though?

 

You'll need to somehow tell the system to prioritize VoIP traffic, tag its packets differently

[mrbios]

"ISP has enabled QoS on the line." -- what does this actually mean though?

 

You'll need to somehow tell the system to prioritize VoIP traffic, tag its packets differently

 

They've enabled QoS between themselves and the ISP supplied router to prioritise SIP/VoIP packets...is my understanding there anyway. I've got no access to the router to clarify further. I've got QoS enabled internally on my VoIP vlan as well, not that it's really needed, and i've confirmed that was working when i looked deeper into it with wireshark.

[TechMonkey]

Disruptive but I would say @Steve21's idea is the most conclusive. If everything is fine with a single device sucking down a large amount of bandwidth then Smoothwall is doing something silly. If the same thing happens with only a laptop connected it is the router/PBX/line.

 

One thing to possibly try is to disable G729 codec on your sip trunk. We have had an issue where intermittently our calls go a bit stuttery and looking around general advice seems to be that some SIP trunks will default to the G729 codec even if it is last on the list. Only just done this on our system so have no evidence but using an 8Kbps codec or a 64Kbps codec would make a hell of a difference.

[mrbios]

Disruptive but I would say @Steve21's idea is the most conclusive. If everything is fine with a single device sucking down a large amount of bandwidth then Smoothwall is doing something silly. If the same thing happens with only a laptop connected it is the router/PBX/line.

 

One thing to possibly try is to disable G729 codec on your sip trunk. We have had an issue where intermittently our calls go a bit stuttery and looking around general advice seems to be that some SIP trunks will default to the G729 codec even if it is last on the list. Only just done this on our system so have no evidence but using an 8Kbps codec or a 64Kbps codec would make a hell of a difference.

 

Thought it would be but just wanted to check Then surely an easy test as to the issue would be when you disconnect the smoothwall, plug a laptop in and run a huge download/upload etc and see if it still affects it.

 

If "lets say" 100mb of smoothwall traffic breaks it, but 100mb of laptop traffic doesn't that should answer what's at fault. Personally I'd expect both to break it still.

 

But end of day "what" is plugged in shouldn't affect anything if the QoS is working correctly. It just sounds as if even if QoS is enabled it's not understanding the traffic coming into it etc

 

Steve

 

Sorry Steve missed that reply when i posted previously. I did do this test back in Easter when it originally became an issue, i can't for the life of me remember what the outcome was. I'll need to redo that test soon and find out, might try it after schools ended today.

 

It looks like alaw, ulaw, g722 and gsm are the only codecs enabled, and in that order on both the SIP trunk itself and in the Asterisk SIP settings of our PBX. All phones are set to alaw by default too. EDIT: Those codecs and that specific order are what our ISP and SIP Trunk provider specifically told us to set too.

Edited September 27, 2017 by mrbios

[TechMonkey]

Maybe speak to the SIP trunk provider and see if there is a log of what codecs where used for each call?

[SchoolsBroadband]

@mrbios are we still your broadband provider?

 

It could be that by putting QOS on the Mikrotik and when you see high traffic and session count that the memory or CPU is getting maxed out on the Mikrotik router. This can cause some packet loss which could then be why you're seeing poor call quality.

 

So get the ISP (or us) to logon to the Mikrotik when you're seeing these problems and see what the CPU and memory usage is. If its 100% then time for a more powerful Mikrotik which will fix the issue.

 

Send me a PM if we are your broadband provider still and I'll help look into it personally for you.

 

Thanks

 

Dave

[mrbios]

@mrbios are we still your broadband provider?

 

It could be that by putting QOS on the Mikrotik and when you see high traffic and session count that the memory or CPU is getting maxed out on the Mikrotik router. This can cause some packet loss which could then be why you're seeing poor call quality.

 

So get the ISP (or us) to logon to the Mikrotik when you're seeing these problems and see what the CPU and memory usage is. If its 100% then time for a more powerful Mikrotik which will fix the issue.

 

Send me a PM if we are your broadband provider still and I'll help look into it personally for you.

 

Thanks

 

Dave

 

We moved ISP in August last year. CPU and memory usage was checked when i was originally looking into the issue and they were fine, it's the RB3011 model if that means much to you?

 

Strange thing is the issue didn't occur between August and March, then one day everyone started complaining of dreadful phone call quality to external calls, and the dodgy voice was only one way. Recipient could hear us fine, but us hearing them was horrendous. I'm not sure what could have changed one day for it to suddenly become bad, but i was calling the ISP asking them what they had changed etc thinking it must have been them, but they hadn't changed anything. Then when i checked if QoS was working, they said it wasn't actually enabled because they only enable it upon request (even though it was a part of the order) so it was working fine even without QoS for a time. We then went through a lot of phone calls and a lot of testing. We changed the auto negotiation of the ports to fixed to see if that helped, set the MTU manually on both my smoothwall and the mikrotik, still the same problems persisted up until i limited the amount of bandwidth we could use via the smoothwall.

[SchoolsBroadband]

Get them to check the CPU isn't maxing out. The RB2011's sometimes struggle under high load when using SIP firewall rules. The RB3011 is a beefed up version of the RB2011.

 

Alternatively come back to us and we'll have it fixed in a jiffy

 

Dave

[mavhc]

Wonder if your VoIP provider has something similar to https://www.ringcentral.co.uk/support/qos.html so you can see data rather than relying on user reports

[mrbios]

Thought it would be but just wanted to check Then surely an easy test as to the issue would be when you disconnect the smoothwall, plug a laptop in and run a huge download/upload etc and see if it still affects it.

 

If "lets say" 100mb of smoothwall traffic breaks it, but 100mb of laptop traffic doesn't that should answer what's at fault. Personally I'd expect both to break it still.

 

But end of day "what" is plugged in shouldn't affect anything if the QoS is working correctly. It just sounds as if even if QoS is enabled it's not understanding the traffic coming into it etc

 

Steve

 

Unplugged the smoothwall, connected a laptop, ran two different speed tests (individually and together) while i had an open call. Not a problem...however the line did only max out at 77down 73up which seemed a bit odd, should make it to 90 odd really on at least one of the speedtests when there's nothing but a pbx + laptop on the line.

 

Going to relay that to my ISP now and see what they say, though I'm pretty sure i did this once in the past.

 

EDIT have asked the ISP to provide me a faster router as a test, if they provide a faster, more powerful router and the issue still remains then I think i can rule the ISPs part in it out and it'll be time to get smoothwall on the case.

 

Interesting memory just came back actually from when i was testing before...It seemed previously as though more devices = more problems. I think running speedtest from one device on my network, such as my PC, when no one else was here didn't cause the problem, but once 6 IT suites were in use hammering the internet, that's when the issue occurred at its worst.

Edited September 27, 2017 by mrbios