Cloud Hosted VOIP Suggestions on Issues

[BKGarry]

Dear All,

 

I need some collective wisdom here, we have a cloud hosted VOIP System, which is through a subcontractor of our Building PFIs contract/suppliers. 

 

We have been seeing issues where every 3rd Thursday of the month, phones go offline, we end up having to factory reset them to get them working (which is very annoying in itself) but I have an even stranger issue where we cannot get them to factory reset where they are connected, we have to take them back to the core switch and plug them into a port there to get them to factory reset properly.

 

I have checked that all uplinks have the VOIP VLAN available, the core is a HP Enterprise modular switch, all other network equipment is Unifi operating in Layer 2, so the ports are tagged with the VOIP VLAN as the Native/Untagged traffic.

I have tried having it restricted to just that VLAN, no others (which breaks the few PCs that are daisy chained through them but it was testing), I have tried the exact same config as the core switch port (set as the VOIP VLAN as untagged and the wired clients VLAN as tagged)

 

I have made sure that nothing is being restricted on the Firewall.

 

I am at a bit of a loss, the phones are all showing as VLAN0, which isn't our voice VLAN but apparently that's how no VLAN configured on the Yealink phones are displayed.

 

I have tested removing the Option 66 from DHCP on the VLAN so they shouldn't be going off for random config updates etc but today, even with that not there, we have had 5 of our approx 70 phones go offline and had to sort it out (those are the ones that have been reported).

 

Where there are subcontactors on Contractors, we have to prove everything that it is not our network, but this weird behaviour, that seems like it is the VLAN isn't available makes it look like it is. I have made sure there are no ACLs in play, nothing is hitting the smoothwall box, and the phones are not getting any drops for traffic on the firewall, but I am now stuck

 

Does anyone have any suggestions?

[BKGarry]

To Add - It is random which phones it happens to, sometimes it can be most of them, others one or two of them. It is really hurting my brain

[Steve21]

If it's working on the HP one and not the unifi have you checked you don't have things like IGMP snooping/multicast/DHCP spoofing etc options enabled on the VOIP port profiles on the unifi switches? Sometimes they can block the provisioning procedures on the phones

And under the same query have you checked on the PBX as to whether the phones are set to automatically reprovision on that 3rd Thursday etc? Just to narrow down if it's related to that process

Steve

[BKGarry]

Thank you @Steve21

I have Rogue DHCP Server Detection on but none of the other stuff.

 

STP is on as is LLDP-MED but no other spoofing on the ports or switches

 

I am afraid I have the most cut down access to the PBX ever, I cannot even see the MAC Addresses of the devices

[BKGarry]

I have managed to speak to someone at the Phone Subcontractors, and they seem confused, I am sure that it is where they are showing as VLAN0 on the phones rather than 900, but let's see what they come back with ay

[psydii]

set up a port mirror to pipe to Wireshark. Reset a phone and connect it to the port you are mirroring.

look to see what a "good" provisioning and reboot sequence looks like.

 

repeat, but on an edge-of-lan ubiquiti switch.  compare the results. (there is probably a better way than 'by eye' but a phone boot sequence is simple enough that doing it by eye is quite possible)

 

You are looking for queries that go unanswered, or responses from unexpected locations.

 

 

On the Thursday specifically issue: It's a good shout re scheduled re-provisioning. That sounds like a very likely root cause. 

 

On the failures - I wonder if there is some TLS MITM impacting some of the phones - perhaps they are not excluded from filtering rules? or perhaps some phones don't care that they've been MITM'd, or perhaps they aren't even trying to download a refreshed config and are happy loading their cahced one.... or maybe when connected to the core they can retrieve them via http rather that HTTPS?


Final anecdote: Long long ago, I improved our phone reliability by breaking our egress from all devices behind one IP to split across multiple IPs. i.e. each source subnet is now (more or less) NAT'd behind  different public IPs.  This made a huge difference for UDP based protocols. I can only assume that some table was overflowing either server-side or in the IP stack of the router and occasionally things on the phones would randomly not quite work right.

[BKGarry]

Wireshark was next on my list, the annoying thing is you reset an OK phone on a unifi switch it is just fine 🙂

 

The phone company are currently putting the VLAN tags back in the phone config, they don't know why someone there suggested they would take it out.

 

I do not see anything for the phone vlan in the filtering logs, but I will check again tomorrow, but the filtering box does sit between the core switch and the firewall so would have thought it would kick in there.

 

Really I just want to look at some nice logs, but I will see what I can get out of Wireshark.

[BKGarry]

Right, today has been fun,

 

Got the phone company to put the VLAN tags back into the phones, all of a sudden all the phones would not get a DHCP address. 

Now I don't pretend to know the ins and out of Layer 3 networking, but I took some wireshark logs, threw them into copilot and it was the DHCP server is offering but the phone isn't acknowledging them. Turns out you cannot have the native VLAN as the tagged VLAN on the phone, it just doesn't respond. Cue me having to redo all the vlan config on the ports and rebooting the phones. Thank goodness for multiple port editing on Unifi and the ability to easily powercycle everything.

 

Let's see what happens on the 3rd Thursday of November ay