Potential VLAN Issue

[J1mb08agg1n5]

Just inherited a problem at a site that has a main building complex then a small centre about a 1KM away that's physically linked by a dedicated leased line. The overall issue is that the centre has no network connectivity, whereas the main complex is fine. The entire network is logically segmented into VLANs . Everything in the centre is configured to feed off the main complex (i.e DHCP, DNS, WiFi, Internet etc.) where all of the core devices reside. Core devices (servers, switches, router etc.) sit within VLAN1 on a 10.12.12.0/24 network. Backbone consists a mixture of Cisco SMB, TP-Link and Netgear switches all with static IP's. RSTP has been enabled on all switches. A L3 Cisco SG300 is set as the gateway and carries out the inter-VLAN routing Gateway IP exisits for each VLAN and DHCP helper has been enabled and set to the main DC. The main DC (DC01) is situated within another cabinet in the main complex away from where the link between both buildings comes in, so at present, there are several truncated switches in between to the main DC. This handles all VLAN addressing. They do have a failover DC (DC02) that is situated at the centre. Both have static IPs.

Now, I can ping DC01 to DC02 and vice versa and I can also remote into DC01 and RDP over to DC02, so I know for sure that the physical link is intact. However, I can't ping or access the centre switch IPs from DC01, however, I can ping and access any switch for the whole site from DC02. I don't see how I can ping and remote from DC to DC but can't ping the centre switches from the other side as I thought that by default, all untagged packets would be associated on VLAN1, where all the core devices sit plus traffic must go through switches on either side or order for me to connect from DC to DC. This might need some clarification on how this works.

Today was just a fact finding mission so I should have the tools and access to troubleshoot going forward. What I have checked so far is making sure all switches and truncated ports are set correctly, set to trunk then VLAN1 as untagged with all other VLANs as tagged. What I would like to test is where traffic is potentially being dropped and/or if VLAN ID's are traversing across the leased line correctly. Traceroute doesn't work for anything internal as it either drops straight away or shows me 1 hop which i would expect. Be interesting to know if anyone has had experience with Wireshark to set this up and point out what I would be looking for. Also below is a basic visual representation of the ping tests I have carried out:

DC01 <----> DC02 - OK

DC01 <----> Main Centre Switch - Unreachable

DC02 <---> Any Switch - OK

Like I said, main complex is fine but for some reason, no devices connected to the switches in the centre can obtain a DHCP address from it's respective VLAN, even when connecting to a port I've set to VLAN1, but I can communicate from DC to DC.

Just wanted to brainstorm potential causes and ideas on where and what to troubleshoot to see if I can find anything glaring or missing. Any tips and/or explanations would be greatly appreciated.

[FN-GM]

Drawings and diagrams might be useful here. This is quite complex.

 

Can you confirm that the main centre switch, DC01 & 02 are all in the same VLAN?

 

How is the link between the sites configured? Is this an access port of 802.1q (trunk / tagged)?

[PyROm]

It kinda sounds like the router or firewall has been setup to allow traffic through from DC02 but not from the rest of the centers subnet.

[PyROm]

Also check that the gateway specified on the equipment at the center matches that of the settings on dc02.

[J1mb08agg1n5]

It kinda sounds like the router or firewall has been setup to allow traffic through from DC02 but not from the rest of the centers subnet.

 

Yeah it's strange. It was apparently all working 3 weeks ago and the only thing that changed is VOIP installers came in 2 weeks ago, so that's why I think it maybe potential VLAN tag related as they might have been messed up some of the tagging.

 

Also checked variables on DHCP on both DC01 and DC02 and can confirm they are correct or as they should.

[PyROm]

Did the voip installers use a different subnet? maybe the voip installers created a new subnet that conflicts with a subnet back on your main network so data is going to the voip network instead of your main site? Could the voip equipment have been given the same ip`s as other equipment on the network, eg the switch/gateway etc?

[J1mb08agg1n5]

Did the voip installers use a different subnet? maybe the voip installers created a new subnet that conflicts with a subnet back on your main network so data is going to the voip network instead of your main site? Could the voip equipment have been given the same ip`s as other equipment on the network, eg the switch/gateway etc?

 

The VOIP install happened on both sites plus a Voice VLAN was already in place, so all they needed to change was switch port VLAN assignment. I'll have to get their contact details to double check but as far as I am aware and can see, the supplied phones were set to acquire addressing via DHCP. The only anomaly that I can see is they run out of POE switch ports in the cab where the leased line comes in, so they supplied the site with a temporary switch. It is a managed Netgear but I have no clue of how it was setup and unfortunately don't have the credentials to check. As far as I am told, the link port on this switch was only set to access the voice VLAN and is linked into the exact switch that the multimode fiber is patched in for the leased line to the small centre. There are a bunch of POE phones plugged into it at the moment but they are all working. I am tempted to remove the link to this switch to see what happens but would have to do it out of hours.

[J1mb08agg1n5]

Drawings and diagrams might be useful here. This is quite complex.

 

Can you confirm that the main centre switch, DC01 & 02 are all in the same VLAN?

 

How is the link between the sites configured? Is this an access port of 802.1q (trunk / tagged)?

 

I will whip up a diagram when I get the chance. Just walked into this issue this morning with scarce documentation apart from a list of switch credentials and truncated ports so I have the outlay of the network in list form.

 

DC01, DC02 and the main centre switch all have static IP's within VLAN1 IP scope and connected switch ports are set to access with VLAN1 as untagged.

 

The link is a single strand fibre that connects into a ADVA FSP150CP optical fibre access switch on either end which are then connected by multimode fiber into a SFP switch port. These switch ports are set to trunk with VLAN1 untagged and all other VLANs tagged.

Below is the diagram for it:

 

https://ibb.co/mDkXGMn

Edited November 9, 2020 by J1mb08agg1n5

[J1mb08agg1n5]

Drawings and diagrams might be useful here. This is quite complex.

 

Can you confirm that the main centre switch, DC01 & 02 are all in the same VLAN?

 

How is the link between the sites configured? Is this an access port of 802.1q (trunk / tagged)?

 

Also here is a very quick and basic network topology, if you need me to go in more depth then I can, it's just what I could muster up in the time. Like I said, all devices have static IP's within VLAN1 IP scope. I am positive I have checked all truncated ports to ensure they are trunks with VLAN1 untagged and all other VLANs tagged.

 

[J1mb08agg1n5]

Took another visual look at the truncation today and the activity on the switch port LEDs did seem very erratic to the point that they are constant. Was mainly looking at Cisco switches so don't know if that's how they usually behave but did seem a little full on. Could I be looking at a potential network loop somewhere or do you think I am clutching at straws?

 

Any other suggestions or troubleshooting roads I could go down as I am starting to run out of ideas.

[FN-GM]

DC01, DC02 and the main centre switch all have static IP's within VLAN1 IP scope and connected switch ports are set to access with VLAN1 as untagged.

 

Are they on the same subnet?

 

It is really hard to diagnose without more information such as a complete network map, IP schemes etc.

[J1mb08agg1n5]

Are they on the same subnet?

 

It is really hard to diagnose without more information such as a complete network map, IP schemes etc.

 

Yes the are on the same subnet.

 

Attached is a spreadsheet I have managed to put together that lists all the network details I have been able to gather, so not exactly a map but I have listed all IP schemes, IPs, trunk ports etc. that hopefully might better explain what I am dealing with.

 

Today, for a brief moment, the switch in the small centre (.118 in the attached sheet) was able to communicate with everything over in the main complex but then stopped and ping tests went back to "request timed out". Don't know what happened there. I also re-configured STP on both switches and it elected the correct root bridge according to all the other switches in the main complex, so communication must be there as it managed to learn the correct one and select the trunk port as the root port.

 

Bit of a head scratcher so far but I do really appreciate all the responses so far, regardless of how small.

Network Details.xlsx

[PyROm]

Sorry was wrong about the /32 subnet below hadnt twigged it was your link.

 

I think you need change your subnet mask on vlan 2 /32 only allows one ip, and you are using 10.11.0.1 and 10.11.0.2

Edited November 11, 2020 by PyROm

[PyROm]

Thinking about it, you have used the same 10.12.12.0 at both sides of the link, you cant do this with your setup as you are routing the traffic. Traffic matching a machines local subnet wont be sent to a gateway (normally) and even when it does the router wont know which end of the link to send it to. You need to change the subnets at one site so they dont match the other end. This will mean a second dhcp scope etc.

[robertcladner55]

Just inherited a problem at a site that has a main building complex then a small centre about a 1KM away that's physically linked by a dedicated leased line. The overall issue is that the centre has no network connectivity, whereas the main complex is fine. The entire network is logically segmented into VLANs . Everything in the centre is configured to feed off the main complex (i.e DHCP, DNS, WiFi, Internet etc.) where all of the core devices reside. Core devices (servers, switches, router etc.) sit within VLAN1 on a 10.12.12.0/24 network. Backbone consists a mixture of Cisco SMB, TP-Link and Netgear switches all with static IP's. RSTP has been enabled on all switches. A L3 Cisco SG300 is set as the gateway and carries out the inter-VLAN routing Gateway IP exisits for each VLAN and DHCP helper has been enabled and set to the main DC. The main DC (DC01) is situated within another cabinet in the main complex away from where the link between both buildings comes in, so at present, there are several truncated switches in between to the main DC. This handles all VLAN addressing. They do have a failover DC (DC02) that is situated at the centre. Both have static IPs.

Now, I can ping DC01 to DC02 and vice versa and I can also remote into DC01 and RDP over to DC02, so I know for sure that the physical link is intact. However, I can't ping or access the centre switch IPs from DC01, however, I can ping and access any switch for the whole site from DC02. I don't see how I can ping and remote from DC to DC but can't ping the centre switches from the other side as I thought that by default, all untagged packets would be associated on VLAN1, where all the core devices sit plus traffic must go through switches on either side or order for me to connect from DC to DC. This might need some clarification on how this works. For more information about g9 led bulbs please visit the website here https://householdprof.com/best-g9-led-bulbs-review/

Today was just a fact finding mission so I should have the tools and access to troubleshoot going forward. What I have checked so far is making sure all switches and truncated ports are set correctly, set to trunk then VLAN1 as untagged with all other VLANs as tagged. What I would like to test is where traffic is potentially being dropped and/or if VLAN ID's are traversing across the leased line correctly. Traceroute doesn't work for anything internal as it either drops straight away or shows me 1 hop which i would expect. Be interesting to know if anyone has had experience with Wireshark to set this up and point out what I would be looking for. Also below is a basic visual representation of the ping tests I have carried out:

DC01 <----> DC02 - OK

DC01 <----> Main Centre Switch - Unreachable

DC02 <---> Any Switch - OK

Like I said, main complex is fine but for some reason, no devices connected to the switches in the centre can obtain a DHCP address from it's respective VLAN, even when connecting to a port I've set to VLAN1, but I can communicate from DC to DC.

Just wanted to brainstorm potential causes and ideas on where and what to troubleshoot to see if I can find anything glaring or missing. Any tips and/or explanations would be greatly appreciated.

How do I troubleshoot a VLAN problem?

Edited November 26, 2020 by robertcladner55

[Jcx500]

How do I troubleshoot a VLAN problem?

 

What switches do u have

[RobD]

How do I troubleshoot a VLAN problem?

 

If I'm really struggling, I'll give each vlan on each switch an IP and see if I can ping it