Strange VLAN to VLAN issue - unable to Ping some VMs

[timbo343]

We have a bit of a strange one where we have multiple VLANs set up and the firewall rules are set on the Smoothwall to allow traffic to pass between each VLAN, however we have got a couple of VMs (Papercut / Impero VMs) where these VMs are not contactable from the VLANs even though they are in the in same rule as the VMs that can be contacted.

 

The server VMs that can be contacted are on the same host as those that cannot be contacted.

 

The things that i have tried:

 

- resetting both hyper-v hosts

- resetting the VMs that aren't able to be contacted

- updated the VMs with the latest round of windows updates

- disabled the Windows firewall

- checked windows firewall to make sure the ICMP rule is allowed

- configured Smoothwall firewall rules so that any traffic from the VLANs can hit the main domain VLAN

 

I'm at real loss as to why these server VMs aren't contactable yet they have been in the past.

 

Any ideas welcome.

[ThomL]

I'm sure you've checked this - network settings correct on the servers, correct gateway set?

[timbo343]

I'm sure you've checked this - network settings correct on the servers, correct gateway set?

 

Yup, IP / Subnet / Gateway / DNS all set correctly.

 

Devices on the Domain VLAN where the VMs live can be contacted without any issue.

[ThomL]

If you run a traceroute to an uncontactable server and compare to a traceroute to a contactable server does this highlight where the traffic stops and point you in a direction to investigate?

[timbo343]

If you run a traceroute to an uncontactable server and compare to a traceroute to a contactable server does this highlight where the traffic stops and point you in a direction to investigate?

 

Good idea - will try that out and see where it fails.

 

The gateways / interfaces on the Smoothwall manage to ping the VMs so the finger is pointing to Smoothwall though the firewall logs say it's working.

Edited October 18, 2023 by timbo343

[timbo343]

So a traceroot comes back with:

 

Traceroot to a problematic VM Server:

 

tracert {ip address of a problematic server}
1. the gateway of the VLAN
2. Request timed out

 

Traceroot to a working VM Server:

 

tracert {ip address of a working VM}
1. the gateway of the VLAN
2. the name of the server displays without any issues

 

Yet pinging the problematic server from the Smoothwall using the VLAN gateway works fine.

[Mako]

ACL on the gateway switch? Funky Smoothwall firewall rule preventing access from network to VM? You're having trouble pinging to the VM, but can the VM ping out to known devices?

[ThomL]

Clients in a vlan have their gateway set to an IP attached to a Smoothwall interface to allow the Smoothwall to apply rules/route the traffic right? It would hint it's the firewall rules somehow.

 

Have you tried rebooting the Smoothwall? I had an issue where port-forwarding and some firewall rules weren't working as expected - escalated to Smoothwall tier 2 support who advised I put everything back to how I had it configured, then reboot and test again ...and everything started working, the old off and on strikes again!

Edited October 18, 2023 by ThomL

[timbo343]

ACL on the gateway switch? Funky Smoothwall firewall rule preventing access from network to VM? You're having trouble pinging to the VM, but can the VM ping out to known devices?

 

No ACLs on the Switch, the switch is the core switch.

 

I've created a Smoothwall firewall rule that says "All Internal interfaces" to "All internal Interfaces" Allow all / everything. That still doesn't allow the problematic VMs to be contacted.

 

The VM servers which work that are on VLAN 1 cannot ping the devices on VLAN 2 but devices on VLAN 2 can ping the servers on VLAN 1.

 

- - - Updated - - -

 

Clients in a vlan have their gateway set to an IP attached to a Smoothwall interface to allow the Smoothwall to apply rules/route the traffic right? It would hint it's the firewall rules somehow.

 

Have you tried rebooting the Smoothwall? I had an issue where port-forwarding and some firewall rules weren't working as expected - escalated to Smoothwall tier 2 support who advised I put everything back to how I had it configured, then reboot and test again ...and everything started working, the old off and on strikes again!

 

Yup, tried a reboot of the Smoothwall - might have to escalate up to Smoothwall.

[Mako]

Has this just started happening or has something recently changed (such as moving the VMs from VLAN 1 to 2) ?

 

If they can ping in one direction but not the other, it sounds like a Smoothwall rule that's mono-directional. But if you've not changed anything then that's a headscratcher.

[DalekSec]

DNS???

[ThomL]

Turn logging on for every firewall rule on the smooth wall and watch the logs from the traffic - see if you can see another rule catching and stopping the traffic?

[Davit2005]

Can you ping from the servers gateway i.e. if the servers vlan is terminated on the firewall you might need to specify the source address of the ping, our firewall you do cause otherwise it attempts to ping out of the management. Ping one you know that works jus to verify the command or process.

 

Use the IP address at first because the firewall/router/core might not be able to resolve DNS

Edited October 18, 2023 by Davit2005

[timbo343]

Has this just started happening or has something recently changed (such as moving the VMs from VLAN 1 to 2) ?

 

If they can ping in one direction but not the other, it sounds like a Smoothwall rule that's mono-directional. But if you've not changed anything then that's a headscratcher.

 

It's only something that i've noticed, for example, our chromebooks (or other devices on the VLANs) cannot connect to our Mobility Print to upload documents but on the domain (VLAN 1) it works fine. Also applies to Impero.

 

DNS???

 

DNS is working fine as the servers that do work, resolve without any issues and pinging the IP address would rule out DNS.

 

Turn logging on for every firewall rule on the smooth wall and watch the logs from the traffic - see if you can see another rule catching and stopping the traffic?

 

When the rule is at the very top of the firewall rules i would have expected this to work - i've raised this with Smoothwall.

 

Can you ping from the servers gateway i.e. if the servers vlan is terminated on the firewall you might need to specify the source address of the ping, our firewall you do cause otherwise it attempts to ping out of the management. Ping one you know that works jus to verify the command or process.

 

Use the IP address at first because the firewall/router/core might not be able to resolve DNS

 

I'm pinging the VM servers based on IP from the Smoothwall using Source IP - the VLAN IP gateway and it works fine.

 

This really is a strange one.

[hallb15]

Long shot - are the time and date correct on the suspect VMs?

[timbo343]

Long shot - are the time and date correct on the suspect VMs?

 

Time and Date all ok on the affected servers.

[mavhc]

Something that's limiting the number of mac addresses per port? I've had port security on switches do that before

[timbo343]

Something that's limiting the number of mac addresses per port? I've had port security on switches do that before

 

Checked the Port Security on the 5406zl2 and there is nothing configured or no alerts in there.

[timbo343]

I've found the issue!

 

It is a Smoothwall issue and under Source NAT & LLB rules, i have a range of different IP addresses referencing VMs which are mapped to use specific outgoing lines as we have 2 lines so wanted to spread the load between out lines.

 

It turns out that disabling the rules where these servers are listed reactivates the connection between VLANs.

 

Looks like i need to do some house keeping on this now.

[timbo343]

Now this is tidied up, if anyone has this issue, under Network > Configuration > Source NAT & LLB Rules you need to add a rule / rules to the top of the list so that:

 

Source IPs = The IPs or networks the device(s) are coming from

 

Destination IPs = The IPs or networks the device(s) are accessing

 

Services = Any

 

SNAT = Ticked

 

LLB Pool / Gateway = The gateway address on what your VMs live

 

This rule is not a firewall rule, a firewall rule still needs to be configured.