Smoothwall BYOD - guests stop being able to talk to the Smoothie

[sonofsanta]

I've tried arguing this with Smoothwall support but they're insistent it must be my network, which I'm convinced it isn't, and makes no sense why it would be. So I'm coming here (a) to see if anyone else is experiencing it (b) to get ideas on more tests I can run to narrow down the cause/prove it's in the Smoothie.

 

So we've been running BYOD through Unifi and Smoothwall for years. Never had any trouble with it. But since a few months back, we're starting to see an issue where clients connecting to the guest network are authorised but no longer get a connection; when you run an ipconfig /all everything is there and correct and it has a DHCP lease, but you can't ping the default gateway, even though said gateway is the Smoothwall that's also the DHCP and just handed out the lease. I either get "Request timed out" or "Destination port unreachable", neither of which are specific enough to really help.

 

All this happens on its own VLAN: the Smoothwall uses port 2 to connect directly into this VLAN at the core, and it's routed at the core. It's a pretty straightforward setup between Unifi and Smoothwall: Unifi has Smoothwall's IP as its RADIUS Auth server, and Smoothwall has the Unifi AP address range as clients. That's all working because (a) nothing has changed in years (b) it does authorise the user.

 

The only fix I've found is to restart the Smoothwall. The whole box. Unsurprisingly, taking down the internet for five minutes while we're in the middle of a hybrid-teaching paradigm is not particularly convenient. It does suggest, however, that there's some table or cache or something inside the Smoothwall hitting a limit; after a restart everything is fine for a couple of days, then at a certain point everything is broken. It doesn't discriminate between user, and once it's broken it seems to be broken for everyone.

 

Nothing has changed on my network layout, physically or logically, to cause this. It can be resolved by restarting the Smoothwall and only the Smoothwall--to the point where I can connect a laptop to the guest network, verify the problem, restart the Smoothwall without disconnecting the laptop at all and as soon as the Smoothwall is back up, the pings start working.

 

I'm at a point now where I'm scheduling an overnight reboot of the Smoothwall every day, which is ridiculous, really. I suspect the problem crept in with an update but with everything going on last year, and so much time off-site, I ended up applying a bunch of patches at once to catch up so I don't know which one it was specifically--plus I applied them before people came back on site, and it's the kind of problem that stayed hidden until everyone came back. It's getting irritating, though, and I'd really love to get it pinned down and sorted out.

 

So: anyone else seeing it? Or got any ideas of testing I can do to narrow down the cause? (Though I just restarted the box this morning, so I probably won't be able to run any suggested tests till Monday morning...)

[robk]

Are you on a recent release? I have seen a simalar problem. I think somewhere smoothwall support gave me a command to empty the "mac1q" table. which magically got everything to burst back into life. Will see if I can find the note of the exact command.

[sonofsanta]

Are you on a recent release? I have seen a simalar problem. I think somewhere smoothwall support gave me a command to empty the "mac1q" table. which magically got everything to burst back into life. Will see if I can find the note of the exact command.

 

 

Leeds 45 right now, which isn't quite the most up to date looking at it, but as best I can tell it only started happening after we installed some updates earlier in the year.

 

A MAC table getting filled up sounds like exactly the sort of thing that would cause this. If you can find the command that'd be amazing, I'll definitely test that next time I see the issue.

[gumbygaz]

We had this issue too with our BYOD system, support put a script on our box to clear the table.

This is the call notes which may help...

 

"We put a script in place in /etc/actions/secondboot/ called 9999deletemac1x with the following content:

 

#!/bin/bash

#This script deletes the mac1x chain from iptables to resolve MAC to IP address binding issue with BYOD/RADIUS setups

 

/sbin/iptables -F mac1x

/sbin/iptables -D INPUT 1

/sbin/iptables -D FORWARD 1

/sbin/iptables -F mac1x

/sbin/iptables -X mac1x

 

This will run at boot to disable the mac1x chain. You do not need to run anything manually.

"

[sonofsanta]

We had this issue too with our BYOD system, support put a script on our box to clear the table.

This is the call notes which may help...

 

"We put a script in place in /etc/actions/secondboot/ called 9999deletemac1x with the following content:

 

#!/bin/bash

#This script deletes the mac1x chain from iptables to resolve MAC to IP address binding issue with BYOD/RADIUS setups

 

/sbin/iptables -F mac1x

/sbin/iptables -D INPUT 1

/sbin/iptables -D FORWARD 1

/sbin/iptables -F mac1x

/sbin/iptables -X mac1x

 

This will run at boot to disable the mac1x chain. You do not need to run anything manually.

"

 

 

Ah, brilliant! I'll get on to support to ask them to replicate this (rather than buggering about and risking this myself) but it's bloody frustrating that two of you got a proper acknowledgement and solution and I got fobbed off with "it must be your network"...

 

Thanks for this!

[robk]

@Grumpygaz that's the perm fix. I would suggest getting smoothwall support to put that in rather than doing it yourself just in case.

 

The command I had was a quick fix that proved it was that fault but it certainly sounds like it.

[foofighterjim]

Oh, now this is interesting. We've just had a new S10 appliance and I patched it to the latest release before putting it into production. We've had a couple of occasions in the last month where the BYOD authenticates in the morning but just seems to stop by the afternoon, everything returns to normal the following day. I've spent very little time investigating has it has only happened a couple of times and I've had a bit on my plate.

 

Pretty much the same setup as you but with Aruba IAP; Smoothwall doing the RADIUS and DHCP.

[sonofsanta]

Oh, now this is interesting. We've just had a new S10 appliance and I patched it to the latest release before putting it into production. We've had a couple of occasions in the last month where the BYOD authenticates in the morning but just seems to stop by the afternoon, everything returns to normal the following day. I've spent very little time investigating has it has only happened a couple of times and I've had a bit on my plate.

 

Pretty much the same setup as you but with Aruba IAP; Smoothwall doing the RADIUS and DHCP.

 

 

We're also on an S10, FWIW, though it's about a year old

[TechMonkey]

Hmmm, interesting we have had some people say their phones wouldn't connect and I assumed it was the Android update to do with certificates. May have to get this tried. Though I don't hold out much hope of getting hold of support to do it

[DrCheese]

Oh there's a perm fix for it now?

 

I was orginally told that typing

 

iptables -F mac1x

 

on a root shell would fix any stuck machines *instantly* but it would occur again, so was told to modify...

 

/etc/actions/secondboot/0090firewall

 

& REM (#) the two lines that look like the below around line 169

 

169 # Connect INPUT and FORWARD chains to the mac1x chain

170 /sbin/iptables -I INPUT -j mac1x

171 /sbin/iptables -I FORWARD -j mac1x

 

And reboot the Smoothwall.

 

Only issue with doing it this way is I have to remember to reapply it after each patch release - If there's a proper fix for it now then I'm interested (as it's a pain doing both the primary & the failover)

[sonofsanta]

Clearly it's a larger problem and it'd be nice if they'd just patch it properly!

[DrCheese]

Agree - It seems it's common to the 802.11x setup and UniFi networks.

[Techie-v2]

We are seeing something similar. Users authenticate on the first day, but come in the next day and recieve a "Connected, no Internet" message.

To fix it you can forget the Network on the users device, sign in with a different set of credentials, the forget again, and sign back in with the original credentials and it works again, untill the next day when you have to rinse and repeat.

Ours is ruckus but again 802.11x with radius on smoothwall.

 

Does this sound like the same issue to you guys?

[DrCheese]

Yes, that sounds like the same kind of issue - When you've got a device that's presenting with it, try loading a website (It'll hang) and run the

iptables -F mac1x

 

 

command via SSH - It should instantly start responding again. If it does, then yes it's the same issue.

[foofighterjim]

What do you all have set for the DHCP lease time?

 

Ours is set to 60 minutes for the default and 120 minutes for the maximum. I think these may be the default values as I actually don't recall changing them. I'm just wondering if longer values may be contributing.

[sonofsanta]

Yes, that sounds like the same kind of issue - When you've got a device that's presenting with it, try loading a website (It'll hang) and run the

iptables -F mac1x

 

 

command via SSH - It should instantly start responding again. If it does, then yes it's the same issue.

 

 

Is it literally a case of putty in as root, run that command and job done? At least, for the next two days till you need to run it again... but it's easier than rebooting the whole dang box

[sonofsanta]

What do you all have set for the DHCP lease time?

 

Ours is set to 60 minutes for the default and 120 minutes for the maximum. I think these may be the default values as I actually don't recall changing them. I'm just wondering if longer values may be contributing.

 

 

Same settings here, and likewise, don't think I've ever looked at them. That does seem short given AD DHCP defaults to 8 days, renewing at 4... I'd expect a guest system to be shorter but not that much shorter

[ibpalle]

We had hoped this issue had been addressed in an earlier update - think it was one in the thirties but apparently not.

 

Getting rid of the mac1x chain will prevent this from happening - possibly the new features on iOS and Android with dynamic MAC addresses is causing problems as well when paired with the mac1x functionality.

[DrCheese]

Ah that would make sense, as I believe the issue started appearing around the time OS 14 came out.

[simpsonj]

Interesting, I was told that using the Smoothie box for RADIUS and DHCP wasn't best practice, so I shifted to using a DC instead.

[ibpalle]

DHCP yes, the DHCP server on MS Servers have much better functionality. RADIUS - sure. If it's just for 802.1x and BYOD< Smoothwall is easy to use for both auth and accounting.

[simpsonj]

DHCP yes, the DHCP server on MS Servers have much better functionality. RADIUS - sure. If it's just for 802.1x and BYOD< Smoothwall is easy to use for both auth and accounting.

 

The Smoothie was fine at authenticating users, but the Safeguarding Report didn't pick up who the users were, only their IP address. Since moving to the DC providing 802.1x for the BYOD network, reporting is far more accurate.

 

Anyway, I don't want to drag this off topic, but I would suggest that moving to a server to provide 802.1x authentication, whilst sending the accounting to the Smoothwall, is a valid alternative.

[robyholmes]

Unifi and Smoothwall here too, with Smoothwall doing DHCP, RADIUS and Accounting using a Unifi wireless system. A few weeks ago we had no new users able to authenticate. Already authenticated ones were fine. It seemed to magically fix itself but I'm wondering if this was something to do with this.

[DrCheese]

The Smoothie was fine at authenticating users, but the Safeguarding Report didn't pick up who the users were, only their IP address. Since moving to the DC providing 802.1x for the BYOD network, reporting is far more accurate.

 

Anyway, I don't want to drag this off topic, but I would suggest that moving to a server to provide 802.1x authentication, whilst sending the accounting to the Smoothwall, is a valid alternative.

 

huh, I didn't think it was possible to have the DC do the 802.11x stuff - I thought you had to use the smoothwall as the auth server (I knew you could do DHCP on a windows box tho)

[simpsonj]

huh, I didn't think it was possible to have the DC do the 802.11x stuff - I thought you had to use the smoothwall as the auth server (I knew you could do DHCP on a windows box tho)

 

It's how I've got it setup here, apparently it's best practice to have a DC conduct the 802.1x stuff as it reduces network traffic... As long as Accounting goes to the Smoothwall, it works fine.