Strange Internet Problem

[matshep]

Hello all, long time reader but first time poster so I'll start by thanking you all for the years of advice, fixes and good ideas I've been able to pick up here!

 

I've got a strange problem that's driving me mad and I wondered if any of the clever people out there might have an idea to point me in the right direction.

 

We've got a small number of devices (three at present, one Microsoft Surface Pro, two iPhones) that are able to connect to our network but not access the Internet and I can't figure out why. Here's the setup and steps I've taken, firstly with the Surface Pro as this was the first device to start misbehaving:

 

We have a fairly simple network, one VLAN, Virgin Internet and Smoothwall for proxy/filtering. The Surface can use local resources (shares, printers, etc.) with no problems either when working wirelessly or when docked but cannot browse the Internet, send/receive email or access any other offsite service when working on wireless - in its dock all works fine. The behaviour is common to multiple browsers - they all return a timeout message, Chrome being the only one that offers anything else as it at least shows 'Waiting for proxy tunnel' in its status bar for a few minutes before failing. It can resolve external addresses but pings fail, pings to our Smoothwall also time out. I've taken it home to test on my home wifi and it works fine. In both cases it is set to auto detect proxy settings but the behaviour is the same even if I manually specify our proxy address when onsite. This led me to think maybe something at the wifi end was blocking it since it works ok when connected to the dock and ethernet. We have two wireless networks in different buildings - Aruba in half the site and an old Extricomm system in the other half that's destined to be replaced with sitewide Aruba this year. These two systems don't talk to each other at all but the behaviour of the Surface is the same on both and neither appear to detect it as a Rogue AP or anything similar - the wireless performance is pretty great for anything on the local network so I don't think the problem is there. I then moved on to IP addressing and tried it with a few different addresses - DHCP, static and DHCP reserved - same results regardless. I tried reinstalling Windows in case something had gone screwy somewhere, both with the built in restore and with a fresh build from one of our images. Problem persists and even continues when I boot from an external OS on a USB drive - I've tried this with Ubuntu, Windows To Go and my trusty Hirens Boot USB - all the same behaviour. Smoothwall promise me that their box doesn't do any filtering based upon MAC, only IP and say that it's definitely nothing at their end so I feel like something between the device and the Smoothwall box is blocking web traffic. Having tried it in different areas of the site with different wifi systems the only other common factor sitting between the device and the Smoothwall box is our core switch (a HP 5406zl) but I can't see anything on there that indicates a problem. I was ready to think it was some extremely odd hardware issue with the individual device (though I couldn't see what since it works ok at home) when the two iPhones started behaving in what appears to be the same way.

 

I've got a horrible feeling I'm going to kick myself when I get to the bottom of this but at the moment I can't see the wood for the trees - can anyone suggest a direction to go in?

 

Thanks in advance for any help you can offer.

[bald_pig]

1) make sure the network config matches.

 

2) what entries do you have on the smoothwall for the ip of the device? Should tell you whether the devices are reaching the smoothwall or not.

[matshep]

Hi, many thanks for replying.

 

1. The devices all have the same configuration and are receiving IP addresses from our normal range via DHCP - if I reconfigure them (either manually or using a reservation in DHCP) to use a different address the behaviour persists, likewise if I configure another device with an IP that hasn't worked on one of these three devices the other device works without a problem. Hope this makes sense!

 

2. The only traffic I can see on Smoothwall for the IPs relates to times when those addresses have been used by other devices - it seems that no matter what address the affected devices have they can't get to the Smoothwall box which is what led me to wonder whether there was something blocking/redirecting traffic from them at the MAC layer since that (and the hardware itself) is the only thing that persists in every non-working scenario. With three separate devices it seems unlikely (though not impossible I admit) that hardware is to blame and I haven't seen this kind of behaviour from a hardware fault in the past but anything's possible - right now it just 'feels' like a network issue rather than a device issue.

[bald_pig]

Next step for me would be to run wireshark to see what the traffic from those devices are doing.

[ibpalle]

I assume the iPhone devices are on WIFI - is the Surface Pro also on Wifi and does the problem only happen when using Wifi? Are you using RADIUS on Smoothwall for BYOD logins?

Does this ever happen with other devices?

[matshep]

Yep, all three are on wifi and the problem only occurs when using wifi. They aren't using RADIUS/BYOD as they're school-owned devices - they go straight onto our wireless via a key and the iphones usually have reserved addresses with exceptions in Smoothwall for filtering so that they can be used for Skype, Facetime, etc. - we're a boarding school and they're used (supervised) for students to call home, often internationally. The Surface is part of our domain and has been tried using IDEX, NTLM and transparent authentication - regardless of the method the behaviour is the same, no Internet when on wifi, Internet when docked. The problem persists whatever IP the devices use, that's what got me thinking it was something blocking them by MAC.

 

I'm waiting to get hold of the Surface so I can see what Wireshark throws up as per bald_pig's suggestion above - as is typical it's used by one of the SLT who can't do without it! I should have it for a bit of time tomorrow to run Wireshark, at the moment he's working from it on his desk via the dock and everything's fine.

[ibpalle]

Hi Matshep

 

One thing to test out to see if the issue could be with Smoothwall and the way it handles Wifi clients is when it happens on the iPhone or Pro, login to the command line of Smoothwall and try this command:

 

# iptables -F mac1x

 

And then test your iPhone/Pro again. If this works, contact Smoothwall support and we can see if we can make this flush permanent.

[matshep]

Thanks - that looks very helpful. I did try and get onto the Smoothwall via Putty a few days ago to see if I could see the ARP tables or anything similar but I don't have the root password so got nowhere. Do you know if the command line can be accessed from within the Smoothwall admin console by any chance? Or the root password reset via it? No worries if not - although Smoothwall said it definitely wasn't a problem at their end they should probably at least reset the root password for me so I can have a look if needs be.

[ibpalle]

Resetting root password does require a reboot - instructions here:

 

https://kb.smoothwall.com/hc/en-us/articles/360002032730

 

By default the standard is just 'smoothwall' so hopefully the pw was changed, just lost in time somewhere

 

I also do not think the issue is on the Smoothwall given your description but better to try the one thing that may interfere.

[matshep]

Thanks, I really appreciate the help. Just tried that default password just in case but had no luck. I'll open a ticket with Smoothwall to see if they can reset the password for me and then try your CLI suggestion once I can get on.

 

Thanks again.

[ibpalle]

Matshep

 

Instructions on resetting the root password are in the link I sent you.

[matshep]

Ah, sorry!! Didn't look properly. Ok, that will need to be an out of hours job due to the reboot but I'll arrange a time to give it a try. Thanks again.

[matshep]

Many thanks ibpalle for your suggestion, once I was able to get onto the Smoothwall console I tried this but the problem persists unfortunately. Am awaiting custody of the Surface Pro so I can do some WireSharking...

[ibpalle]

I didn't think that would be the issue but worth a try. Thanks for getting back.

[TRS-80]

Did you get your hands on it? Also Wireshark on a Surface isn't going to be useful for wireless troubleshooting, doing a monitor mode capture on a Mac next to it will give you much more information.