Clients receive IP address but have no network access

[MrBrownPr]

Not sure if this is the right place to start looking for a solution to this, but I've tried everything else I can think of so I am sure this must be server related!

 

In our school we have trolleys of student laptops that we use in lessons. We usually use around 15-16 laptops in a lesson so we have 1 laptop between 2 students. When we turn them all on together they all connect to the Wifi and receive an IP address, but some of them will have no network access for a period. This period can range from a few seconds to 10 minutes or more.

 

Some note:

- This does not affect the same laptops every time

- It is not always an exact number, some days you turn on 12 laptops and they will all be fine, another day you can turn on 12 and only 9 will be properly connected

- This isn't just a Windows issue, we have Ubuntu laptops that are similarly affected

- Checking ipconfig shows a valid IP address and correct addresses for the DNS

 

When a laptop has no network access I mean that nothing is working, no ping, no nslookup, no access to network shares, no internet access, nothing. I usually start pinging our proxy server and leave the laptop until I can see that the pings are getting responses and then I know that all network connectivity is restored.

 

Any suggestions?

[psydii]

What authentication are you using?

What APs? And is their firmware up to date?

Are the drivers on the laptops up to date?

Have you got the W7 enterprise hotfix rollup installed?

[mjk]

try pinging the default route and see if the mac address is changing. it's worth doing to rule out layer2 errors.

[Michael]

I'm guess you have more than one DC and if so, this sounds like a DNS issue. Make sure DNS can replicate on all servers and that the servers themselves point locally, then to each other.

[Muz]

What sort of WIFI access points are you using?

 

I'm guessing you're testing network connectivity using an IP and Hostname to rule out DNS?

[MrBrownPr]

Thanks for all the responses so far, here are some answers to the various questions raised:

 

 

What authentication are you using?

WPA2, but had the same problem using a RADIUS authentication server. I switched the student laptops to the secondary (WPA2) WiFi network in the hope that it would alleviate the problem.

 

 

What APs? And is their firmware up to date?

AreoHive APs, all their firmware is up to date.

 

 

Are the drivers on the laptops up to date?

Latest WiFi drivers available from Dell for the Windows 7 laptops and I believe the Ubuntu laptops have the latest Broadcom firmware/drivers - I always find it a bit tricky to tell with Linux!

 

 

Have you got the W7 enterprise hotfix rollup installed?

Not as far as I am aware, but like I stated this isn't a Windows-exclusive problem, so I don't believe it is a problem on the laptops themselves. Still, I am willing to try anything at this point so if you can point me to where I can read up on this I would appreciate it.

 

 

try pinging the default route and see if the mac address is changing. it's worth doing to rule out layer2 errors.

Not sure I follow what you're suggesting I do here. I have tried pinging various things, the domain server through both it's hostname and IP address, the proxy server by IP address, the DNS by IP address and other devices connected to the network by both IP address and name. What else do you suggest?

 

 

I'm guess you have more than one DC and if so, this sounds like a DNS issue. Make sure DNS can replicate on all servers and that the servers themselves point locally, then to each other.

Only one domain controller on the network. We have two DNS though, but like I said earlier when I check ipconfig on the affected laptops the DNS address is listed correctly.

 

 

What sort of WIFI access points are you using?

I'm guessing you're testing network connectivity using an IP and Hostname to rule out DNS?

As above, AeroHive APs. And again as above, I've tried testing using both IPs and hostnames.

 

 

What's next?

[mjk]

Not sure I follow what you're suggesting I do here. I have tried pinging various things, the domain server through both it's hostname and IP address, the proxy server by IP address, the DNS by IP address and other devices connected to the network by both IP address and name. What else do you suggest?

 

Ping the default route /server when it is working and run "arp -a" to show the arp table, note the mac addresses. Also do this from the switch that the AP is connected to. When the connection disappears run arp -again and see if the mac changes. If there is any layer 2 issues like network loops you will see here. If not then move up to layer 3.

[MrBrownPr]

Thanks, I've run "arp -a" from a working laptop and saved the output to compare next time I have an issue. Will update here later!

[mjk]

also see if you can do it from a switch, if any address changes you have network loops./

[Michael]

Try disabling DNS Server on your non DC temporarily. I have a feeling it's something to do with this, seeing as it affects not just Windows.

[MrBrownPr]

also see if you can do it from a switch, if any address changes you have network loops./

Thinking about this, wouldn't this affect all clients if there were network loops? I'm confused!

 

Try disabling DNS Server on your non DC temporarily. I have a feeling it's something to do with this, seeing as it affects not just Windows.

Was considering doing this actually, so will try this today and see if it makes any difference. Nothing ventured, etc.!

[mjk]

Thinking about this, wouldn't this affect all clients if there were network loops? I'm confused!

 

I made the assumption that your WiFi clients are on a different vlan. Clearly assumptions being the mother of all screw ups.

[MrBrownPr]

This morning I disabled one of the DNS to see if I could rule out an issue between them. Had the laptops out this afternoon and it was no better. Couldn't do an arp -a as these were the Linux laptops and I wasn't sure of the equivalent command.

 

Any new ideas?

[Deanuk]

We had this issue on some printers. Try dropping the network speed down to half-duplex. We had an issue where the printer was struggling to auto negotiate.

[Edsure]

We had a similar problem in one of our schools with laptops connecting through a netgear wifi system. Turned out to be problematic access points causing the problem. We changed the AP's in the locations that this seemed to be happening and this solved the problem. Some of them needed firmware upgrading on the AP's also.

[MrBrownPr]

Unfortunately, all the APs are on the latest firmware and it seems to happen everywhere!

 

I was starting to wonder if it is a problem with one of the switches, but I've no idea how I am going to start to diagnose that!

[Edsure]

Very strange. Are the switches managed or unmanaged?

[andydis]

Ping the default route /server when it is working and run "arp -a" to show the arp table, note the mac addresses. Also do this from the switch that the AP is connected to. When the connection disappears run arp -again and see if the mac changes. If there is any layer 2 issues like network loops you will see here. If not then move up to layer 3.

 

THIS.

 

I have seen this twice before,

The first Time a student was running a WPA Cracking utility to try and "intercept" all wifi packets , result was network loss and "stop"

the second time a student had the wifi password allready and was running "cain and abel" network password sniffer, this basically changes all the ARP requests to his laptop then onto the default router and caused the network to "stop".

[MrBrownPr]

Update on this as I am really starting to tear my hear out with it!

 

Firstly, the problem is still on-going. I've looked at everything I can think of.

 

In a nutshell, the problem is that if I turn on 15 laptops at a time in one room, only around 10 of them will get a network connection straight away and will work fine. The others will show an IP address if I do an ipconfig, but will have NO internet connection. Pinging anything on the network (I usually use the proxy server's address to test) just times out.

 

After waiting anywhere from 5 to 20 minutes, the affected laptops will eventually gain full access to the network. This is without me doing anything more than running a ping using ping xxx.xxx.xxx.xxx -t so I can see that the timeouts have stopped and we've started to receive responses.

 

I don't have laptops that will start with a connection and then see it drop and later come back. If a laptop gets connected when I turn it on, it will stay connected.

 

Server is running Windows Server 2012 R2. Clients are all Windows 7 laptops currently, although I have seen the same problem on Windows XP and Ubuntu. These are wireless connections through Aerohive APs, all of which have been updated to the latest firmware. DC, DNS and DHCP are all on the same server. We also have a secondary DNS and disabling this has made no difference.

 

Any suggestions? Any other info required?

Edited June 29, 2015 by MrBrownPr

[Edsure]

Have you checked the load balancing on the access points? It maybe that the AP's are limited on how many clients can be connected at the same time, this can cause issues like those you are having.

[Muz]

If I was troubleshooting this for you I'd go down the path of the access points not being able to keep up with the number of associations (capacity issue) although this shouldn't be a problem with these WAPs.

 

Have you got a spare WAP you can put nearby to try and load balance between 2 sets of laptops and see if that improves the issue?

 

When I was working in a school these were the same symptoms we faced before we did a full WIFI upgrade.

[Michael]

Re-reading through this, rather than DNS it does look like an Access Point issue. Are these powered separately or do they use PoE?

 

The reason I ask, is I have come across bad wiring which delivers power (via PoE), but no data, so the device connects to the WiFi in question but cannot get an IP. I think the reason you're seeing randomness is because the notebooks are switching from a bad Access Point to a good Access Point.

 

If they are PoE based, I'd suggest you start disconnecting them in small batches until you can isolate the bad Access Point.

[psydii]

I had exactly this (number of devices, problems only for the last few and the 20 minute wait before they all just work). It was misconfigured 802.1x on the APs and via Group Policy.

 

We paid for a wifi expert to pop in diagnose and resolve it for us, so the details are hazy.

[Deanuk]

Have you tried dropping the connection down from auto negotiate? We had this issue with a printer and this solution worked for us.

[MrBrownPr]

Ugh. Still struggling with this!

 

Have you checked the load balancing on the access points? It maybe that the AP's are limited on how many clients can be connected at the same time, this can cause issues like those you are having.

 

The AeroHive APs we are using are set to support 100 clients each. We have them in almost every classroom, so we are getting nowhere near their capacity.

 

If I was troubleshooting this for you I'd go down the path of the access points not being able to keep up with the number of associations (capacity issue) although this shouldn't be a problem with these WAPs.

 

Have you got a spare WAP you can put nearby to try and load balance between 2 sets of laptops and see if that improves the issue?

 

When I was working in a school these were the same symptoms we faced before we did a full WIFI upgrade.

 

As above, we should have plenty of capacity on the APs, so I can't see this being the issue.

 

Re-reading through this, rather than DNS it does look like an Access Point issue. Are these powered separately or do they use PoE?

 

The AeroHive APs are PoE. I really don't think the issue is with the laptops communicating with the APs, as I can login to the AP management interface and see that the laptops are connected. And as I've said before, the laptops have all the correct information, i.e. IP address, DNS and DHCP addresses... just can't access the network shares or the internet!

 

I had exactly this (number of devices, problems only for the last few and the 20 minute wait before they all just work). It was misconfigured 802.1x on the APs and via Group Policy.

 

I would LOVE to hear more about this! We had an engineer in on Tuesday from our tech support company. He was hear two hours, said he didn't know what the problem was either and left. Not heard from him since!

 

Have you tried dropping the connection down from auto negotiate? We had this issue with a printer and this solution worked for us.

 

Yes, tried setting the APs to half-duplex, didn't make any difference unfortunately.

 

Oh, and while I remember, I have done an arp -a on a laptop with the problem and one without and the outputs were identical.

 

Here is the text from the email I sent to the engineer last week, giving a summary of the problems I'm having:

 

These are the current problems we are experiencing with our WiFi network:

 

1) Some laptops will appear to connect to the network but are unable to access any network shares or internet.

 

In these cases an ipconfig shows that the laptop has received a valid IP address from the DHCP. If I check the DHCP leases on the network I can see the laptop and IP address listed. The ipconfig will also show the correct settings for the gateway and the DNS servers. Everything appears as it should, but still no internet/network connectivity. Pinging the server times out, as does a ping to the school’s proxy server.

 

If laptops are connected to the same Access Point then there appears to be some connectivity between them. E.g. I was able to connect game clients (Minetest) on laptops that were suffering this problem to a laptop which was running the game server, albeit with full network access on that laptop.

 

Until recently if I turned on a class set of 15 laptops then I would find that around 10 of them would connect without problems after a minute or two, the remaining 5 could take anything from 10 to 30 minutes or more. Recently I have been finding the opposite, only a few laptops will work correctly with the rest unable to get connections for some time, if at all.

 

This problem has also started to affect staff laptops recently which is causing a lot of issues for teaching staff. Note that this is only a problem on WiFi and usually if I connect them to a wired Ethernet connection then full network (and internet) access is almost immediately restored.

 

I have tried a WireShark capture when my laptop has been in this state and reviewing the capture afterwards it seems that my laptop can definitely see the network traffic, so I am at a loss as to why it can’t send/receive data itself.

 

I am starting to think this is a capacity issue as like I said it seems to have got worse recently and I can only attribute this to the additional teacher laptops and ipads that we’ve recently added to the network to accommodate new classes.

 

If it is a capacity issue I am struggling to find where the bottleneck is.

 

The DHCP scope has plenty of capacity, it is configured with a 255.255.252.0 netmask so should give us over 1,000 addresses and we are usually hovering around the 250 mark. We use VLANs and I have checked that the netmask is set appropriately on all of these.

 

2) Losing connection to our proxy server (and therefore internet connection)

 

The other issue we have is that we are losing connection to our proxy server. This is connected to the core switch via Ethernet cabling, with a SonicWall device sitting between the switch and the proxy server. When we lose connection, pings to both the SonicWall and the proxy server itself will time out. Resetting either the SonicWall or the proxy server does nothing to correct the problem, the only thing that does seem to work reliably is unplugging the Ethernet cables from the core switch to the POE switch that powers the access points in one part of the building, waiting a few seconds and then plugging them back in.

 

I don’t know if this would seem to indicate that the core switch or the connection to the proxy server/SonicWall is being overwhelmed with requests that is then alleviated by disconnecting the POE switch?

 

Does anyone have any suggestions?

Edited October 16, 2015 by MrBrownPr