Meru/Fortinet with HP Chromebooks G3/G4/G5 (connectivity issues)

[Bruce123]

Good afternoon,

 

We've been experiencing issues with our HP Chromebooks (models G3/G4 and G5) connecting to Meru WiFi and wondered if anyone else had noticed any problems.

 

The issue users are experiencing is intermittent drop in connection (the users sees the wifi icon on the status bar, drop and re-establish intermittently).

 

We ran some logs for the stations affected (from the controller) while they were experiencing the issue and sent them to Meru and they say they indicate the device is switching between 2.4Mhz and 5Ghz, but don't know why.

 

Initially they asked us to reduce roaming aggressiveness on the adapter on the Chromebooks (actually I think lowing Band Steering aggressiveness on the client might be more appropriate). I explained that none of the properties can be changed on Chromebooks (I had to facilitate remote access to prove it), so they are now asking us to setup a separate SSID that only broadcasts on the 5Ghz band, so they never try to change band.

 

Anyway, I was just wondering if anyone knew of any way to adjust the wifi adaptor settings on Chromebooks? Is there a text file containing these values? We can get root access via developer mode, so if it is there we should be able to modify it.

 

In summary, I think there is some kind of incompatibility between Meru's Virtual Cell Technology (where the clients see all APs as a single AP with one MAC address operating on one channel).

 

I have a theory that might be happening is the Chromebook is picking up a poll frame from a distant AP with a weak signal (as well as a string signal from the closest AP), wrongly assume the signal is weak on that band and switch to a stronger signal on the other band.

 

Kind Regards,

 

Bruce.

[TheRobins]

Hi Bruce, Feel your pain here. I have had issues with Intel chips on Meru especially older chips. I must admit since upgrading firmware on the controller and AP's to 8.0.5.0 Fortinet branded I have not had any issues. Must admit I only have 6 Chromebooks here. My main issue was with laptops, but as you have been advised you can change all settings easily via windows drivers.

 

There were also very good with modifying some of my RF settings, turning down the aggressiveness of the roaming on their AP's along with slightly reducing the power on a large block of AP's, which would help null out those weak distant signals your Chromebooks may be picking up.

 

I wonder if you are using them in large groups at the same time?

[Bruce123]

Hi Bruce, Feel your pain here. I have had issues with Intel chips on Meru especially older chips. I must admit since upgrading firmware on the controller and AP's to 8.0.5.0 Fortinet branded I have not had any issues. Must admit I only have 6 Chromebooks here. My main issue was with laptops, but as you have been advised you can change all settings easily via windows drivers.

 

There were also very good with modifying some of my RF settings, turning down the aggressiveness of the roaming on their AP's along with slightly reducing the power on a large block of AP's, which would help null out those weak distant signals your Chromebooks may be picking up.

 

I wonder if you are using them in large groups at the same time?

 

Sorry, do you mean Fortinet support were good with modifying some of RF settings, turning down the aggressiveness of roaming and slightly reducing the power on large block APs?

 

Because with the support jobs we have opened, they have never got as far as tinkering with settings, they just confirmed that they looked to be OK, but perhaps they have been too focussed on (what we believed to be) a Chromebook specific issue.

 

Yes we are using them in large groups at the same time (and APs are located quite densely within the buildings). We have classes of up to 30 Chromebooks in use in a classroom at a time. We own several thousand of the HP Chromebooks (G3s and G4s). The network adapter on G4 is Intel 7260NGW . But the Acer Chromebook we've also been testing has been rock solid (no drops in connection).

 

What we've realised is that just adding additional APs doesn't increase/aggregate bandwidth when operating in Virtual Cell mode (unlike standard multi cell networks, where AP can operate on different channels).

 

Looking at Monitoring -> Wireless Statistics on the controller reveals that many APs are experiencing high noise levels (-90db) and many see a high number of neighboring APs (up to 50). Some others have a high tx loss rate of up to 50%, but what any of this means....

 

As regards to Virtual Cell. I am seriously wondering if we might be better off without it.

 

Out of interest, do you have band steering enabled on the controller? We have band steering to A band (5Ghz).

 

In relation to the inability to modify wireless settings on the Chromebooks, it occurred to me on my drive home yesterday that ChromeOS is based on Linux, so I am going to do some research to see if there is a standard location on Linux where the settings are stored, and could they be in the same location on the Chromebooks. I will try lowering band steering aggressiveness and roaming aggressiveness (and maybe even disable 2.4 altogether). Mine you, whether the change will invalidate their warranty is another matter..

 

Kind Regards,

 

Bruce.

[TheRobins]

Hi Bruce

 

I must admit, it took a few weeks to get to this stage, but it did work. If you can see 50 Ap's from some AP's you have far to many. I am quite experience with rf and it doesn't matter how good your software is, if you ap can hear even 4 - 5 other AP's it is going to impact.

 

A post I made well back last year could be a good starting point to do some monitoring of the actual station information. I have attached at the bottom of this message. Its just worth having that window open for a day and looking at any AP handoff's and see if there is a lot of switching going on. I cannot remember off hand but you can also narrow down the station log to a certain MAC address, maybe pick one of your Chrome books and watch it over a day or two when its in use. I must admit maximum number of devices I have in use are about 130, average of about 60. This is spread over 25 well spaced AP832e's. I did have issues in our hall area where 6 AP's were all equal distance in signal terms to the AP in the hall which was high up on the roof. I have disconnected that now for 6 months and I have never had any other issues in the Hall with dropouts at all. That AP is waiting to be moved when some new classrooms are built soon.

 

I think sadly that the Fortinet support don't tend to look at your network setup as it is. RF is a wonderful technology to learn, but it never sticks to the rule book at all. When these companies who sell the kit do their survey's they do make me laugh. I take their survey and knock off half the AP's suggested, not had many issues since

 

The 7260 is the problem chip here. I must admit I have had 30 new Lenovo laptop's with the Intel 3 AC series chip and they are much better with the hand off issues. Since making the driver changes on our 30 7260 laptops I have not had any complaints. As you mentioned I don't think you will be able to modify these settings on the chromebook. Looking at a quick google search they tend to use older Linux drivers from Intel, whether google will let you have access to these settings is another matter.

 

 

 

 

 

 

 

 

 

Login into an SSH session the running the following. Station log > enable

 

Code:

Meru-Controller(15)#

Meru-Controller(15)#

Meru-Controller(15)# station-log

Interactive Per-Station Event Logging Shell (enter "help" for help)

By default logging is Disabled (enter "enable" to Enable logging)

station-log> enable

Logging enabled

station-log>

What you will see is the devices on your network as the communicate with the controller and as it hands it around ap's similar to below

 

Code:

2015-May- 7 13:37:11.677882 | e8:de:27:1d:68:c7 | 802.11 State | handoff RSSI (-65 -66) RSSI (-61 -62)

2015-May- 7 13:37:17.682556 | e8:de:27:1d:68:c7 | 802.11 State | handoff RSSI (-66 -61) RSSI (-59 -59)

2015-May- 7 13:38:08.279700 | 20:68:9d:e0:61:03 | Station Assign | assigned to

2015-May- 7 13:38:08.280160 | 20:68:9d:e0:61:03 | Station Assign | Reassigning from (rssi=-75) to (rssi=-39)

2015-May- 7 13:38:08.280164 | 20:68:9d:e0:61:03 | Station Assign | Assign Removed From

2015-May- 7 13:38:08.280319 | 20:68:9d:e0:61:03 | Station Assign | assigned to

2015-May- 7 13:39:20.781974 | 80:19:34:a4:c8:8d | 802.11 State | handoff RSSI (-80 -76) RSSI (-74 -75)

 

You can see that as the signal of a device gets weaker, it will switch it to the new AP and report the signal strength back. Usually the numbers are very close as the system tries to balance the load, or as people move laptops, even walk in front of a laptop can give 10-20 Db of attenuation.

With the intel roaming aggressiveness I expect you to be seeing some -256 ie RSSI (-70 -256) or (-256 -256)

[SpaceInvader83]

Out of curiosity what AP's do you have Bruce?

[Bruce123]

832e, 832i and 332i.

 

Thanks,

 

Bruce.

[mjk]

I would advise turning off band steering completely.

Band steering only works by actively blocking access to the 2.4GHz channel.

If the device decides that the 2.4GHz signal is better but the wireless is actively denying you will see a lot of connection issues.

[Bruce123]

I would advise turning off band steering completely.

Band steering only works by actively blocking access to the 2.4GHz channel.

If the device decides that the 2.4GHz signal is better but the wireless is actively denying you will see a lot of connection issues.

 

From what I've read about how band steering works, it shouldn't cause any significant issues with connectivity:

 

With Band steering to 5Ghz enabled:

 

The AP stops sending beacons on the 2.4Ghz Band (or hides them).

 

So devices that employ "passive scanning" will not see the AP on 2.4Ghz, only 5Ghz.

Devices that employ "active scanning" will send a probe (on each band that it supports).

 

If the AP receives a probe on both bands (or 5Ghz only), it will only respond on 5Ghz and the device will connect at 5Ghz. If the AP receives a probe on 2.4Ghz, it will respond on 2.4Ghz and connect at 2.4Ghz.

 

Therefore the only clients that might have issues connecting are 2.4Ghz-only (legacy really) devices employing passive scanning (e.g. to save power).

 

I did wonder whether band steering may be having a detrimental affect, but from what I have read and understand, it shouldn't. But whether that is the case in practice, is another matter...

 

Many Thanks,

 

Bruce.

Edited October 20, 2016 by Bruce123

[mjk]

I did wonder whether band steering may be having a detrimental affect, but from what I have read and understand, it shouldn't. But whether that is the case in practice, is another matter...

 

Nothing to lose by trying it. Let us know if it makes a difference. it did for us.

[AlanD]

Just browsing through this stuff....really surprised that MERU/Fortinet are not really keen to get this sorted for you. They charge a fortune in annual charges so they should be all over you to get systems working reliably. I have always quite like the MERU concept of single channel architecture and it does seem effective at delivery to classrooms of devices - but clearly not in your case. They are not going to sell stuff in schools unless they prove themselves capable of sorting this. I understand that using their architecture require the APs work at max power. This means that they see lots of neighbouring APs ... and they work cooperatively to minimize co channel interference by backing off transmit in a co-operative fashion. Nevertheless - I am wondering what you physical arrangement is when they see 50 other APs! I suspect you need to turn off the 2.4 radio on a good number of APs. Only use it in an area where there is no AP in a room to prove 5GHz....but really its for Fortinet to sort this. You can "layer" stuff so use multiple channels to provide greater total bandwidth - if bandwidth is an issue - so that additional APs are on a different channel...no ...definitely a massive black mark for Fortinet here.

[Shawn-HDSB]

We had the same issues with Chromebooks (dell ones)

Power reduced to the entire school (in 2 sections as one section has an AP in each classroom and the other is spread out)

 

Section 1 High Density area

Interface 1 14

Interface 2 11

 

Section Rest of the school

Interface 1 17

Interface 2 14

 

Probe Response Threshold was increased to 20 for the entire school (set on the actual Access point)

 

Reduced the supported channels (dumped the B channel all together bu only allowing 11 mbps and removing anything below 12 mbps on all other transmit (or base transmit) rates

 

I am testing reducing the channels we use to 6 & 36 for the first floor 11&44 for second floor and for the few that have 3rd floors 6&36 again if they have portables as portables are 1&149

 

we have mostly ap 320i for the schools and 1020 in the portables (cant let the ones in the portables see internal as there is a chipset issue) upgrading a few schools to the 832's but that will take a few years

Edited January 26, 2017 by Shawn-HDSB

[TheRobins]

Can I just say thank you ever so much. I have been plodding along with some issues here and there and continuing on the single channel environment which in the large and for AC devices works fine.

 

I have now applied your recommendations as suggested and what a difference. Luckily I am in a large but very rural school with 25 AP's spread around 700 users so neighbouring AP's is not an issue, in fact our Rouge AP detection has 2 BT Home hubs that are seen from the top floor of our main building which is near 200ft off the floor so I have full use of the spectrum.

 

By splitting across Ch1,6,11 on 2.4ghz, setting to 40Mhz channel spacing and dropping EIRP to 14 it has made the world of difference. I would sometimes get G/N laptops that would just lose connection and not regain by just getting confused with the single channel environment. Also throughput has massively improved. This attached screen shot just shows a comparison of our G/N devices.

 

Even without chromebooks I am really happy with the improvements to our system since this change has been made. We are running a MC1550 Controller with AP832e's as AP's. Also running fortinet firmware of 8.0.5.0

 

Once again thanks for your advice.

We had the same issues with Chromebooks (dell ones)

Power reduced to the entire school (in 2 sections as one section has an AP in each classroom and the other is spread out)

 

Section 1 High Density area

Interface 1 14

Interface 2 11

 

Section Rest of the school

Interface 1 17

Interface 2 14

 

Probe Response Threshold was increased to 20 for the entire school (set on the actual Access point)

 

Reduced the supported channels (dumped the B channel all together bu only allowing 11 mbps and removing anything below 12 mbps on all other transmit (or base transmit) rates

 

I am testing reducing the channels we use to 6 & 36 for the first floor 11&44 for second floor and for the few that have 3rd floors 6&36 again if they have portables as portables are 1&149

 

we have mostly ap 320i for the schools and 1020 in the portables (cant let the ones in the portables see internal as there is a chipset issue) upgrading a few schools to the 832's but that will take a few years

[Shawn-HDSB]

Im glad it is working for you. We are still on version 7.10 but they will have to spend the money shortly to upgrade as the 320's are EOL maybe I should stop making them work so well Do you see any "sticky client" issues in version 8.X and the 832's I still have clients that want to connect to access points 4 rooms away (that is where the testing started for this design) and increasing the Probe Response Threshold on each radio from 15 to 20 has helped.

[AlanD]

"By splitting across Ch1,6,11 on 2.4ghz, setting to 40Mhz channel spacing​" I don't use Meru, so don't fully understand the implications of the single channel system...but....if you are using a channel width of 40Mhz on Ch1 then surely you can't use Ch6 because it will get trampled over by the wide Ch1 transmissions. So By using a "wide " channel - you only have capability to add one further layer on Ch 13. So either use 3 Layers with narrow 20Mhz channels on Ch 1,6, and 11 or use a single wide channel on Ch1 with a second narrow layer on Ch 11.

[Bruce123]

Just to update people on this issue.

 

We have largely resolved the issue through a combination of methods;

 

1. Reducing power to APs

 

We ended up turning down the power on each band:

 

2.4Ghz: 20 to 18dB

5Ghz: 23 to 18dB

 

This reduced the highest neighborhood counter on the 5Ghz band from 50 to 33.

 

2. Turning off certain APs where APs are densely located.

 

3. Implementing channel layering in areas of high density (i.e. cabinets of Chromebooks).

 

As we operate a 802.11ac 80Mhz single channel virtual cell (primary channels 36,40,44,48), the only other channels available to use for layering are DFS channels (3x 80Mhz wide channels and 1x 40Mhz wide channel) . But there is a surprising number of "radar detection" events on these channels, causing the affected AP to drop to the non-DFS 80Mhz channel for 30 minutes (merging back into the main virtual cell channel layer), before trying again.

 

4. Band steering to 5Ghz

 

The AP logs showed the Chromebooks intermittently switching between bands (to switch between bands it has to drop the link and re-establish it unlike switching between channels on the same band which is basically just a roaming operation). I am not sure why this was happening, perhaps an issue with virtual cell or perhaps this happens when the Chromebook detects high utilization on the band/channel and decides to switch.

 

5. Ensure all APs within same area connect to the same controller.

 

We had a case where the APs in one building connected to one controller and the APs in the adjoining building connected to another (licensing reasons). This was causing issues as the controllers couldn't cooperate to schedule the transmission of frames in the virtual cell. I notice that version 8 of SD introduces a feature that allows cooperation between controllers, so this wouldn't be an issue.

 

6. Positioned distribution layer switch stack between controller and core layer 3 switch.

 

The addressed an issue with a large ARP table (wireless clients) on the core.

 

7. Firmware upgrade to Bloxx

 

We use Bloxx as a transparent proxy, it would intermittently seem to stop operating as a transparent proxy (for a few seconds), which seems to have been resolved on the latest firmware

 

8. Upgrade SD to V8.1 (as pointed out - not available for 320i APs).

 

This seems to have resolved the issue where APs intermittently stop allowing clients to connect, requiring a reboot of the AP. It also introduces Auto Radio Provisioning for "Native Cell" (Micro Cell), which allows the controller to automatically select the best channels/power.

 

We may try also try disabling the very low connection speeds on the APs (as advised here and elsewhere) and enable 802.11k and 80.211r.

 

Moving forward: we are looking at switching from Virtual Cell to Native Cell (40Mhz wide channels). The choice of 40Mhz channel width is the "best" a trade off between number of channels and bandwidth per channel. It would provide us with 2x non-DFS channels and 8 DFS channels, which should be sufficient to minimize overlap (co-channel interference) between nearby APs (lower neighor counter), but also provide each AP/client with sufficient bandwidth (on testing a realistic throughput of 150Mbit for single a .ac client connected to a single 832 AP).

 

Kind Regards,

 

Bruce.

Edited February 2, 2017 by Bruce123

[Bruce123]

"By splitting across Ch1,6,11 on 2.4ghz, setting to 40Mhz channel spacing​" I don't use Meru, so don't fully understand the implications of the single channel system...but....if you are using a channel width of 40Mhz on Ch1 then surely you can't use Ch6 because it will get trampled over by the wide Ch1 transmissions. So By using a "wide " channel - you only have capability to add one further layer on Ch 13. So either use 3 Layers with narrow 20Mhz channels on Ch 1,6, and 11 or use a single wide channel on Ch1 with a second narrow layer on Ch 11.

 

Correct

[Bruce123]

Just browsing through this stuff....really surprised that MERU/Fortinet are not really keen to get this sorted for you. They charge a fortune in annual charges so they should be all over you to get systems working reliably. I have always quite like the MERU concept of single channel architecture and it does seem effective at delivery to classrooms of devices - but clearly not in your case. They are not going to sell stuff in schools unless they prove themselves capable of sorting this. I understand that using their architecture require the APs work at max power. This means that they see lots of neighbouring APs ... and they work cooperatively to minimize co channel interference by backing off transmit in a co-operative fashion. Nevertheless - I am wondering what you physical arrangement is when they see 50 other APs!

 

We have an eleven story building with 90 832 APs (average of 8 per floor), the AP with the highest neighborhood counter on 5Ghz band is (predictably) located on the middle of the 5th floor (i.e. central to all the other APs), the count is currently 26. The AP with the highest neighborhood counter on the 2.4Ghz band is 45 (also located centrally). We could probably do with reducing the power on the 2.4Ghz band further...

 

I suspect you need to turn off the 2.4 radio on a good number of APs. Only use it in an area where there is no AP in a room to prove 5GHz....but really its for Fortinet to sort this. You can "layer" stuff so use multiple channels to provide greater total bandwidth - if bandwidth is an issue - so that additional APs are on a different channel...no ...definitely a massive black mark for Fortinet here.

 

We aren't so bothered about the the 2.4Ghz band as we use band steering to 5Ghz, resulting about 20% of clients connecting on 2.4Ghz.

 

We're trying out layering (for both 2.4Ghz and 5Ghz) and it's working well to increase the available bandwidth (/reduce contention) in those areas of high client density, but we are experiencing occasional DFS drop-outs.

 

Re the level of support, sometimes it can be good, but they are too eager keen to close the job quickly, without waiting for feedback on whether the solution has worked.

 

Fortinet didn't provide any help/advise on with reducing the power on APs or nor channel layering, it's something worked out for ourselves.

 

Thanks,

 

Bruce.

[tcasper]

i am so glad to see this thread! Im having problems all of a sudden with the districts chomebooks. We have 540 hp g4 chromebooks. All of a sudden about a week ago chromebooks started dropping connection to the ap's. We have a meru mc3200 controller with ap1020. The chromebooks seem to be jumping from the 2.4 to 5ghz constantly. We basically have every other room in the building with an ap. Ive thought about doing some of the fixes seen on this thread but i dont thing we have the config that is listed here. any suggestions would be great. thank you in advance.

[Shawn-HDSB]

Are you forcing the band steering to A band with a timeout of 1 second? that is set in the ESS. dropping connection to the internet or connection to the AP? or to google docs? can the chromebooks surf other pages when it cant get to google docs ? how many connections to the access point are there? we see that when there are over 35 (and in use) some chromebooks get kicked off with older access points (320,1020) - or schools have also run out of ip addresses

[tcasper]

band steering is enabled. dropping connection to the ap. in the middle of a google doc students lose connection. they are unable to do anything until it reconnects itself. ip address are good. in theory 2aps across 3 rooms shouldbe enough for 90. its in every part of the building. timeout is set to 1.

[tcasper]

everything was fine until the controller took a hard reboot doing a typical diag run on sundays. its set by default. ever since then weve been having problems.

[AlanD]

I'd be really wary of band steering - especially if you have every other room where 5Ghz coverage is not great. If you know you have a strong signal everywhere - its usually fine. Anyhow- you need to work your way methodically through trying to isolate the problem....even if you think you have not changed anything...perhaps something you changed that didn't take effect before has now kicked in (and band steering would be an easy one to eliminate).

[Shawn-HDSB]

the reboot would have set some of your configs back to a previous version if it was not saved properly. Double check everything. have you pulled the station logs?

to get station logs for single computer

station-log show -mac 60:af:6d:a6:9b:6a

show station mac-address 60:af:6d:a6:9b:6a

 

that might point to where the issue is

[tcasper]

band steering was not only recommended by the company we bought the product from but it has also been recommneded by the manufactuer. the only other real changes were the radios. but those were changed back because that changed nothing. no other changes have been made 100%.

[tcasper]

this is what is coming of the station logs. according to the company this is the only thing that they see that could be the issue

 

 

2017-Feb-10 12:40:44.120897 | f0:42:1c:08:8b:c1 | Station Assign | [abgn] assigned to ESSID=irover BSSID=00:0c:e6:02:53:53 Ch=149 reason=Station probed

 

2017-Feb-10 12:40:47.241735 | f0:42:1c:08:8b:c1 | Station Assign | [abgn] assigned to ESSID=irover BSSID=00:0c:e6:02:72:00 Ch=11 reason=Station probed

 

2017-Feb-10 12:42:34.589757 | f0:42:1c:08:8b:c1 | 802.11 State | state change ESSID=irover Ch=149 type abgn

 

2017-Feb-10 12:42:34.666149 | f0:42:1c:08:8b:c1 | Station Assign | [abgn] assigned to ESSID=irover BSSID=00:0c:e6:02:72:00 Ch=11 reason=Forced assignment for sync with AP

 

2017-Feb-10 12:42:49.076611 | f0:42:1c:08:8b:c1 | 802.11 State | state change ESSID=irover Ch=149 type abgn